autofile

README

go-autofile

Documentation

Index

• func GroupCheckDuration(duration time.Duration) func(*Group)
• func GroupHeadSizeLimit(limit int64) func(*Group)
• func GroupTotalSizeLimit(limit int64) func(*Group)
• type AutoFile
  • func OpenAutoFile(path string) (*AutoFile, error)
  • func (af *AutoFile) Close() error
  • func (af *AutoFile) Size() (int64, error)
  • func (af *AutoFile) Sync() error
  • func (af *AutoFile) Write(b []byte) (n int, err error)
• type Group
  • func OpenGroup(headPath string, groupOptions ...func(*Group)) (g *Group, err error)
  • func (g *Group) Buffered() int
  • func (g *Group) Close()
  • func (g *Group) FindLast(prefix string) (match string, found bool, err error)
  • func (g *Group) FlushAndSync() error
  • func (g *Group) HeadSizeLimit() int64
  • func (g *Group) MaxIndex() int
  • func (g *Group) MinIndex() int
  • func (g *Group) NewReader(index int) (*GroupReader, error)
  • func (g *Group) OnStart() error
  • func (g *Group) OnStop()
  • func (g *Group) ReadGroupInfo() GroupInfo
  • func (g *Group) RotateFile()
  • func (g *Group) Search(prefix string, cmp SearchFunc) (*GroupReader, bool, error)
  • func (g *Group) TotalSizeLimit() int64
  • func (g *Group) Wait()
  • func (g *Group) Write(p []byte) (nn int, err error)
  • func (g *Group) WriteLine(line string) error
• type GroupInfo
• type GroupReader
  • func (gr *GroupReader) Close() error
  • func (gr *GroupReader) CurIndex() int
  • func (gr *GroupReader) PushLine(line string)
  • func (gr *GroupReader) Read(p []byte) (n int, err error)
  • func (gr *GroupReader) ReadLine() (string, error)
  • func (gr *GroupReader) SetIndex(index int) error
• type SearchFunc
  • func MakeSimpleSearchFunc(prefix string, target int) SearchFunc

Functions

func GroupCheckDuration

func GroupCheckDuration(duration time.Duration) func(*Group)

GroupCheckDuration allows you to overwrite default groupCheckDuration.

func GroupHeadSizeLimit

func GroupHeadSizeLimit(limit int64) func(*Group)

GroupHeadSizeLimit allows you to overwrite default head size limit - 10MB.

func GroupTotalSizeLimit

func GroupTotalSizeLimit(limit int64) func(*Group)

GroupTotalSizeLimit allows you to overwrite default total size limit of the group - 1GB.

Types

type AutoFile

type AutoFile struct {
	ID   string
	Path string
	// contains filtered or unexported fields
}

AutoFile automatically closes and re-opens file for writing. The file is automatically setup to close itself every 1s and upon receiving SIGHUP.

This is useful for using a log file with the logrotate tool.

func OpenAutoFile

func OpenAutoFile(path string) (*AutoFile, error)

OpenAutoFile creates an AutoFile in the path (with random ID). If there is an error, it will be of type *PathError or *ErrPermissionsChanged (if file's permissions got changed (should be 0600)).

func (*AutoFile) Close

func (af *AutoFile) Close() error

Close shuts down the closing goroutine, SIGHUP handler and closes the AutoFile.

func (*AutoFile) Size

func (af *AutoFile) Size() (int64, error)

Size returns the size of the AutoFile. It returns -1 and an error if fails get stats or open file. Opens AutoFile if needed.

func (*AutoFile) Sync

func (af *AutoFile) Sync() error

Sync commits the current contents of the file to stable storage. Typically, this means flushing the file system's in-memory copy of recently written data to disk. Opens AutoFile if needed.

func (*AutoFile) Write

func (af *AutoFile) Write(b []byte) (n int, err error)

Write writes len(b) bytes to the AutoFile. It returns the number of bytes written and an error, if any. Write returns a non-nil error when n != len(b). Opens AutoFile if needed.

type Group

type Group struct {
	cmn.BaseService

	ID   string
	Head *AutoFile // The head AutoFile to write to

	Dir string // Directory that contains .Head
	// contains filtered or unexported fields
}

You can open a Group to keep restrictions on an AutoFile, like the maximum size of each chunk, and/or the total amount of bytes stored in the group.

The first file to be written in the Group.Dir is the head file.

Dir/
- <HeadPath>

Once the Head file reaches the size limit, it will be rotated.

Dir/
- <HeadPath>.000   // First rolled file
- <HeadPath>       // New head path, starts empty.
									 // The implicit index is 001.

As more files are written, the index numbers grow...

Dir/
- <HeadPath>.000   // First rolled file
- <HeadPath>.001   // Second rolled file
- ...
- <HeadPath>       // New head path

The Group can also be used to binary-search for some line, assuming that marker lines are written occasionally.

func OpenGroup

func OpenGroup(headPath string, groupOptions ...func(*Group)) (g *Group, err error)

OpenGroup creates a new Group with head at headPath. It returns an error if it fails to open head file.

func (*Group) Buffered

func (g *Group) Buffered() int

Buffered returns the size of the currently buffered data.

func (*Group) Close

func (g *Group) Close()

Close closes the head file. The group must be stopped by this moment.

func (*Group) FindLast

func (g *Group) FindLast(prefix string) (match string, found bool, err error)

Searches backwards for the last line in Group with prefix. Scans each file forward until the end to find the last match.

func (*Group) FlushAndSync

func (g *Group) FlushAndSync() error

FlushAndSync writes any buffered data to the underlying file and commits the current content of the file to stable storage (fsync).

func (*Group) HeadSizeLimit

func (g *Group) HeadSizeLimit() int64

HeadSizeLimit returns the current head size limit.

func (*Group) MaxIndex

func (g *Group) MaxIndex() int

MaxIndex returns index of the last file in the group.

func (*Group) MinIndex

func (g *Group) MinIndex() int

MinIndex returns index of the first file in the group.

func (*Group) NewReader

func (g *Group) NewReader(index int) (*GroupReader, error)

NewReader returns a new group reader. CONTRACT: Caller must close the returned GroupReader.

func (*Group) OnStart

func (g *Group) OnStart() error

OnStart implements cmn.Service by starting the goroutine that checks file and group limits.

func (*Group) OnStop

func (g *Group) OnStop()

OnStop implements cmn.Service by stopping the goroutine described above. NOTE: g.Head must be closed separately using Close.

func (*Group) ReadGroupInfo

func (g *Group) ReadGroupInfo() GroupInfo

Returns info after scanning all files in g.Head's dir.

func (*Group) RotateFile

func (g *Group) RotateFile()

RotateFile causes group to close the current head and assign it some index. Note it does not create a new head.

func (*Group) Search

func (g *Group) Search(prefix string, cmp SearchFunc) (*GroupReader, bool, error)

Searches for the right file in Group, then returns a GroupReader to start streaming lines. Returns true if an exact match was found, otherwise returns the next greater line that starts with prefix. CONTRACT: Caller must close the returned GroupReader

func (*Group) TotalSizeLimit

func (g *Group) TotalSizeLimit() int64

TotalSizeLimit returns total size limit of the group.

func (*Group) Wait

func (g *Group) Wait()

Wait blocks until all internal goroutines are finished. Supposed to be called after Stop.

func (*Group) Write

func (g *Group) Write(p []byte) (nn int, err error)

Write writes the contents of p into the current head of the group. It returns the number of bytes written. If nn < len(p), it also returns an error explaining why the write is short. NOTE: Writes are buffered so they don't write synchronously TODO: Make it halt if space is unavailable

func (*Group) WriteLine

func (g *Group) WriteLine(line string) error

WriteLine writes line into the current head of the group. It also appends "\n". NOTE: Writes are buffered so they don't write synchronously TODO: Make it halt if space is unavailable

type GroupInfo

type GroupInfo struct {
	MinIndex  int   // index of the first file in the group, including head
	MaxIndex  int   // index of the last file in the group, including head
	TotalSize int64 // total size of the group
	HeadSize  int64 // size of the head
}

GroupInfo holds information about the group.

type GroupReader

type GroupReader struct {
	*Group
	// contains filtered or unexported fields
}

GroupReader provides an interface for reading from a Group.

func (*GroupReader) Close

func (gr *GroupReader) Close() error

Close closes the GroupReader by closing the cursor file.

func (*GroupReader) CurIndex

func (gr *GroupReader) CurIndex() int

CurIndex returns cursor's file index.

func (*GroupReader) PushLine

func (gr *GroupReader) PushLine(line string)

PushLine makes the given line the current one, so the next time somebody calls ReadLine, this line will be returned. panics if called twice without calling ReadLine.

func (*GroupReader) Read

func (gr *GroupReader) Read(p []byte) (n int, err error)

Read implements io.Reader, reading bytes from the current Reader incrementing index until enough bytes are read.

func (*GroupReader) ReadLine

func (gr *GroupReader) ReadLine() (string, error)

ReadLine reads a line (without delimiter). just return io.EOF if no new lines found.

func (*GroupReader) SetIndex

func (gr *GroupReader) SetIndex(index int) error

SetIndex sets the cursor's file index to index by opening a file at this position.

type SearchFunc

type SearchFunc func(line string) (int, error)

Returns -1 if line comes after, 0 if found, 1 if line comes before.

func MakeSimpleSearchFunc

func MakeSimpleSearchFunc(prefix string, target int) SearchFunc

A simple SearchFunc that assumes that the marker is of form <prefix><number>. For example, if prefix is '#HEIGHT:', the markers of expected to be of the form:

#HEIGHT:1 ... #HEIGHT:2 ...