autofile

README

go-autofile

Documentation

Index

• Variables
• func GroupCheckDuration(duration time.Duration) func(*Group)
• func GroupHeadSizeLimit(limit int64) func(*Group)
• func GroupTotalSizeLimit(limit int64) func(*Group)
• type AutoFile
  • func OpenAutoFile(ctx context.Context, path string) (*AutoFile, error)
  • func (af *AutoFile) Close() error
  • func (af *AutoFile) Size() (int64, error)
  • func (af *AutoFile) Sync() error
  • func (af *AutoFile) Write(b []byte) (n int, err error)
• type Group
  • func OpenGroup(ctx context.Context, logger log.Logger, headPath string, ...) (*Group, error)
  • func (g *Group) Buffered() int
  • func (g *Group) Close()
  • func (g *Group) FlushAndSync() error
  • func (g *Group) HeadSizeLimit() int64
  • func (g *Group) MaxIndex() int
  • func (g *Group) MinIndex() int
  • func (g *Group) NewReader(index int) (*GroupReader, error)
  • func (g *Group) OnStart(ctx context.Context) error
  • func (g *Group) OnStop()
  • func (g *Group) ReadGroupInfo() GroupInfo
  • func (g *Group) TotalSizeLimit() int64
  • func (g *Group) Write(p []byte) (nn int, err error)
  • func (g *Group) WriteLine(line string) error
• type GroupInfo
• type GroupReader
  • func (gr *GroupReader) Close() error
  • func (gr *GroupReader) CurIndex() int
  • func (gr *GroupReader) Read(p []byte) (n int, err error)
  • func (gr *GroupReader) SetIndex(index int) error

Variables

var ErrAutoFileClosed = errors.New("autofile is closed")

ErrAutoFileClosed is reported when operations attempt to use an autofile after it has been closed.

Functions

func GroupCheckDuration

func GroupCheckDuration(duration time.Duration) func(*Group)

GroupCheckDuration allows you to overwrite default groupCheckDuration.

func GroupHeadSizeLimit

func GroupHeadSizeLimit(limit int64) func(*Group)

GroupHeadSizeLimit allows you to overwrite default head size limit - 10MB.

func GroupTotalSizeLimit

func GroupTotalSizeLimit(limit int64) func(*Group)

GroupTotalSizeLimit allows you to overwrite default total size limit of the group - 1GB.

Types

type AutoFile

type AutoFile struct {
	ID   string
	Path string
	// contains filtered or unexported fields
}

AutoFile automatically closes and re-opens file for writing. The file is automatically setup to close itself every 1s and upon receiving SIGHUP.

This is useful for using a log file with the logrotate tool.

func OpenAutoFile

func OpenAutoFile(ctx context.Context, path string) (*AutoFile, error)

OpenAutoFile creates an AutoFile in the path (with random ID). If there is an error, it will be of type *PathError or *ErrPermissionsChanged (if file's permissions got changed (should be 0600)).

func (*AutoFile) Close

func (af *AutoFile) Close() error

Close shuts down the service goroutine and marks af as invalid. Operations on af after Close will report an error.

func (*AutoFile) Size

func (af *AutoFile) Size() (int64, error)

Size returns the size of the AutoFile. It returns -1 and an error if fails get stats or open file. Opens AutoFile if needed.

func (*AutoFile) Sync

func (af *AutoFile) Sync() error

Sync commits the current contents of the file to stable storage. Typically, this means flushing the file system's in-memory copy of recently written data to disk.

func (*AutoFile) Write

func (af *AutoFile) Write(b []byte) (n int, err error)

Write writes len(b) bytes to the AutoFile. It returns the number of bytes written and an error, if any. Write returns a non-nil error when n != len(b). Opens AutoFile if needed.

type Group

type Group struct {
	service.BaseService

	ID   string
	Head *AutoFile // The head AutoFile to write to

	Dir string // Directory that contains .Head
	// contains filtered or unexported fields
}

You can open a Group to keep restrictions on an AutoFile, like the maximum size of each chunk, and/or the total amount of bytes stored in the group.

The first file to be written in the Group.Dir is the head file.

Dir/
- <HeadPath>

Once the Head file reaches the size limit, it will be rotated.

Dir/
- <HeadPath>.000   // First rolled file
- <HeadPath>       // New head path, starts empty.
									 // The implicit index is 001.

As more files are written, the index numbers grow...

Dir/
- <HeadPath>.000   // First rolled file
- <HeadPath>.001   // Second rolled file
- ...
- <HeadPath>       // New head path

The Group can also be used to binary-search for some line, assuming that marker lines are written occasionally.

func OpenGroup

func OpenGroup(ctx context.Context, logger log.Logger, headPath string, groupOptions ...func(*Group)) (*Group, error)

OpenGroup creates a new Group with head at headPath. It returns an error if it fails to open head file.

func (*Group) Buffered

func (g *Group) Buffered() int

Buffered returns the size of the currently buffered data.

func (*Group) Close

func (g *Group) Close()

Close closes the head file. The group must be stopped by this moment.

func (*Group) FlushAndSync

func (g *Group) FlushAndSync() error

FlushAndSync writes any buffered data to the underlying file and commits the current content of the file to stable storage (fsync).

func (*Group) HeadSizeLimit

func (g *Group) HeadSizeLimit() int64

HeadSizeLimit returns the current head size limit.

func (*Group) MaxIndex

func (g *Group) MaxIndex() int

MaxIndex returns index of the last file in the group.

func (*Group) MinIndex

func (g *Group) MinIndex() int

MinIndex returns index of the first file in the group.

func (*Group) NewReader

func (g *Group) NewReader(index int) (*GroupReader, error)

NewReader returns a new group reader. CONTRACT: Caller must close the returned GroupReader.

func (*Group) OnStart

func (g *Group) OnStart(ctx context.Context) error

OnStart implements service.Service by starting the goroutine that checks file and group limits.

func (*Group) OnStop

func (g *Group) OnStop()

OnStop implements service.Service by stopping the goroutine described above. NOTE: g.Head must be closed separately using Close.

func (*Group) ReadGroupInfo

func (g *Group) ReadGroupInfo() GroupInfo

Returns info after scanning all files in g.Head's dir.

func (*Group) TotalSizeLimit

func (g *Group) TotalSizeLimit() int64

TotalSizeLimit returns total size limit of the group.

func (*Group) Write

func (g *Group) Write(p []byte) (nn int, err error)

Write writes the contents of p into the current head of the group. It returns the number of bytes written. If nn < len(p), it also returns an error explaining why the write is short. NOTE: Writes are buffered so they don't write synchronously TODO: Make it halt if space is unavailable

func (*Group) WriteLine

func (g *Group) WriteLine(line string) error

WriteLine writes line into the current head of the group. It also appends "\n". NOTE: Writes are buffered so they don't write synchronously TODO: Make it halt if space is unavailable

type GroupInfo

type GroupInfo struct {
	MinIndex  int   // index of the first file in the group, including head
	MaxIndex  int   // index of the last file in the group, including head
	TotalSize int64 // total size of the group
	HeadSize  int64 // size of the head
}

GroupInfo holds information about the group.

type GroupReader

type GroupReader struct {
	*Group
	// contains filtered or unexported fields
}

GroupReader provides an interface for reading from a Group.

func (*GroupReader) Close

func (gr *GroupReader) Close() error

Close closes the GroupReader by closing the cursor file.

func (*GroupReader) CurIndex

func (gr *GroupReader) CurIndex() int

CurIndex returns cursor's file index.

func (*GroupReader) Read

func (gr *GroupReader) Read(p []byte) (n int, err error)

Read implements io.Reader, reading bytes from the current Reader incrementing index until enough bytes are read.

func (*GroupReader) SetIndex

func (gr *GroupReader) SetIndex(index int) error

SetIndex sets the cursor's file index to index by opening a file at this position.