iox

README

iox: I/O tools for Go programs

Package iox contains two Go objects of note: Filer and BufferFile.

https://godoc.org/crawshaw.io/iox

Filer

Managing file resources in highly-concurrent programs gets tricky. A process easily, even typically, has more in-flight goroutines than allowed file descriptors from the operating system. This requires programmers limit the number of open descriptors with some kind of throttle object.

An iox.Filer wraps the functions used to open file descriptors and makes sure it never opens more than some maximum (typically derived from the processes rlimit).

It wraps *os.File pointers in a new object which returns the file descriptor allotment to the Filer pool when Close is called.

BufferFile

A BufferFile is a file-like object that stores its first N bytes in memory, and the rest in a temporary file on disk.

It is designed for loads where the typical case fits in some small amount of memory, but the worst case requires more space than can be provisioned in RAM. (This usually means a server is handling tens to hundreds of thousands of simultaneous requests.)

BufferFile does not create its temporary backing file until its contents exceed the memory buffer, so the typical case does not require any file descriptors. Programs can begin (and usually complete) processing a request without ever blocking on file descriptors, meaning a server never runs into file descriptors as a bottleneck when processing a typical workload.

Installation

Install with:

go get crawshaw.io/iox

There are no version numbers yet, this package needs some time to bake.

Documentation

Overview

Package iox contains I/O utilities.

The primary concern of the package is managing and minimizing the use of file descriptors, an operating system resource which is often in short supply in high-concurrency servers.

The two objects that help in this are the Filer and BufferFile.

A filer manages a allotment of file descriptors, blocking on file creation until an old file closes and frees up a descriptor allotment.

A BufferFile keeps a fraction of its contents in memory. If the number of bytes stored in a BufferFile is small, no file descriptor is ever used.

Index

• type BufferFile
  • func (bf *BufferFile) Close() (err error)
  • func (bf *BufferFile) Read(p []byte) (n int, err error)
  • func (bf *BufferFile) ReadAt(p []byte, off int64) (n int, err error)
  • func (bf *BufferFile) Seek(offset int64, whence int) (int64, error)
  • func (bf *BufferFile) Size() int64
  • func (bf *BufferFile) Truncate(size int64) error
  • func (bf *BufferFile) Write(p []byte) (n int, err error)
• type File
  • func (file *File) Close() error
• type Filer
  • func NewFiler(fdLimit int) *Filer
  • func (f *Filer) BufferFile(memSize int) *BufferFile
  • func (f *Filer) Open(name string) (*File, error)
  • func (f *Filer) OpenFile(name string, flag int, perm os.FileMode) (*File, error)
  • func (f *Filer) SetTempdir(tempdir string)
  • func (f *Filer) Shutdown(ctx context.Context) error
  • func (f *Filer) TempFile(dir, prefix, suffix string) (file *File, err error)

Types

type BufferFile

type BufferFile struct {
	io.Reader
	io.ReaderAt
	io.Writer
	io.Seeker
	io.Closer
	// contains filtered or unexported fields
}

BufferFile is a temporary file that stores its first N bytes in memory.

A BufferFile will not create an underlying temporary file until a Write or a Seek pushes it beyond its memory limit. This allows for typical cases where the contents fit in memory to avoid the disk entirely and not hold a file descriptor.

func (*BufferFile) Close

func (bf *BufferFile) Close() (err error)

Close closes the BufferFile, deleting any underlying temporary file.

func (*BufferFile) Read

func (bf *BufferFile) Read(p []byte) (n int, err error)

func (*BufferFile) ReadAt

func (bf *BufferFile) ReadAt(p []byte, off int64) (n int, err error)

func (*BufferFile) Seek

func (bf *BufferFile) Seek(offset int64, whence int) (int64, error)

func (*BufferFile) Size

func (bf *BufferFile) Size() int64

Size returns the current length of the buffer file. It is equivalent to the position returned by bf.Seek(0, os.SEEK_END).

func (*BufferFile) Truncate

func (bf *BufferFile) Truncate(size int64) error

Truncate changes the file size. It does not move the offset, use Seek for that.

func (*BufferFile) Write

func (bf *BufferFile) Write(p []byte) (n int, err error)

type File

type File struct {
	*os.File
	// contains filtered or unexported fields
}

File is an *os.File managed by a Filer.

The Close method must be called on a File.

func (*File) Close

func (file *File) Close() error

Close closes the underlying file descriptor and informs the Filer.

type Filer

type Filer struct {
	DefaultBufferMemSize int // default value: 64kb

	Logf func(format string, v ...interface{}) // used to report open files at Shutdown
	// contains filtered or unexported fields
}

A Filer creates files, managing load on file descriptors.

Exported fields can only be modified after NewFiler is called and before any methods are called.

func NewFiler

func NewFiler(fdLimit int) *Filer

NewFiler creates a Filer which will open at most fdLimit files simultaneously. If fdLimit is 0, a Filer is limited to 90% of the process's allowed files.

func (*Filer) BufferFile

func (f *Filer) BufferFile(memSize int) *BufferFile

BufferFile creates a buffered file with up to memSize bytes stored in memory.

If memSize is zero, the Filer's default value is used.

The underlying file descriptor should not be handled directly as the fraction of the contents stored in the OS file may change.

func (*Filer) Open

func (f *Filer) Open(name string) (*File, error)

Open opens the named file for reading.

It is similar to os.Open except it will block if Filer has exhasted its file descriptors until one is available.

func (*Filer) OpenFile

func (f *Filer) OpenFile(name string, flag int, perm os.FileMode) (*File, error)

OpenFile is a generalized file open method.

It is similar to os.OpenFile except it will block if Filer has exhasted its file descriptors until one is available.

func (*Filer) SetTempdir

func (f *Filer) SetTempdir(tempdir string)

SetTempdir sets the default directory used to hold temporary files.

func (*Filer) Shutdown

func (f *Filer) Shutdown(ctx context.Context) error

Shutdown gracefully shuts down the Filer. Any active files continue to work until the passed context is done. At that point they are explicitly closed and further operations return errors. Shutdown returns the error from ctx.

func (*Filer) TempFile

func (f *Filer) TempFile(dir, prefix, suffix string) (file *File, err error)