buffer

package module
v1.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 2, 2019 License: MIT Imports: 13 Imported by: 17

README

Buffer

GoDoc Release Software License Build Status Coverage Status Go Report Card

Usage

The following buffers provide simple unique behaviours which when composed can create complex buffering strategies. For use with github.com/djherbis/nio for Buffered io.Pipe and io.Copy implementations.

For example:

import (
  "github.com/djherbis/buffer"
  "github.com/djherbis/nio"
  
  "io/ioutil"
)

// Buffer 32KB to Memory, after that buffer to 100MB chunked files
buf := buffer.NewUnboundedBuffer(32*1024, 100*1024*1024)
nio.Copy(w, r, buf) // Reads from r, writes to buf, reads from buf writes to w (concurrently).

// Buffer 32KB to Memory, discard overflow
buf = buffer.NewSpill(32*1024, ioutil.Discard)
nio.Copy(w, r, buf)

Supported Buffers

Bounded Buffers

Memory: Wrapper for bytes.Buffer

File: File-based buffering. The file never exceeds Cap() in length, no matter how many times its written/read from. It accomplishes this by "wrapping" around the fixed max-length file when the data gets too long but there is available freed space at the beginning of the file. The caller is responsible for closing and deleting the file when done.

import (
  "ioutil"
  "os"
  
  "github.com/djherbis/buffer"
)

// Create a File-based Buffer with max size 100MB
file, err := ioutil.TempFile("", "buffer")
if err != nil {
	return err
}
defer os.Remove(file.Name())
defer file.Close()

buf := buffer.NewFile(100*1024*1024, file)

// A simpler way:
pool := buffer.NewFilePool(100*1024*1024, "") // "" -- use temp dir
buf, err := pool.Get()   // allocate the buffer
if err != nil {
  return err
}
defer pool.Put(buf) // close and remove the allocated file for the buffer

Multi: A fixed length linked-list of buffers. Each buffer reads from the next buffer so that all the buffered data is shifted upwards in the list when reading. Writes are always written to the first buffer in the list whose Len() < Cap().

import (
  "github.com/djherbis/buffer"
)

mem  := buffer.New(32*1024)
file := buffer.NewFile(100*1024*1024, someFileObj)) // you'll need to manage Open(), Close() and Delete someFileObj

// Buffer composed of 32KB of memory, and 100MB of file.
buf := buffer.NewMulti(mem, file)
Unbounded Buffers

Partition: A queue of buffers. Writes always go to the last buffer in the queue. If all buffers are full, a new buffer is "pushed" to the end of the queue (generated by a user-given function). Reads come from the first buffer, when the first buffer is emptied it is "popped" off the queue.

import (
  "github.com/djherbis/buffer"
)

// Create 32 KB sized-chunks of memory as needed to expand/contract the buffer size.
buf := buffer.NewPartition(buffer.NewMemPool(32*1024))

// Create 100 MB sized-chunks of files as needed to expand/contract the buffer size.
buf = buffer.NewPartition(buffer.NewFilePool(100*1024*1024, ""))

Ring: A single buffer which begins overwriting the oldest buffered data when it reaches its capacity.

import (
  "github.com/djherbis/buffer"
)

// Create a File-based Buffer with max size 100MB
file := buffer.NewFile(100*1024*1024, someFileObj) // you'll need to Open(), Close() and Delete someFileObj.

// If buffered data exceeds 100MB, overwrite oldest data as new data comes in
buf := buffer.NewRing(file) // requires BufferAt interface.

Spill: A single buffer which when full, writes the overflow to a given io.Writer. -> Note that it will actually "spill" whenever there is an error while writing, this should only be a "full" error.

import (
  "github.com/djherbis/buffer"
  "github.com/djherbis/nio"
  
  "io/ioutil"
)

// Buffer 32KB to Memory, discard overflow
buf := buffer.NewSpill(32*1024, ioutil.Discard)
nio.Copy(w, r, buf)
Empty Buffer

Discard: Reads always return EOF, writes goto ioutil.Discard.

import (
  "github.com/djherbis/buffer"
)

// Reads will return io.EOF, writes will return success (nil error, full write) but no data was written.
buf := buffer.Discard

Custom Buffers

Feel free to implement your own buffer, just meet the required interface (Buffer/BufferAt) and compose away!


// Buffer Interface used by Multi and Partition
type Buffer interface {
	Len() int64
	Cap() int64
	io.Reader
	io.Writer
	Reset()
}

// BufferAt interface used by Ring
type BufferAt interface {
	Buffer
	io.ReaderAt
	io.WriterAt
}

Installation

go get github.com/djherbis/buffer

Documentation

Overview

Package buffer implements a series of Buffers which can be composed to implement complicated buffering strategies

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Empty

func Empty(buf Buffer) bool

Empty returns false iff buf.Len() == 0

func Full

func Full(buf Buffer) bool

Full returns true iff buf.Len() == buf.Cap()

func Gap

func Gap(buf Buffer) int64

Gap returns buf.Cap() - buf.Len()

Types

type Buffer

type Buffer interface {
	Len() int64 // How much data is Buffered in bytes
	Cap() int64 // How much data can be Buffered at once in bytes.
	io.Reader   // Read() will read from the top of the buffer [io.EOF if empty]
	io.Writer   // Write() will write to the end of the buffer [io.ErrShortWrite if not enough space]
	Reset()     // Truncates the buffer, Len() == 0.
}

Buffer is used to Write() data which will be Read() later.

var Discard Buffer = discard{}

Discard is a Buffer which writes to ioutil.Discard and read's return 0, io.EOF. All of its methods are concurrent safe.

func NewMulti

func NewMulti(buffers ...Buffer) Buffer

NewMulti returns a Buffer which is the logical concatenation of the passed buffers. The data in the buffers is shifted such that there is no non-empty buffer following a non-full buffer, this process is also run after every Read. If no buffers are passed, the returned Buffer is nil.

func NewPartition

func NewPartition(pool Pool, buffers ...Buffer) Buffer

NewPartition returns a Buffer which uses a Pool to extend or shrink its size as needed. It automatically allocates new buffers with pool.Get() to extend is length, and pool.Put() to release unused buffers as it shrinks.

func NewRing

func NewRing(buffer BufferAt) Buffer

NewRing returns a Ring Buffer from a BufferAt. It overwrites old data in the Buffer when needed (when its full).

func NewSpill

func NewSpill(buf Buffer, w io.Writer) Buffer

NewSpill returns a Buffer which writes data to w when there's an error writing to buf. Such as when buf is full, or the disk is full, etc.

func NewSwap

func NewSwap(a, b Buffer) Buffer

NewSwap creates a Buffer which writes to a until you write past a.Cap() then it io.Copy's from a to b and writes to b. Once the Buffer is empty again, it starts over writing to a. Note that if b.Cap() <= a.Cap() it will cause a panic, b is expected to be larger in order to accommodate writes past a.Cap().

func NewUnboundedBuffer

func NewUnboundedBuffer(mem, file int64) Buffer

NewUnboundedBuffer returns a Buffer which buffers "mem" bytes to memory and then creates file's of size "file" to buffer above "mem" bytes.

type BufferAt

type BufferAt interface {
	Buffer
	io.ReaderAt
	io.WriterAt
}

BufferAt is a buffer which supports io.ReaderAt and io.WriterAt

func New

func New(n int64) BufferAt

New returns a new in memory BufferAt with max size N. It's backed by a bytes.Buffer.

func NewFile

func NewFile(N int64, file File) BufferAt

NewFile returns a new BufferAt backed by "file" with max-size N.

func NewMultiAt

func NewMultiAt(buffers ...BufferAt) BufferAt

NewMultiAt returns a BufferAt which is the logical concatenation of the passed BufferAts. The data in the buffers is shifted such that there is no non-empty buffer following a non-full buffer, this process is also run after every Read. If no buffers are passed, the returned Buffer is nil.

func NewPartitionAt added in v1.1.0

func NewPartitionAt(pool PoolAt, buffers ...BufferAt) BufferAt

NewPartitionAt returns a BufferAt which uses a PoolAt to extend or shrink its size as needed. It automatically allocates new buffers with pool.Get() to extend is length, and pool.Put() to release unused buffers as it shrinks.

func NewSwapAt

func NewSwapAt(a, b BufferAt) BufferAt

NewSwapAt creates a BufferAt which writes to a until you write past a.Cap() then it io.Copy's from a to b and writes to b. Once the Buffer is empty again, it starts over writing to a. Note that if b.Cap() <= a.Cap() it will cause a panic, b is expected to be larger in order to accommodate writes past a.Cap().

type File

type File interface {
	Name() string
	Stat() (fi os.FileInfo, err error)
	io.ReaderAt
	io.WriterAt
	Close() error
}

File is used as the backing resource for a the NewFile BufferAt.

type List

type List []Buffer

List is a slice of Buffers, it's the backing for NewPartition

func (*List) Cap

func (l *List) Cap() (n int64)

Cap is the sum of the Cap()'s of the Buffers in the List.

func (*List) Len

func (l *List) Len() (n int64)

Len is the sum of the Len()'s of the Buffers in the List.

func (*List) Pop

func (l *List) Pop() (b Buffer)

Pop removes and returns a Buffer from the front of the List

func (*List) Push

func (l *List) Push(b Buffer)

Push adds a Buffer to the end of the List

func (*List) Reset

func (l *List) Reset()

Reset calls Reset() on each of the Buffers in the list.

type ListAt added in v1.1.0

type ListAt []BufferAt

ListAt is a slice of BufferAt's, it's the backing for NewPartitionAt

func (*ListAt) Cap added in v1.1.0

func (l *ListAt) Cap() (n int64)

Cap is the sum of the Cap()'s of the BufferAt's in the list.

func (*ListAt) Len added in v1.1.0

func (l *ListAt) Len() (n int64)

Len is the sum of the Len()'s of the BufferAt's in the list.

func (*ListAt) Pop added in v1.1.0

func (l *ListAt) Pop() (b BufferAt)

Pop removes and returns a BufferAt from the front of the list

func (*ListAt) Push added in v1.1.0

func (l *ListAt) Push(b BufferAt)

Push adds a BufferAt to the end of the list

func (*ListAt) Reset added in v1.1.0

func (l *ListAt) Reset()

Reset calls Reset() on each of the BufferAt's in the list.

type Pool

type Pool interface {
	Get() (Buffer, error) // Allocate a Buffer
	Put(buf Buffer) error // Release or Reuse a Buffer
}

Pool provides a way to Allocate and Release Buffer objects Pools mut be concurrent-safe for calls to Get() and Put().

func NewFilePool

func NewFilePool(N int64, dir string) Pool

NewFilePool returns a Pool, Get() returns a file-based buffer of max size N. Put() closes and deletes the underlying file for the buffer. Get() may return an error if it fails to create a file for the buffer. Put() may return an error if it fails to delete the file.

func NewMemPool

func NewMemPool(N int64) Pool

NewMemPool returns a Pool, Get() returns an in memory buffer of max size N. Put() returns the buffer to the pool after resetting it. Get() and Put() errors will always be nil.

func NewPool

func NewPool(New func() Buffer) Pool

NewPool returns a Pool(), it's backed by a sync.Pool so its safe for concurrent use. Get() and Put() errors will always be nil. It will not work with gob.

type PoolAt added in v1.1.0

type PoolAt interface {
	Get() (BufferAt, error) // Allocate a BufferAt
	Put(buf BufferAt) error // Release or Reuse a BufferAt
}

PoolAt provides a way to Allocate and Release BufferAt objects PoolAt's mut be concurrent-safe for calls to Get() and Put().

func NewFilePoolAt added in v1.1.0

func NewFilePoolAt(N int64, dir string) PoolAt

NewFilePoolAt returns a PoolAt, Get() returns a file-based buffer of max size N. Put() closes and deletes the underlying file for the buffer. Get() may return an error if it fails to create a file for the buffer. Put() may return an error if it fails to delete the file.

func NewMemPoolAt added in v1.1.0

func NewMemPoolAt(N int64) PoolAt

NewMemPoolAt returns a PoolAt, Get() returns an in memory buffer of max size N. Put() returns the buffer to the pool after resetting it. Get() and Put() errors will always be nil.

func NewPoolAt added in v1.1.0

func NewPoolAt(New func() BufferAt) PoolAt

NewPoolAt returns a PoolAt(), it's backed by a sync.Pool so its safe for concurrent use. Get() and Put() errors will always be nil. It will not work with gob.

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL