pool

Documentation

Overview

Package pool implements a memory pool similar in concept to sync.Pool but with more determinism.

Index

• type DelayAccountinger
• type Pool
  • func New(flushTime time.Duration, bufferSize, poolSize int, useMmap bool) *Pool
  • func (bp *Pool) Alloced() int
  • func (bp *Pool) Flush()
  • func (bp *Pool) Get() []byte
  • func (bp *Pool) InPool() int
  • func (bp *Pool) InUse() int
  • func (bp *Pool) Put(buf []byte)
• type RW
  • func NewRW(pool *Pool) *RW
  • func (rw *RW) Close() error
  • func (rw *RW) DelayAccounting(i int)
  • func (rw *RW) Read(p []byte) (n int, err error)
  • func (rw *RW) ReadFrom(r io.Reader) (n int64, err error)
  • func (rw *RW) Seek(offset int64, whence int) (int64, error)
  • func (rw *RW) SetAccounting(account RWAccount) *RW
  • func (rw *RW) Size() int64
  • func (rw *RW) WaitWrite(ctx context.Context)
  • func (rw *RW) Write(p []byte) (n int, err error)
  • func (rw *RW) WriteTo(w io.Writer) (n int64, err error)
• type RWAccount

Types

type DelayAccountinger

type DelayAccountinger interface {
	// DelayAccounting makes sure the accounting function only
	// gets called on the i-th or later read of the data from this
	// point (counting from 1).
	//
	// This is useful so that we don't account initial reads of
	// the data e.g. when calculating hashes.
	//
	// Set this to 0 to account everything.
	DelayAccounting(i int)
}

DelayAccountinger enables an accounting delay

type Pool

type Pool struct {
	// contains filtered or unexported fields
}

Pool of internal buffers

We hold buffers in cache. Every time we Get or Put we update minFill which is the minimum len(cache) seen.

Every flushTime we remove minFill buffers from the cache as they were not used in the previous flushTime interval.

func New

func New(flushTime time.Duration, bufferSize, poolSize int, useMmap bool) *Pool

New makes a buffer pool

flushTime is the interval the buffer pools is flushed bufferSize is the size of the allocations poolSize is the maximum number of free buffers in the pool useMmap should be set to use mmap allocations

func (*Pool) Alloced

func (bp *Pool) Alloced() int

Alloced returns the number of buffers allocated and not yet freed

func (*Pool) Flush

func (bp *Pool) Flush()

Flush the entire buffer pool

func (*Pool) Get

func (bp *Pool) Get() []byte

Get a buffer from the pool or allocate one

func (*Pool) InPool

func (bp *Pool) InPool() int

InPool returns the number of buffers in the pool

func (*Pool) InUse

func (bp *Pool) InUse() int

InUse returns the number of buffers in use which haven't been returned to the pool

func (*Pool) Put

func (bp *Pool) Put(buf []byte)

Put returns the buffer to the buffer cache or frees it

Note that if you try to return a buffer of the wrong size to Put it will panic.

type RW

type RW struct {
	// contains filtered or unexported fields
}

RW contains the state for the read/writer

It can be used as a FIFO to read data from a source and write it out again.

func NewRW

func NewRW(pool *Pool) *RW

NewRW returns a reader / writer which is backed from pages from the pool passed in.

Data can be stored in it by calling Write and read from it by calling Read.

When writing it only appends data. Seek only applies to reading.

func (*RW) Close

func (rw *RW) Close() error

Close the buffer returning memory to the pool

func (*RW) DelayAccounting

func (rw *RW) DelayAccounting(i int)

DelayAccounting makes sure the accounting function only gets called on the i-th or later read of the data from this point (counting from 1).

This is useful so that we don't account initial reads of the data e.g. when calculating hashes.

Set this to 0 to account everything.

Not thread safe - call in initialization only.

func (*RW) Read

func (rw *RW) Read(p []byte) (n int, err error)

Read reads up to len(p) bytes into p. It returns the number of bytes read (0 <= n <= len(p)) and any error encountered. If some data is available but not len(p) bytes, Read returns what is available instead of waiting for more.

func (*RW) ReadFrom

func (rw *RW) ReadFrom(r io.Reader) (n int64, err error)

ReadFrom reads data from r until EOF or error. The return value n is the number of bytes read. Any error except EOF encountered during the read is also returned.

The Copy function uses ReadFrom if available. This avoids an allocation and a copy.

func (*RW) Seek

func (rw *RW) Seek(offset int64, whence int) (int64, error)

Seek sets the offset for the next Read (not Write - this is always appended) to offset, interpreted according to whence: SeekStart means relative to the start of the file, SeekCurrent means relative to the current offset, and SeekEnd means relative to the end (for example, offset = -2 specifies the penultimate byte of the file). Seek returns the new offset relative to the start of the file or an error, if any.

Seeking to an offset before the start of the file is an error. Seeking beyond the end of the written data is an error.

func (*RW) SetAccounting

func (rw *RW) SetAccounting(account RWAccount) *RW

SetAccounting should be provided with a function which will be called after every read from the RW.

It may return an error which will be passed back to the user.

Not thread safe - call in initialization only.

func (*RW) Size

func (rw *RW) Size() int64

Size returns the number of bytes in the buffer

func (*RW) WaitWrite

func (rw *RW) WaitWrite(ctx context.Context)

WaitWrite sleeps until a data is written to the RW or Close is called or the context is cancelled occurs or for a maximum of 1 Second then returns.

This can be used when calling Read while the buffer is filling up.

func (*RW) Write

func (rw *RW) Write(p []byte) (n int, err error)

Write writes len(p) bytes from p to the underlying data stream. It returns the number of bytes written len(p). It cannot return an error.

func (*RW) WriteTo

func (rw *RW) WriteTo(w io.Writer) (n int64, err error)

WriteTo writes data to w until there's no more data to write or when an error occurs. The return value n is the number of bytes written. Any error encountered during the write is also returned.

The Copy function uses WriteTo if available. This avoids an allocation and a copy.

type RWAccount

type RWAccount func(n int) error

RWAccount is a function which will be called after every read from the RW.

It may return an error which will be passed back to the user.