README
¶
bufpool
Maintained version of the original repo
bufpool is an implementation of a pool of byte buffers with anti-memory-waste protection. It is based on the code and ideas from these 2 projects:
bufpool consists of global pool of buffers that have a capacity of a power of 2 starting from 64 bytes to 32 megabytes. It also provides individual pools that maintain usage stats to provide buffers of the size that satisfies 95% of the calls. Global pool is used to reuse buffers between different parts of the app.
Installation
go get github.com/johnbenedictyan/bufpool
Usage
bufpool can be used as a replacement for sync.Pool:
var jsonPool bufpool.Pool // basically sync.Pool with usage stats
func writeJSON(w io.Writer, obj interface{}) error {
buf := jsonPool.Get()
defer jsonPool.Put(buf)
if err := json.NewEncoder(buf).Encode(obj); err != nil {
return err
}
_, err := w.Write(buf.Bytes())
return err
}
or to allocate buffer of the given size:
func writeHex(w io.Writer, data []byte) error {
n := hex.EncodedLen(len(data)))
buf := bufpool.Get(n) // buf.Len() is guaranteed to equal n
defer bufpool.Put(buf)
tmp := buf.Bytes()
hex.Encode(tmp, data)
_, err := w.Write(tmp)
return err
}
If you need to append data to the buffer you can use following pattern:
buf := bufpool.Get(n)
defer bufpool.Put(buf)
bb := buf.Bytes()[:0]
bb = append(bb, ...)
buf.ResetBuf(bb)
You can also change default pool thresholds:
var jsonPool = bufpool.Pool{
ServePctile: 0.95, // serve p95 buffers
}
Documentation
¶
Index ¶
- func Put(buf *Buffer)
- type Buffer
- func (b *Buffer) Bytes() []byte
- func (b *Buffer) Cap() int
- func (b *Buffer) Grow(n int)
- func (b *Buffer) Len() int
- func (b *Buffer) Next(n int) []byte
- func (b *Buffer) Read(p []byte) (n int, err error)
- func (b *Buffer) ReadByte() (byte, error)
- func (b *Buffer) ReadBytes(delim byte) (line []byte, err error)
- func (b *Buffer) ReadFrom(r io.Reader) (n int64, err error)
- func (b *Buffer) ReadRune() (r rune, size int, err error)
- func (b *Buffer) ReadString(delim byte) (line string, err error)
- func (b *Buffer) Reset()
- func (b *Buffer) ResetBuf(buf []byte)
- func (b *Buffer) String() string
- func (b *Buffer) Truncate(n int)
- func (b *Buffer) UnreadByte() error
- func (b *Buffer) UnreadRune() error
- func (b *Buffer) Write(p []byte) (n int, err error)
- func (b *Buffer) WriteByte(c byte) error
- func (b *Buffer) WriteRune(r rune) (n int, err error)
- func (b *Buffer) WriteString(s string) (n int, err error)
- func (b *Buffer) WriteTo(w io.Writer) (n int64, err error)
- type Pool
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Buffer ¶
type Buffer struct {
// contains filtered or unexported fields
}
A Buffer is a variable-sized buffer of bytes with Read and Write methods. The zero value for Buffer is an empty buffer ready to use.
func Get ¶
Get retrieves a buffer of the appropriate length from the buffer pool or allocates a new one. Get may choose to ignore the pool and treat it as empty. Callers should not assume any relation between values passed to Put and the values returned by Get.
If no suitable buffer exists in the pool, Get creates one.
Example ¶
package main
import (
"fmt"
"github.com/johnbenedictyan/bufpool"
)
func main() {
buf := bufpool.Get(1234)
fmt.Println("len", buf.Len())
bufpool.Put(buf)
}
Output: len 1234
func NewBuffer ¶
NewBuffer creates and initializes a new Buffer using buf as its initial contents. The new Buffer takes ownership of buf, and the caller should not use buf after this call. NewBuffer is intended to prepare a Buffer to read existing data. It can also be used to set the initial size of the internal buffer for writing. To do that, buf should have the desired capacity but a length of zero.
In most cases, new(Buffer) (or just declaring a Buffer variable) is sufficient to initialize a Buffer.
func NewBufferString ¶
NewBufferString creates and initializes a new Buffer using string s as its initial contents. It is intended to prepare a buffer to read an existing string.
In most cases, new(Buffer) (or just declaring a Buffer variable) is sufficient to initialize a Buffer.
func (*Buffer) Bytes ¶
Bytes returns a slice of length b.Len() holding the unread portion of the buffer. The slice is valid for use only until the next buffer modification (that is, only until the next call to a method like Read, Write, Reset, or Truncate). The slice aliases the buffer content at least until the next buffer modification, so immediate changes to the slice will affect the result of future reads.
func (*Buffer) Cap ¶
Cap returns the capacity of the buffer's underlying byte slice, that is, the total space allocated for the buffer's data.
func (*Buffer) Grow ¶
Grow grows the buffer's capacity, if necessary, to guarantee space for another n bytes. After Grow(n), at least n bytes can be written to the buffer without another allocation. If n is negative, Grow will panic. If the buffer can't grow it will panic with ErrTooLarge.
func (*Buffer) Len ¶
Len returns the number of bytes of the unread portion of the buffer; b.Len() == len(b.Bytes()).
func (*Buffer) Next ¶
Next returns a slice containing the next n bytes from the buffer, advancing the buffer as if the bytes had been returned by Read. If there are fewer than n bytes in the buffer, Next returns the entire buffer. The slice is only valid until the next call to a read or write method.
func (*Buffer) Read ¶
Read reads the next len(p) bytes from the buffer or until the buffer is drained. The return value n is the number of bytes read. If the buffer has no data to return, err is io.EOF (unless len(p) is zero); otherwise it is nil.
func (*Buffer) ReadByte ¶
ReadByte reads and returns the next byte from the buffer. If no byte is available, it returns error io.EOF.
func (*Buffer) ReadBytes ¶
ReadBytes reads until the first occurrence of delim in the input, returning a slice containing the data up to and including the delimiter. If ReadBytes encounters an error before finding a delimiter, it returns the data read before the error and the error itself (often io.EOF). ReadBytes returns err != nil if and only if the returned data does not end in delim.
func (*Buffer) ReadFrom ¶
ReadFrom reads data from r until EOF and appends it to the buffer, growing the buffer as needed. The return value n is the number of bytes read. Any error except io.EOF encountered during the read is also returned. If the buffer becomes too large, ReadFrom will panic with ErrTooLarge.
func (*Buffer) ReadRune ¶
ReadRune reads and returns the next UTF-8-encoded Unicode code point from the buffer. If no bytes are available, the error returned is io.EOF. If the bytes are an erroneous UTF-8 encoding, it consumes one byte and returns U+FFFD, 1.
func (*Buffer) ReadString ¶
ReadString reads until the first occurrence of delim in the input, returning a string containing the data up to and including the delimiter. If ReadString encounters an error before finding a delimiter, it returns the data read before the error and the error itself (often io.EOF). ReadString returns err != nil if and only if the returned data does not end in delim.
func (*Buffer) Reset ¶
func (b *Buffer) Reset()
Reset resets the buffer to be empty, but it retains the underlying storage for use by future writes. Reset is the same as Truncate(0).
func (*Buffer) String ¶
String returns the contents of the unread portion of the buffer as a string. If the Buffer is a nil pointer, it returns "<nil>".
To build strings more efficiently, see the strings.Builder type.
func (*Buffer) Truncate ¶
Truncate discards all but the first n unread bytes from the buffer but continues to use the same allocated storage. It panics if n is negative or greater than the length of the buffer.
func (*Buffer) UnreadByte ¶
UnreadByte unreads the last byte returned by the most recent successful read operation that read at least one byte. If a write has happened since the last read, if the last read returned an error, or if the read read zero bytes, UnreadByte returns an error.
func (*Buffer) UnreadRune ¶
UnreadRune unreads the last rune returned by ReadRune. If the most recent read or write operation on the buffer was not a successful ReadRune, UnreadRune returns an error. (In this regard it is stricter than UnreadByte, which will unread the last byte from any read operation.)
func (*Buffer) Write ¶
Write appends the contents of p to the buffer, growing the buffer as needed. The return value n is the length of p; err is always nil. If the buffer becomes too large, Write will panic with ErrTooLarge.
func (*Buffer) WriteByte ¶
WriteByte appends the byte c to the buffer, growing the buffer as needed. The returned error is always nil, but is included to match bufio.Writer's WriteByte. If the buffer becomes too large, WriteByte will panic with ErrTooLarge.
func (*Buffer) WriteRune ¶
WriteRune appends the UTF-8 encoding of Unicode code point r to the buffer, returning its length and an error, which is always nil but is included to match bufio.Writer's WriteRune. The buffer is grown as needed; if it becomes too large, WriteRune will panic with ErrTooLarge.
func (*Buffer) WriteString ¶
WriteString appends the contents of s to the buffer, growing the buffer as needed. The return value n is the length of s; err is always nil. If the buffer becomes too large, WriteString will panic with ErrTooLarge.
type Pool ¶
type Pool struct {
ServePctile float64 // default is 0.95
// contains filtered or unexported fields
}
Pool represents byte buffer pool.
Different pools should be used for different usage patterns to achieve better performance and lower memory usage.
Example ¶
package main
import (
"fmt"
"github.com/johnbenedictyan/bufpool"
)
var jsonPool bufpool.Pool
func main() {
const avgPayloadSize = 1000
for i := 0; i < 100000; i++ {
buf := jsonPool.Get()
buf.Reset()
_, _ = buf.Write(make([]byte, avgPayloadSize))
jsonPool.Put(buf)
}
buf := jsonPool.Get()
fmt.Println("len", buf.Cap() >= 1000 && buf.Cap() <= 1200)
jsonPool.Put(buf)
}
Output: len true
func (*Pool) Get ¶
Get returns an empty buffer from the pool. Returned buffer capacity is determined by accumulated usage stats and changes over time.
The buffer may be returned to the pool using Put or retained for further usage. In latter case buffer length must be updated using UpdateLen.
Source Files