README
¶
Buffer 
This package contains several buffer types used in https://github.com/tdewolff/parse for example.
Installation
Run the following command
go get github.com/tdewolff/buffer
or add the following import and run the project with go get
import "github.com/tdewolff/buffer"
Reader
Reader is a wrapper around a []byte that implements the io.Reader interface. It is a much thinner layer than bytes.Buffer provides and is therefore faster.
Writer
Writer is a buffer that implements the io.Writer interface. It is a much thinner layer than bytes.Buffer provides and is therefore faster. It will expand the buffer when needed.
The reset functionality allows for better memory reuse. After calling Reset, it will overwrite the current buffer and thus reduce allocations.
Lexer
Lexer is a read buffer specifically designed for building lexers. It keeps track of two positions: a start and end position. The start position is the beginning of the current token being parsed, the end position is being moved forward until a valid token is found. Calling Shift will collapse the positions to the end and return the parsed []byte.
Moving the end position can go through Move(int) which also accepts negative integers. One can also use Pos() int to try and parse a token, and if it fails rewind with Rewind(int), passing the previously saved position.
Peek(int) byte will peek forward (relative to the end position) and return the byte at that location. PeekRune(int) (rune, int) returns UTF-8 runes and its length at the given byte position. Upon an error Peek will return 0, the user must peek at every character and not skip any, otherwise it may skip a 0 and panic on out-of-bounds indexing.
Lexeme() []byte will return the currently selected bytes, Skip() will collapse the selection. Shift() []byte is a combination of Lexeme() []byte and Skip().
When the passed io.Reader returned an error, Err() error will return that error even if not at the end of the buffer.
StreamLexer
StreamLexer behaves like Lexer but uses a buffer pool to read in chunks from io.Reader, retaining old buffers in memory that are still in use, and re-using old buffers otherwise. Calling Free(n int) frees up n bytes from the internal buffer(s). It holds an array of buffers to accommodate for keeping everything in-memory. Calling ShiftLen() int returns the number of bytes that have been shifted since the previous call to ShiftLen, which can be used to specify how many bytes need to be freed up from the buffer. If you don't need to keep returned byte slices around, call Free(ShiftLen()) after every Shift call.
License
Released under the MIT license.
Documentation
¶
Overview ¶
Package buffer contains buffer and wrapper types for byte slices. It is useful for writing lexers or other high-performance byte slice handling.
The `Reader` and `Writer` types implement the `io.Reader` and `io.Writer` respectively and provide a thinner and faster interface than `bytes.Buffer`. The `Shifter` type is useful for building lexers because it keeps track of the start and end position of a byte selection, and shifts the bytes whenever a valid token is found. The `Lexer` is however an improved version of `Shifter`, allowing zero-copy for the parser by using a (kind of) ring buffer underneath.
Index ¶
- Variables
- type Lexer
- func (z *Lexer) Bytes() []byte
- func (z *Lexer) Err() error
- func (z *Lexer) Lexeme() []byte
- func (z *Lexer) Move(n int)
- func (z *Lexer) Offset() int
- func (z *Lexer) Peek(pos int) byte
- func (z *Lexer) PeekRune(pos int) (rune, int)
- func (z *Lexer) Pos() int
- func (z *Lexer) Restore()
- func (z *Lexer) Rewind(pos int)
- func (z *Lexer) Shift() []byte
- func (z *Lexer) Skip()
- type Reader
- type StreamLexer
- func (z *StreamLexer) Err() error
- func (z *StreamLexer) Free(n int)
- func (z *StreamLexer) Lexeme() []byte
- func (z *StreamLexer) Move(n int)
- func (z *StreamLexer) Peek(pos int) byte
- func (z *StreamLexer) PeekRune(pos int) (rune, int)
- func (z *StreamLexer) Pos() int
- func (z *StreamLexer) Rewind(pos int)
- func (z *StreamLexer) Shift() []byte
- func (z *StreamLexer) ShiftLen() int
- func (z *StreamLexer) Skip()
- type Writer
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var MinBuf = defaultBufSize
MinBuf specifies the default initial length of internal buffers. Solely here to support old versions of parse.
Functions ¶
This section is empty.
Types ¶
type Lexer ¶
type Lexer struct {
// contains filtered or unexported fields
}
Lexer is a buffered reader that allows peeking forward and shifting, taking an io.Reader. It keeps data in-memory until Free, taking a byte length, is called to move beyond the data.
func NewLexer ¶
NewLexerBytes returns a new Lexer for a given io.Reader, and uses ioutil.ReadAll to read it into a byte slice. If the io.Reader implements Bytes, that is used instead. It will append a NULL at the end of the buffer.
func NewLexerBytes ¶
NewLexerBytes returns a new Lexer for a given byte slice, and appends NULL at the end. To avoid reallocation, make sure the capacity has room for one more byte.
func (*Lexer) Err ¶
Err returns the error returned from io.Reader or io.EOF when the end has been reached.
func (*Lexer) Peek ¶
Peek returns the ith byte relative to the end position. Peek returns 0 when an error has occurred, Err returns the error.
func (*Lexer) PeekRune ¶
PeekRune returns the rune and rune length of the ith byte relative to the end position.
func (*Lexer) Restore ¶
func (z *Lexer) Restore()
Restore restores the replaced byte past the end of the buffer by NULL.
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader implements an io.Reader over a byte slice.
func NewReader ¶
NewReader returns a new Reader for a given byte slice.
Example ¶
r := NewReader([]byte("Lorem ipsum"))
w := &bytes.Buffer{}
io.Copy(w, r)
fmt.Println(w.String())
Output: Lorem ipsum
type StreamLexer ¶
type StreamLexer struct {
// contains filtered or unexported fields
}
StreamLexer is a buffered reader that allows peeking forward and shifting, taking an io.Reader. It keeps data in-memory until Free, taking a byte length, is called to move beyond the data.
func NewStreamLexer ¶
func NewStreamLexer(r io.Reader) *StreamLexer
NewStreamLexer returns a new StreamLexer for a given io.Reader with a 4kB estimated buffer size. If the io.Reader implements Bytes, that buffer is used instead.
func NewStreamLexerSize ¶
func NewStreamLexerSize(r io.Reader, size int) *StreamLexer
NewStreamLexerSize returns a new StreamLexer for a given io.Reader and estimated required buffer size. If the io.Reader implements Bytes, that buffer is used instead.
func (*StreamLexer) Err ¶
func (z *StreamLexer) Err() error
Err returns the error returned from io.Reader. It may still return valid bytes for a while though.
func (*StreamLexer) Free ¶
func (z *StreamLexer) Free(n int)
Free frees up bytes of length n from previously shifted tokens. Each call to Shift should at one point be followed by a call to Free with a length returned by ShiftLen.
func (*StreamLexer) Lexeme ¶
func (z *StreamLexer) Lexeme() []byte
Lexeme returns the bytes of the current selection.
func (*StreamLexer) Peek ¶
func (z *StreamLexer) Peek(pos int) byte
Peek returns the ith byte relative to the end position and possibly does an allocation. Peek returns zero when an error has occurred, Err returns the error. TODO: inline function
func (*StreamLexer) PeekRune ¶
func (z *StreamLexer) PeekRune(pos int) (rune, int)
PeekRune returns the rune and rune length of the ith byte relative to the end position.
func (*StreamLexer) Pos ¶
func (z *StreamLexer) Pos() int
Pos returns a mark to which can be rewinded.
func (*StreamLexer) Rewind ¶
func (z *StreamLexer) Rewind(pos int)
Rewind rewinds the position to the given position.
func (*StreamLexer) Shift ¶
func (z *StreamLexer) Shift() []byte
Shift returns the bytes of the current selection and collapses the position to the end of the selection. It also returns the number of bytes we moved since the last call to Shift. This can be used in calls to Free.
func (*StreamLexer) ShiftLen ¶
func (z *StreamLexer) ShiftLen() int
ShiftLen returns the number of bytes moved since the last call to ShiftLen. This can be used in calls to Free because it takes into account multiple Shifts or Skips.
func (*StreamLexer) Skip ¶
func (z *StreamLexer) Skip()
Skip collapses the position to the end of the selection.
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer implements an io.Writer over a byte slice.
func NewWriter ¶
NewWriter returns a new Writer for a given byte slice.
Example ¶
w := NewWriter(make([]byte, 0, 11)) // initial buffer length is 11
w.Write([]byte("Lorem ipsum"))
fmt.Println(string(w.Bytes()))
Output: Lorem ipsum
func (*Writer) Reset ¶
func (w *Writer) Reset()
Reset empties and reuses the current buffer. Subsequent writes will overwrite the buffer, so any reference to the underlying slice is invalidated after this call.
Example ¶
w := NewWriter(make([]byte, 0, 11)) // initial buffer length is 10
w.Write([]byte("garbage that will be overwritten")) // does reallocation
w.Reset()
w.Write([]byte("Lorem ipsum"))
fmt.Println(string(w.Bytes()))
Output: Lorem ipsum
Source Files