recordbuffer

Documentation

Overview

Package recordbuffer adds support for sorting large number of records by temporarly storing them in a byte store. This sorthing method is meant for when the size of the sorting will at some point exceed the available ram and disk-spooling will be needed. It is significantly slower than iterator.NewBufferedRecordIteratorBTree or iterator.NewBufferedClusterIteartor since recordbuffer will both JSON-marshal; Write; Read and Unmarshal the record before being returned. It is slower but more scalable as it can be backed by RAM (bytesbuffer), Disk (e.g. https://github.com/ShoshinNikita/go-disk-buffer/) or GCS/S3 by writing a simple wrapper for thier APIs.

Index

• type ReadWriteResetter
• type ReadWriteResetterFactory
• type SortedRecordBuffers
  • func NewSortedRecordBuffers(factory ReadWriteResetterFactory, newer func() iterator.Lesser) *SortedRecordBuffers
  • func (srb *SortedRecordBuffers) AddLesser(nextVal iterator.Lesser) (int, error)
  • func (srb *SortedRecordBuffers) AddRecord(record interface{}) (int, error)
  • func (srb *SortedRecordBuffers) GetSortedIterator() (iterator.LesserIterator, error)
  • func (srb *SortedRecordBuffers) LoadFromLesserIterator(it iterator.LesserIterator) (int, error)
  • func (srb *SortedRecordBuffers) LoadFromRecordIterator(it iterator.RecordIterator) (int, error)

Types

type ReadWriteResetter

type ReadWriteResetter interface {
	Read(p []byte) (n int, err error)
	Write(p []byte) (n int, err error)
	Reset()
}

ReadWriteResetter is the bare minimum needed from the bytes.Buffer interface to implement a record cache

type ReadWriteResetterFactory

type ReadWriteResetterFactory func() ReadWriteResetter

ReadWriteResetterFactory should return a RecordBuffer. The simplest implementation of this factory is func(_ []byte) *bytes.Buffer {return &bytes.NewBuffer{}} but theoretically (not yet tested) wrapping https://github.com/ShoshinNikita/go-disk-buffer should also do.

type SortedRecordBuffers

type SortedRecordBuffers struct {
	// contains filtered or unexported fields
}

SortedRecordBuffers creates a simple LSM like buffer for records and uses the ReadWriteResetterFactory for byte storage of the records

func NewSortedRecordBuffers

func NewSortedRecordBuffers(factory ReadWriteResetterFactory, newer func() iterator.Lesser) *SortedRecordBuffers

NewSortedRecordBuffers creates a new SortedRecordBuffer using the underlying ReadWriteResetterFactory

func (*SortedRecordBuffers) AddLesser

func (srb *SortedRecordBuffers) AddLesser(nextVal iterator.Lesser) (int, error)

AddLesser will add a single record to the buffer

func (*SortedRecordBuffers) AddRecord

func (srb *SortedRecordBuffers) AddRecord(record interface{}) (int, error)

AddRecord will add a single record to the buffer

func (*SortedRecordBuffers) GetSortedIterator

func (srb *SortedRecordBuffers) GetSortedIterator() (iterator.LesserIterator, error)

GetSortedIterator returns an (sorted) interator for all the records stored in the buffer

func (*SortedRecordBuffers) LoadFromLesserIterator

func (srb *SortedRecordBuffers) LoadFromLesserIterator(it iterator.LesserIterator) (int, error)

LoadFromLesserIterator will add a single record to the buffer

func (*SortedRecordBuffers) LoadFromRecordIterator

func (srb *SortedRecordBuffers) LoadFromRecordIterator(it iterator.RecordIterator) (int, error)

LoadFromRecordIterator populates the buffer through a record iterator.