store

package
v1.2.7 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 20, 2021 License: Apache-2.0 Imports: 6 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source
var (
	DefaultProvider                   = Provider(BufferedPaginatedStoreConstructor)
	DenseStoreConstructor             = Provider(func() Store { return NewDenseStore() })
	BufferedPaginatedStoreConstructor = Provider(func() Store { return NewBufferedPaginatedStore() })
	SparseStoreConstructor            = Provider(func() Store { return NewSparseStore() })
)

Functions

func DecodeAndMergeWith

func DecodeAndMergeWith(s Store, b *[]byte, binEncodingMode enc.SubFlag) error

Types

type Bin

type Bin struct {
	// contains filtered or unexported fields
}

func NewBin

func NewBin(index int, count float64) (*Bin, error)

func (Bin) Count

func (b Bin) Count() float64

func (Bin) Index

func (b Bin) Index() int

type BufferedPaginatedStore

type BufferedPaginatedStore struct {
	// contains filtered or unexported fields
}

BufferedPaginatedStore allocates storage for counts in aligned fixed-size pages, themselves stored in a dynamically-sized slice. A page encodes the counts for a contiguous range of indexes, and two pages that are contiguous in the slice encode ranges that are contiguous. In addition, input indexes that are added to the store with a count equal to 1 can be stored in a buffer. The store favors using the buffer and only creates pages when the memory size of the page is no greater than the memory space that is needed to keep in the buffer the indexes that could otherwise be encoded in that page. That means that some indexes may stay indefinitely in the buffer if, to be removed from the buffer, they would create a page that is almost empty. The process that transfers indexes from the buffer to pages is called compaction. This store never collapses or merges bins, therefore, it does not introduce any error in itself. In particular, MinIndex(), MaxIndex(), Bins() and KeyAtRank() return exact results. There is no upper bound on the memory size that this store needs to encode input indexes, and some input data distributions may make it reach large sizes. However, thanks to the buffer and the fact that only required pages are allocated, it can be much more space efficient than alternative stores, especially dense stores, in various situations, including when only few indexes are added (with their counts equal to 1), when the input data has a few outliers or when the input data distribution is multimodal.

func NewBufferedPaginatedStore

func NewBufferedPaginatedStore() *BufferedPaginatedStore

func (*BufferedPaginatedStore) Add

func (s *BufferedPaginatedStore) Add(index int)

func (*BufferedPaginatedStore) AddBin

func (s *BufferedPaginatedStore) AddBin(bin Bin)

func (*BufferedPaginatedStore) AddWithCount

func (s *BufferedPaginatedStore) AddWithCount(index int, count float64)

func (*BufferedPaginatedStore) Bins

func (s *BufferedPaginatedStore) Bins() <-chan Bin

func (*BufferedPaginatedStore) Clear

func (s *BufferedPaginatedStore) Clear()

func (*BufferedPaginatedStore) Copy

func (s *BufferedPaginatedStore) Copy() Store

func (*BufferedPaginatedStore) DecodeAndMergeWith

func (s *BufferedPaginatedStore) DecodeAndMergeWith(b *[]byte, encodingMode enc.SubFlag) error

func (*BufferedPaginatedStore) Encode

func (s *BufferedPaginatedStore) Encode(b *[]byte, t enc.FlagType)

func (*BufferedPaginatedStore) ForEach

func (s *BufferedPaginatedStore) ForEach(f func(index int, count float64) (stop bool))

func (*BufferedPaginatedStore) IsEmpty

func (s *BufferedPaginatedStore) IsEmpty() bool

func (*BufferedPaginatedStore) KeyAtRank

func (s *BufferedPaginatedStore) KeyAtRank(rank float64) int

func (*BufferedPaginatedStore) MaxIndex

func (s *BufferedPaginatedStore) MaxIndex() (int, error)

func (*BufferedPaginatedStore) MergeWith

func (s *BufferedPaginatedStore) MergeWith(other Store)

func (*BufferedPaginatedStore) MinIndex

func (s *BufferedPaginatedStore) MinIndex() (int, error)

func (*BufferedPaginatedStore) Reweight

func (s *BufferedPaginatedStore) Reweight(w float64) error

func (*BufferedPaginatedStore) TotalCount

func (s *BufferedPaginatedStore) TotalCount() float64

type CollapsingHighestDenseStore

type CollapsingHighestDenseStore struct {
	DenseStore
	// contains filtered or unexported fields
}

func NewCollapsingHighestDenseStore

func NewCollapsingHighestDenseStore(maxNumBins int) *CollapsingHighestDenseStore

func (*CollapsingHighestDenseStore) Add

func (s *CollapsingHighestDenseStore) Add(index int)

func (*CollapsingHighestDenseStore) AddBin

func (s *CollapsingHighestDenseStore) AddBin(bin Bin)

func (*CollapsingHighestDenseStore) AddWithCount

func (s *CollapsingHighestDenseStore) AddWithCount(index int, count float64)

func (*CollapsingHighestDenseStore) Clear

func (s *CollapsingHighestDenseStore) Clear()

func (*CollapsingHighestDenseStore) Copy

func (*CollapsingHighestDenseStore) DecodeAndMergeWith

func (s *CollapsingHighestDenseStore) DecodeAndMergeWith(r *[]byte, encodingMode enc.SubFlag) error

func (*CollapsingHighestDenseStore) MergeWith

func (s *CollapsingHighestDenseStore) MergeWith(other Store)

type CollapsingLowestDenseStore

type CollapsingLowestDenseStore struct {
	DenseStore
	// contains filtered or unexported fields
}

CollapsingLowestDenseStore is a dynamically growing contiguous (non-sparse) store. The lower bins get combined so that the total number of bins do not exceed maxNumBins.

func NewCollapsingLowestDenseStore

func NewCollapsingLowestDenseStore(maxNumBins int) *CollapsingLowestDenseStore

func (*CollapsingLowestDenseStore) Add

func (s *CollapsingLowestDenseStore) Add(index int)

func (*CollapsingLowestDenseStore) AddBin

func (s *CollapsingLowestDenseStore) AddBin(bin Bin)

func (*CollapsingLowestDenseStore) AddWithCount

func (s *CollapsingLowestDenseStore) AddWithCount(index int, count float64)

func (*CollapsingLowestDenseStore) Clear

func (s *CollapsingLowestDenseStore) Clear()

func (*CollapsingLowestDenseStore) Copy

func (*CollapsingLowestDenseStore) DecodeAndMergeWith

func (s *CollapsingLowestDenseStore) DecodeAndMergeWith(r *[]byte, encodingMode enc.SubFlag) error

func (*CollapsingLowestDenseStore) MergeWith

func (s *CollapsingLowestDenseStore) MergeWith(other Store)

type DenseStore

type DenseStore struct {
	// contains filtered or unexported fields
}

DenseStore is a dynamically growing contiguous (non-sparse) store. The number of bins are bound only by the size of the slice that can be allocated.

func NewDenseStore

func NewDenseStore() *DenseStore

func (*DenseStore) Add

func (s *DenseStore) Add(index int)

func (*DenseStore) AddBin

func (s *DenseStore) AddBin(bin Bin)

func (*DenseStore) AddWithCount

func (s *DenseStore) AddWithCount(index int, count float64)

func (*DenseStore) Bins

func (s *DenseStore) Bins() <-chan Bin

func (*DenseStore) Clear

func (s *DenseStore) Clear()

func (*DenseStore) Copy

func (s *DenseStore) Copy() Store

func (*DenseStore) DecodeAndMergeWith

func (s *DenseStore) DecodeAndMergeWith(b *[]byte, encodingMode enc.SubFlag) error

func (*DenseStore) Encode

func (s *DenseStore) Encode(b *[]byte, t enc.FlagType)

func (*DenseStore) ForEach

func (s *DenseStore) ForEach(f func(index int, count float64) (stop bool))

func (*DenseStore) IsEmpty

func (s *DenseStore) IsEmpty() bool

func (*DenseStore) KeyAtRank

func (s *DenseStore) KeyAtRank(rank float64) int

Return the key for the value at rank

func (*DenseStore) MaxIndex

func (s *DenseStore) MaxIndex() (int, error)

func (*DenseStore) MergeWith

func (s *DenseStore) MergeWith(other Store)

func (*DenseStore) MinIndex

func (s *DenseStore) MinIndex() (int, error)

func (*DenseStore) Reweight

func (s *DenseStore) Reweight(w float64) error

func (*DenseStore) TotalCount

func (s *DenseStore) TotalCount() float64

type Provider

type Provider func() Store

type SparseStore

type SparseStore struct {
	// contains filtered or unexported fields
}

func NewSparseStore

func NewSparseStore() *SparseStore

func (*SparseStore) Add

func (s *SparseStore) Add(index int)

func (*SparseStore) AddBin

func (s *SparseStore) AddBin(bin Bin)

func (*SparseStore) AddWithCount

func (s *SparseStore) AddWithCount(index int, count float64)

func (*SparseStore) Bins

func (s *SparseStore) Bins() <-chan Bin

func (*SparseStore) Clear

func (s *SparseStore) Clear()

func (*SparseStore) Copy

func (s *SparseStore) Copy() Store

func (*SparseStore) DecodeAndMergeWith

func (s *SparseStore) DecodeAndMergeWith(b *[]byte, encodingMode enc.SubFlag) error

func (*SparseStore) Encode

func (s *SparseStore) Encode(b *[]byte, t enc.FlagType)

func (*SparseStore) ForEach

func (s *SparseStore) ForEach(f func(index int, count float64) (stop bool))

func (*SparseStore) IsEmpty

func (s *SparseStore) IsEmpty() bool

func (*SparseStore) KeyAtRank

func (s *SparseStore) KeyAtRank(rank float64) int

func (*SparseStore) MaxIndex

func (s *SparseStore) MaxIndex() (int, error)

func (*SparseStore) MergeWith

func (s *SparseStore) MergeWith(store Store)

func (*SparseStore) MinIndex

func (s *SparseStore) MinIndex() (int, error)

func (*SparseStore) Reweight

func (s *SparseStore) Reweight(w float64) error

func (*SparseStore) TotalCount

func (s *SparseStore) TotalCount() float64

type Store

type Store interface {
	Add(index int)
	AddBin(bin Bin)
	AddWithCount(index int, count float64)
	// Bins returns a channel that emits the bins that are encoded in the store.
	// Note that this leaks a channel and a goroutine if it is not iterated to completion.
	Bins() <-chan Bin
	// ForEach applies f to all elements of the store or until f returns true.
	ForEach(f func(index int, count float64) (stop bool))
	Copy() Store
	// Clear empties the store while allowing reusing already allocated memory.
	// In some situations, it may be advantageous to clear and reuse a store
	// rather than instantiating a new one. Keeping reusing the same store again
	// and again on varying input data distributions may however ultimately make
	// the store overly large and may waste memory space.
	Clear()
	IsEmpty() bool
	MaxIndex() (int, error)
	MinIndex() (int, error)
	TotalCount() float64
	KeyAtRank(rank float64) int
	// Reweight multiplies all values from the store by w, but keeps the same global distribution.
	Reweight(w float64) error
	// Encode encodes the bins of the store and appends its content to the
	// provided []byte.
	// The provided FlagType indicates whether the store encodes positive or
	// negative values.
	Encode(b *[]byte, t enc.FlagType)
	// DecodeAndMergeWith decodes bins that have been encoded in the format of
	// the provided binEncodingMode and merges them within the receiver store.
	// It updates the provided []byte so that it starts immediately after the
	// encoded bins.
	DecodeAndMergeWith(b *[]byte, binEncodingMode enc.SubFlag) error
}

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL