inverted

package
v1.18.0-rc.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 1, 2023 License: BSD-3-Clause Imports: 40 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ConfigFromModel 

func ConfigFromModel(iicm *models.InvertedIndexConfig) schema.InvertedIndexConfig

func HasFrequency 

func HasFrequency(dt schema.DataType) bool

func LexicographicallySortableFloat64 

func LexicographicallySortableFloat64(in float64) ([]byte, error)

LexicographicallySortableFloat64 transforms a conversion to a lexicographically sortable byte slice. In general, for lexicographical sorting big endian notatino is required. Additionally the sign needs to be flipped in any case, but additionally each remaining byte also needs to be flipped if the number is negative

func LexicographicallySortableInt64 

func LexicographicallySortableInt64(in int64) ([]byte, error)

LexicographicallySortableInt64 performs a conversion to a lexicographically sortable byte slice. For this, big endian notation is required and the sign must be flipped

func LexicographicallySortableUint64 

func LexicographicallySortableUint64(in uint64) ([]byte, error)

LexicographicallySortableUint64 performs a conversion to a lexicographically sortable byte slice. For this, big endian notation is required.

func ParseLexicographicallySortableFloat64 

func ParseLexicographicallySortableFloat64(in []byte) (float64, error)

ParseLexicographicallySortableFloat64 reverses the changes in LexicographicallySortableFloat64

func ParseLexicographicallySortableInt64 

func ParseLexicographicallySortableInt64(in []byte) (int64, error)

ParseLexicographicallySortableInt64 reverses the changes in LexicographicallySortableInt64

func ParseLexicographicallySortableUint64 

func ParseLexicographicallySortableUint64(in []byte) (uint64, error)

ParseLexicographicallySortableUint64 reverses the changes in LexicographicallySortableUint64

func ValidateConfig 

func ValidateConfig(conf *models.InvertedIndexConfig) error

func ValidateUserConfigUpdate 

func ValidateUserConfigUpdate(initial, updated *models.InvertedIndexConfig) error

Types

type Analyzer 

type Analyzer struct {
	// contains filtered or unexported fields
}

func NewAnalyzer 

func NewAnalyzer(stopwords stopwords.StopwordDetector) *Analyzer

func (*Analyzer) Bool 

func (a *Analyzer) Bool(in bool) ([]Countable, error)

Bool requires no analysis, so it's actually just a simple conversion to a little-endian ordered byte slice

func (*Analyzer) BoolArray 

func (a *Analyzer) BoolArray(in []bool) ([]Countable, error)

BoolArray requires no analysis, so it's actually just a simple conversion to a little-endian ordered byte slice

func (*Analyzer) Float 

func (a *Analyzer) Float(in float64) ([]Countable, error)

Float requires no analysis, so it's actually just a simple conversion to a lexicographically sortable byte slice.

func (*Analyzer) FloatArray 

func (a *Analyzer) FloatArray(in []float64) ([]Countable, error)

Float array requires no analysis, so it's actually just a simple conversion to a lexicographically sortable byte slice.

func (*Analyzer) Int 

func (a *Analyzer) Int(in int64) ([]Countable, error)

Int requires no analysis, so it's actually just a simple conversion to a string-formatted byte slice of the int

func (*Analyzer) IntArray 

func (a *Analyzer) IntArray(in []int64) ([]Countable, error)

Int array requires no analysis, so it's actually just a simple conversion to a string-formatted byte slice of the int

func (*Analyzer) Object 

func (a *Analyzer) Object(input map[string]interface{}, props []*models.Property,
	uuid strfmt.UUID,
) ([]Property, error)

func (*Analyzer) Ref 

func (a *Analyzer) Ref(in models.MultipleRef) ([]Countable, error)

Ref indexes references as beacon-strings

func (*Analyzer) RefCount 

func (a *Analyzer) RefCount(in models.MultipleRef) ([]Countable, error)

RefCount does not index the content of the refs, but only the count with 0 being an explicitly allowed value as well.

func (*Analyzer) String 

func (a *Analyzer) String(tokenization, in string) []Countable

String splits only on spaces and does not lowercase, then aggregates duplicates

func (*Analyzer) StringArray 

func (a *Analyzer) StringArray(tokenization string, in []string) []Countable

StringArray splits only on spaces and does not lowercase, then aggregates duplicates

func (*Analyzer) Text 

func (a *Analyzer) Text(tokenization, in string) []Countable

Text removes non alpha-numeric and splits into lowercased words, then aggregates duplicates

func (*Analyzer) TextArray 

func (a *Analyzer) TextArray(tokenization string, in []string) []Countable

TextArray removes non alpha-numeric and splits into lowercased words, then aggregates duplicates

type BM25Searcher 

type BM25Searcher struct {
	// contains filtered or unexported fields
}

func NewBM25Searcher 

func NewBM25Searcher(config schema.BM25Config, store *lsmkv.Store, schema schema.Schema,
	rowCache cacher, propIndices propertyspecific.Indices,
	classSearcher ClassSearcher, deletedDocIDs DeletedDocIDChecker,
	propLengths propLengthRetriever, logger logrus.FieldLogger,
	shardVersion uint16,
) *BM25Searcher

func (*BM25Searcher) BM25F 

func (b *BM25Searcher) BM25F(ctx context.Context, filterDocIds helpers.AllowList, className schema.ClassName, limit int,
	keywordRanking *searchparams.KeywordRanking,
	filter *filters.LocalFilter, sort []filters.Sort, additional additional.Properties,
	objectByIndexID func(index uint64) *storobj.Object,
) ([]*storobj.Object, []float32, error)

func (*BM25Searcher) Objects 

func (b *BM25Searcher) Objects(ctx context.Context, filterDocIds helpers.AllowList, limit int,
	keywordRanking *searchparams.KeywordRanking,
	filter *filters.LocalFilter, sort []filters.Sort, additional additional.Properties,
	className schema.ClassName,
) ([]*storobj.Object, []float32, error)

Objects returns a list of full objects

type CacheEntry 

type CacheEntry struct {
	Hash      []byte
	AllowList helpers.AllowList
}

func (*CacheEntry) Size 

func (ce *CacheEntry) Size() uint64

type ClassSearcher 

type ClassSearcher interface {
	ClassSearch(ctx context.Context,
		params dto.GetParams) ([]search.Result, error)
	GetQueryMaximumResults() int
}

ClassSearcher is anything that allows a root-level ClassSearch

type Cleaner 

type Cleaner struct {
	// contains filtered or unexported fields
}

func NewCleaner 

func NewCleaner(db *bolt.DB, class *models.Class, deletedDocIDs []uint64, deleteFn deleteFn) *Cleaner

func (*Cleaner) Cleanup 

func (c *Cleaner) Cleanup() ([]uint64, error)

Cleanup cleans up properties for given documents

type Countable 

type Countable struct {
	Data          []byte
	TermFrequency float32
}

type DeletedDocIDChecker 

type DeletedDocIDChecker interface {
	Contains(id uint64) bool
}

type DeltaMergeResult 

type DeltaMergeResult struct {
	Additions []MergeProperty
	Deletions []MergeProperty
}

type DeltaMerger 

type DeltaMerger struct {
	// contains filtered or unexported fields
}

DeltaMerger can be used to condense the number of single writes into one big one. Additionally it removes overlaps between additions and deletions. It is meant to be used in batch situation, where 5 ref objects in a row might each increase the doc count by one. Instead of writing 5 additions and 4 deletions, this can be condensed to write just one addition

func NewDeltaMerger 

func NewDeltaMerger() *DeltaMerger

func (*DeltaMerger) AddAdditions 

func (dm *DeltaMerger) AddAdditions(props []Property, docID uint64)

func (*DeltaMerger) AddDeletions 

func (dm *DeltaMerger) AddDeletions(props []Property, docID uint64)

func (*DeltaMerger) Merge 

func (dm *DeltaMerger) Merge() DeltaMergeResult

type DeltaResults 

type DeltaResults struct {
	ToDelete []Property
	ToAdd    []Property
}

func Delta 

func Delta(previous, next []Property) DeltaResults

type MergeDocIDWithFrequency 

type MergeDocIDWithFrequency struct {
	DocID     uint64
	Frequency float32
}

type MergeItem 

type MergeItem struct {
	Data   []byte
	DocIDs []MergeDocIDWithFrequency
}

func (MergeItem) Countable 

func (mi MergeItem) Countable() Countable

Countable converts the merge item to a regular (non-merge) Countable. Note that this loses the IDs and Frequency information, so IDs have to be passed separately using .IDs()

func (MergeItem) IDs 

func (mi MergeItem) IDs() []uint64

IDs is meant for cases such as deletion, where the frequency is irrelevant, but the expected format is a []docID

type MergeProperty 

type MergeProperty struct {
	Name         string
	HasFrequency bool
	MergeItems   []MergeItem
}

type Property 

type Property struct {
	Name         string
	Items        []Countable
	HasFrequency bool
	Length       int
}

type PropertyLengthTracker 

type PropertyLengthTracker struct {
	sync.Mutex
	// contains filtered or unexported fields
}

Page Design | Bytes | Description | | --------- | ------------------------------------------------ | | start | page is now 0 | 0-1 | uint16 pointer to last index byte | 2-3 | uint16 pointer for property name length | 4-n | property name | ... | repeat length+pointer pattern | 3584-3840 | second property buckets (64 buckets of float32) | 3840-4096 | first property buckets | repeat | page is now 1, repeat all of above

Fixed Assumptions:

  • First two bytes always used to indicate end of index, minimal value is 02, as the first possible value with index length=0 is after the two bytes themselves.
  • 64 buckets of float32 per property (=256B per prop), excluding the index
  • One index row is always 4+len(propName), consisting of a uint16 prop name length pointer, the name itself and an offset pointer pointing to the start (first byte) of the buckets

The counter to the last index byte is only an uint16, so it can at maximum address 65535. This will overflow when the 16th page is added (eg at page=15). To avoid a crash an error is returned in this case, but we will need to change the byteformat to fix this.

func NewPropertyLengthTracker 

func NewPropertyLengthTracker(path string) (*PropertyLengthTracker, error)

func (*PropertyLengthTracker) Close 

func (t *PropertyLengthTracker) Close() error

func (*PropertyLengthTracker) Drop 

func (t *PropertyLengthTracker) Drop() error

func (*PropertyLengthTracker) FileName 

func (t *PropertyLengthTracker) FileName() string

func (*PropertyLengthTracker) Flush 

func (t *PropertyLengthTracker) Flush() error

func (*PropertyLengthTracker) PropertyMean 

func (t *PropertyLengthTracker) PropertyMean(propName string) (float32, error)

func (*PropertyLengthTracker) TrackProperty 

func (t *PropertyLengthTracker) TrackProperty(propName string, value float32) error

type ReadFn 

type ReadFn func(k []byte, values [][]byte) (bool, error)

ReadFn will be called 1..n times per match. This means it will also be called on a non-match, in this case v == nil. It is up to the caller to decide if that is an error case or not.

Note that because what we are parsing is an inverted index row, it can sometimes become confusing what a key and value actually resembles. The variables k and v are the literal row key and value. So this means, the data-value as in "less than 17" where 17 would be the "value" is in the key variable "k". The value will contain the docCount, hash and list of pointers (with optional frequency) to the docIDs

The boolean return argument is a way to stop iteration (e.g. when a limit is reached) without producing an error. In normal operation always return true, if false is returned once, the loop is broken.

type ReadFnFrequency 

type ReadFnFrequency func(k []byte, values []lsmkv.MapPair) (bool, error)

ReadFnFrequency will be called 1..n times per match. This means it will also be called on a non-match, in this case v == nil. It is up to the caller to decide if that is an error case or not.

Note that because what we are parsing is an inverted index row, it can sometimes become confusing what a key and value actually resembles. The variables k and v are the literal row key and value. So this means, the data-value as in "less than 17" where 17 would be the "value" is in the key variable "k". The value will contain the docCount, hash and list of pointers (with optional frequency) to the docIDs

The boolean return argument is a way to stop iteration (e.g. when a limit is reached) without producing an error. In normal operation always return true, if false is returned once, the loop is broken.

type RoaringSetReadFn added in v1.18.0 

type RoaringSetReadFn func(k []byte, v *sroar.Bitmap) (bool, error)

RoaringSetReadFn will be called 1..n times per match. This means it will also be called on a non-match, in this case v == empty bitmap. It is up to the caller to decide if that is an error case or not.

Note that because what we are parsing is an inverted index row, it can sometimes become confusing what a key and value actually resembles. The variables k and v are the literal row key and value. So this means, the data-value as in "less than 17" where 17 would be the "value" is in the key variable "k". The value will contain bitmap with docIDs having value "k"

The boolean return argument is a way to stop iteration (e.g. when a limit is reached) without producing an error. In normal operation always return true, if false is returned once, the loop is broken.

type RowCacher 

type RowCacher struct {
	// contains filtered or unexported fields
}

func NewRowCacher 

func NewRowCacher(maxSize uint64) *RowCacher

func (*RowCacher) Load 

func (rc *RowCacher) Load(id []byte) (*CacheEntry, bool)

func (*RowCacher) Size 

func (rc *RowCacher) Size() uint64

func (*RowCacher) Store 

func (rc *RowCacher) Store(id []byte, row *CacheEntry)

type RowReader 

type RowReader struct {
	// contains filtered or unexported fields
}

RowReader reads one or many row(s) depending on the specified operator

func NewRowReader 

func NewRowReader(bucket *lsmkv.Bucket, value []byte,
	operator filters.Operator, keyOnly bool,
) *RowReader

If keyOnly is set, the RowReader will request key-only cursors wherever cursors are used, the specified value arguments in the ReadFn will always be nil

func (*RowReader) Read 

func (rr *RowReader) Read(ctx context.Context, readFn ReadFn) error

Read a row using the specified ReadFn. If RowReader was created with keysOnly==true, the values argument in the readFn will always be nil on all requests involving cursors

type RowReaderFrequency 

type RowReaderFrequency struct {
	// contains filtered or unexported fields
}

RowReaderFrequency reads one or many row(s) depending on the specified operator

func NewRowReaderFrequency 

func NewRowReaderFrequency(bucket *lsmkv.Bucket, value []byte,
	operator filters.Operator, keyOnly bool, shardVersion uint16,
) *RowReaderFrequency

func (*RowReaderFrequency) Read 

func (rr *RowReaderFrequency) Read(ctx context.Context, readFn ReadFnFrequency) error

type RowReaderRoaringSet added in v1.18.0 

type RowReaderRoaringSet struct {
	// contains filtered or unexported fields
}

RowReaderRoaringSet reads one or many row(s) depending on the specified operator

func NewRowReaderRoaringSet added in v1.18.0 

func NewRowReaderRoaringSet(bucket *lsmkv.Bucket, value []byte,
	operator filters.Operator, keyOnly bool,
) *RowReaderRoaringSet

If keyOnly is set, the RowReaderRoaringSet will request key-only cursors wherever cursors are used, the specified value arguments in the RoaringSetReadFn will always be empty

func (*RowReaderRoaringSet) Read added in v1.18.0 

func (rr *RowReaderRoaringSet) Read(ctx context.Context, readFn RoaringSetReadFn) error

Read a row using the specified ReadFn. If RowReader was created with keysOnly==true, the values argument in the readFn will always be nil on all requests involving cursors

type Searcher 

type Searcher struct {
	// contains filtered or unexported fields
}

func NewSearcher 

func NewSearcher(store *lsmkv.Store, schema schema.Schema,
	rowCache cacher, propIndices propertyspecific.Indices,
	classSearcher ClassSearcher, deletedDocIDs DeletedDocIDChecker,
	stopwords stopwords.StopwordDetector, shardVersion uint16,
) *Searcher

func (*Searcher) DocIDs 

func (s *Searcher) DocIDs(ctx context.Context, filter *filters.LocalFilter,
	additional additional.Properties, className schema.ClassName,
) (helpers.AllowList, error)

DocIDs is similar to Objects, but does not actually resolve the docIDs to full objects. Instead it returns the pure object id pointers. They can then be used in a secondary index (e.g. vector index)

DocID queries does not contain a limit by design, as we won't know if the limit wouldn't remove the item that is most important for the follow up query. Imagine the user sets the limit to 1 and the follow-up is a vector search. If we already limited the allowList to 1, the vector search would be pointless, as only the first element would be allowed, regardless of which had the shortest distance

func (*Searcher) DocIDsPreventCaching 

func (s *Searcher) DocIDsPreventCaching(ctx context.Context, filter *filters.LocalFilter,
	additional additional.Properties, className schema.ClassName,
) (helpers.AllowList, error)

DocIDsPreventCaching is the same as DocIDs, but makes sure that no filter cache entries are written. This can be used when we can guarantee that the filter is part of an operation that will lead to a state change, such as batch delete. The state change would make the cached filter unusable anyway, so we don't need to unnecessarily populate the cache with an entry.

func (*Searcher) Objects 

func (s *Searcher) Objects(ctx context.Context, limit int,
	filter *filters.LocalFilter, sort []filters.Sort, additional additional.Properties,
	className schema.ClassName,
) ([]*storobj.Object, error)

Objects returns a list of full objects

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.