db

package
v1.18.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 4, 2023 License: BSD-3-Clause Imports: 65 Imported by: 1

Documentation

Index

Constants

View Source 
const (
	ShardCodeBaseVersion                  = uint16(2)
	ShardCodeBaseMinimumVersionForStartup = uint16(1)
)

ShardCodeBaseVersion must be increased whenever there are breaking changes - including those that we can handle in a non-breaking way the version checker can then decide on init if it should prevent startup completely. If it does not prevent startup, but there is still a version mismatch, the version can be used to make specific decisions

CHANGELOG

  • Version 1 - Everything up until Weaviate v1.10.1 inclusive
  • Version 2 - Inverted Index is now stored in an always sorted fashion and doc ids are stored as BigEndian. To make this backward-compatible with v1, doc ids need to be read and written as Little Endian. In addition, an additional sort step is required in three places: during a MapList call, during a Map Cursor and during Map Compactions. BM25 is entirely disabled prior to this version
View Source 
const IdLockPoolSize = 128

Variables

This section is empty.

Functions

func CheckSupportedPropertyIndexType added in v1.18.0 

func CheckSupportedPropertyIndexType(indexType PropertyIndexType)

func IsIndexTypeSupportedByStrategy added in v1.18.0 

func IsIndexTypeSupportedByStrategy(indexType PropertyIndexType, strategy string) bool

Some index types are supported by specific strategies only Method ensures both index type and strategy work together

func IsSupportedPropertyIndexType added in v1.18.0 

func IsSupportedPropertyIndexType(indexType PropertyIndexType) bool

Types

type BackupState 

type BackupState struct {
	BackupID   string
	InProgress bool
}

type Config 

type Config struct {
	RootPath                  string
	QueryLimit                int64
	QueryMaximumResults       int64
	ResourceUsage             config.ResourceUsage
	MaxImportGoroutinesFactor float64
	MemtablesFlushIdleAfter   int
	MemtablesInitialSizeMB    int
	MemtablesMaxSizeMB        int
	MemtablesMinActiveSeconds int
	MemtablesMaxActiveSeconds int
	TrackVectorDimensions     bool
	ServerVersion             string
	GitHash                   string
}

type DB 

type DB struct {
	// contains filtered or unexported fields
}

func New 

func New(logger logrus.FieldLogger, config Config,
	remoteIndex sharding.RemoteIndexClient, nodeResolver nodeResolver,
	remoteNodesClient sharding.RemoteNodeClient, replicaClient replica.Client,
	promMetrics *monitoring.PrometheusMetrics,
) (*DB, error)

func (*DB) AbortReplication 

func (db *DB) AbortReplication(class,
	shard, requestID string,
) interface{}

func (*DB) AddBatchReferences 

func (db *DB) AddBatchReferences(ctx context.Context, references objects.BatchReferences,
	repl *additional.ReplicationProperties,
) (objects.BatchReferences, error)

func (*DB) AddReference 

func (d *DB) AddReference(ctx context.Context,
	className string, source strfmt.UUID, propName string,
	ref *models.SingleRef, repl *additional.ReplicationProperties,
) error

func (*DB) Aggregate 

func (db *DB) Aggregate(ctx context.Context,
	params aggregation.Params,
) (*aggregation.Result, error)

func (*DB) AggregateNeighbors 

func (db *DB) AggregateNeighbors(ctx context.Context, vector []float32,
	class string, properties []string, k int,
	filter *libfilters.LocalFilter,
) ([]classification.NeighborRef, error)

TODO: why is this logic in the persistence package? This is business-logic, move out of here!

func (*DB) BackupDescriptors 

func (db *DB) BackupDescriptors(ctx context.Context, bakid string, classes []string,
) <-chan backup.ClassDescriptor

BackupDescriptors returns a channel of class descriptors. Class descriptor records everything needed to restore a class If an error happens a descriptor with an error will be written to the channel just before closing it.

func (*DB) Backupable 

func (db *DB) Backupable(ctx context.Context, classes []string) error

Backupable returns whether all given class can be backed up.

func (*DB) BatchDeleteObjects 

func (db *DB) BatchDeleteObjects(ctx context.Context, params objects.BatchDeleteParams,
	repl *additional.ReplicationProperties,
) (objects.BatchDeleteResult, error)

func (*DB) BatchPutObjects 

func (db *DB) BatchPutObjects(ctx context.Context, objects objects.BatchObjects,
	repl *additional.ReplicationProperties,
) (objects.BatchObjects, error)

func (*DB) ClassExists 

func (db *DB) ClassExists(name string) bool

func (*DB) ClassObjectSearch 

func (db *DB) ClassObjectSearch(ctx context.Context,
	params dto.GetParams,
) ([]*storobj.Object, []float32, error)

ClassObjectSearch is used to perform an inverted index search on the db

Earlier use cases required only []search.Result as a return value from the db, and the Class ClassSearch method fit this need. Later on, other use cases presented the need for the raw storage objects, such as hybrid search.

func (*DB) ClassObjectVectorSearch 

func (db *DB) ClassObjectVectorSearch(ctx context.Context, class string, vector []float32, offset, limit int,
	filters *filters.LocalFilter,
) ([]*storobj.Object, []float32, error)

ClassObjectVectorSearch is used to perform a vector search on the db

Earlier use cases required only []search.Result as a return value from the db, and the Class VectorSearch method fit this need. Later on, other use cases presented the need for the raw storage objects, such as hybrid search.

func (*DB) ClassSearch 

func (db *DB) ClassSearch(ctx context.Context,
	params dto.GetParams,
) ([]search.Result, error)

func (*DB) ClassVectorSearch 

func (db *DB) ClassVectorSearch(ctx context.Context, class string, vector []float32, offset, limit int,
	filters *filters.LocalFilter,
) ([]search.Result, error)

func (*DB) CommitReplication 

func (db *DB) CommitReplication(class,
	shard, requestID string,
) interface{}

func (*DB) DeleteIndex 

func (d *DB) DeleteIndex(className schema.ClassName) error

DeleteIndex deletes the index

func (*DB) DeleteObject 

func (d *DB) DeleteObject(ctx context.Context, class string,
	id strfmt.UUID, repl *additional.ReplicationProperties,
) error

DeleteObject from of a specific class giving its ID

func (*DB) DigestObjects added in v1.18.0 

func (db *DB) DigestObjects(ctx context.Context,
	class, shardName string, ids []strfmt.UUID,
) (result []replica.RepairResponse, err error)

func (*DB) Exists 

func (d *DB) Exists(ctx context.Context, class string,
	id strfmt.UUID, repl *additional.ReplicationProperties,
) (bool, error)

func (*DB) FetchObject added in v1.18.0 

func (db *DB) FetchObject(ctx context.Context,
	class, shardName string, id strfmt.UUID,
) (objects.Replica, error)

func (*DB) FetchObjects added in v1.18.0 

func (db *DB) FetchObjects(ctx context.Context,
	class, shardName string, ids []strfmt.UUID,
) ([]objects.Replica, error)

func (*DB) GetIndex 

func (d *DB) GetIndex(className schema.ClassName) *Index

GetIndex returns the index if it exists or nil if it doesn't

func (*DB) GetIndexForIncoming 

func (d *DB) GetIndexForIncoming(className schema.ClassName) sharding.RemoteIndexIncomingRepo

GetIndexForIncoming returns the index if it exists or nil if it doesn't

func (*DB) GetNodeStatuses 

func (db *DB) GetNodeStatuses(ctx context.Context) ([]*models.NodeStatus, error)

GetNodeStatuses returns the status of all Weaviate nodes.

func (*DB) GetQueryMaximumResults 

func (db *DB) GetQueryMaximumResults() int

func (*DB) GetUnclassified 

func (db *DB) GetUnclassified(ctx context.Context, class string,
	properties []string, filter *libfilters.LocalFilter,
) ([]search.Result, error)

TODO: why is this logic in the persistence package? This is business-logic, move out of here!

func (*DB) IncomingGetNodeStatus 

func (db *DB) IncomingGetNodeStatus(ctx context.Context) (*models.NodeStatus, error)

IncomingGetNodeStatus returns the index if it exists or nil if it doesn't

func (*DB) ListBackupable 

func (db *DB) ListBackupable() []string

ListBackupable returns a list of all classes which can be backed up.

func (*DB) ListClasses 

func (db *DB) ListClasses(ctx context.Context) []string

func (*DB) Merge 

func (d *DB) Merge(ctx context.Context, merge objects.MergeDocument,
	repl *additional.ReplicationProperties,
) error

func (*DB) MultiGet 

func (d *DB) MultiGet(ctx context.Context,
	query []multi.Identifier,
	additional additional.Properties,
) ([]search.Result, error)

func (*DB) Object 

func (d *DB) Object(ctx context.Context, class string,
	id strfmt.UUID, props search.SelectProperties,
	adds additional.Properties, repl *additional.ReplicationProperties,
) (*search.Result, error)

Object gets object with id from index of specified class.

func (*DB) ObjectByID 

func (d *DB) ObjectByID(ctx context.Context, id strfmt.UUID,
	props search.SelectProperties,
	additional additional.Properties,
) (*search.Result, error)

ObjectByID checks every index of the particular kind for the ID

@warning: this function is deprecated by Object()

func (*DB) ObjectSearch 

func (d *DB) ObjectSearch(ctx context.Context, offset, limit int,
	filters *filters.LocalFilter, sort []filters.Sort,
	additional additional.Properties,
) (search.Results, error)

ObjectSearch search each index. Deprecated by Query which searches a specific index

func (*DB) ObjectsByID 

func (d *DB) ObjectsByID(ctx context.Context, id strfmt.UUID,
	props search.SelectProperties,
	additional additional.Properties,
) (search.Results, error)

ObjectsByID checks every index of the particular kind for the ID this method is only used for Explore queries where we don't have a class context

func (*DB) OverwriteObjects added in v1.18.0 

func (db *DB) OverwriteObjects(ctx context.Context,
	class, shardName string, vobjects []*objects.VObject,
) ([]replica.RepairResponse, error)

func (*DB) PutObject 

func (d *DB) PutObject(ctx context.Context, obj *models.Object,
	vector []float32, repl *additional.ReplicationProperties,
) error

func (*DB) Query 

func (d *DB) Query(ctx context.Context, q *objects.QueryInput) (search.Results, *objects.Error)

Query a specific class

func (*DB) ReleaseBackup 

func (db *DB) ReleaseBackup(ctx context.Context, bakID, class string) error

ReleaseBackup release resources acquired by the index during backup

func (*DB) ReplicateDeletion 

func (db *DB) ReplicateDeletion(ctx context.Context, class,
	shard, requestID string, uuid strfmt.UUID,
) replica.SimpleResponse

func (*DB) ReplicateDeletions 

func (db *DB) ReplicateDeletions(ctx context.Context, class,
	shard, requestID string, docIDs []uint64, dryRun bool,
) replica.SimpleResponse

func (*DB) ReplicateObject 

func (db *DB) ReplicateObject(ctx context.Context, class,
	shard, requestID string, object *storobj.Object,
) replica.SimpleResponse

func (*DB) ReplicateObjects 

func (db *DB) ReplicateObjects(ctx context.Context, class,
	shard, requestID string, objects []*storobj.Object,
) replica.SimpleResponse

func (*DB) ReplicateReferences 

func (db *DB) ReplicateReferences(ctx context.Context, class,
	shard, requestID string, refs []objects.BatchReference,
) replica.SimpleResponse

func (*DB) ReplicateUpdate 

func (db *DB) ReplicateUpdate(ctx context.Context, class,
	shard, requestID string, mergeDoc *objects.MergeDocument,
) replica.SimpleResponse

func (*DB) ResolveReferences 

func (d *DB) ResolveReferences(ctx context.Context, objs search.Results,
	props search.SelectProperties, additional additional.Properties,
) (search.Results, error)

ResolveReferences takes a list of search results and enriches them with any referenced objects

func (*DB) SetSchemaGetter 

func (d *DB) SetSchemaGetter(sg schemaUC.SchemaGetter)

func (*DB) Shards 

func (db *DB) Shards(ctx context.Context, class string) []string

func (*DB) ShardsBackup 

func (db *DB) ShardsBackup(
	ctx context.Context, bakID, class string, shards []string,
) (_ backup.ClassDescriptor, err error)

func (*DB) Shutdown 

func (d *DB) Shutdown(ctx context.Context) error

func (*DB) StartupComplete 

func (d *DB) StartupComplete() bool

func (*DB) VectorClassSearch 

func (db *DB) VectorClassSearch(ctx context.Context,
	params dto.GetParams,
) ([]search.Result, error)

func (*DB) VectorSearch 

func (db *DB) VectorSearch(ctx context.Context, vector []float32, offset, limit int,
	filters *filters.LocalFilter,
) ([]search.Result, error)

func (*DB) WaitForStartup 

func (d *DB) WaitForStartup(ctx context.Context) error

func (*DB) ZeroShotSearch 

func (db *DB) ZeroShotSearch(ctx context.Context, vector []float32,
	class string, properties []string,
	filter *libfilters.LocalFilter,
) ([]search.Result, error)

TODO: why is this logic in the persistence package? This is business-logic, move out of here!

type Index 

type Index struct {
	Shards map[string]*Shard
	Config IndexConfig
	// contains filtered or unexported fields
}

Index is the logical unit which contains all the data for one particular class. An index can be further broken up into self-contained units, called Shards, to allow for easy distribution across Nodes

func NewIndex 

func NewIndex(ctx context.Context, config IndexConfig,
	shardState *sharding.State, invertedIndexConfig schema.InvertedIndexConfig,
	vectorIndexUserConfig schema.VectorIndexConfig, sg schemaUC.SchemaGetter,
	cs inverted.ClassSearcher, logger logrus.FieldLogger,
	nodeResolver nodeResolver, remoteClient sharding.RemoteIndexClient,
	replicaClient replica.Client,
	promMetrics *monitoring.PrometheusMetrics, class *models.Class, jobQueueCh chan job,
) (*Index, error)

NewIndex creates an index with the specified amount of shards, using only the shards that are local to a node

func (*Index) AbortReplication 

func (i *Index) AbortReplication(shard, requestID string) interface{}

func (*Index) CommitReplication 

func (i *Index) CommitReplication(shard, requestID string) interface{}

func (*Index) ID 

func (i *Index) ID() string

func (*Index) IncomingAggregate 

func (i *Index) IncomingAggregate(ctx context.Context, shardName string,
	params aggregation.Params,
) (*aggregation.Result, error)

func (*Index) IncomingBatchAddReferences 

func (i *Index) IncomingBatchAddReferences(ctx context.Context, shardName string,
	refs objects.BatchReferences,
) []error

func (*Index) IncomingBatchPutObjects 

func (i *Index) IncomingBatchPutObjects(ctx context.Context, shardName string,
	objects []*storobj.Object,
) []error

func (*Index) IncomingCreateShard 

func (i *Index) IncomingCreateShard(ctx context.Context,
	shardName string,
) error

func (*Index) IncomingDeleteObject 

func (i *Index) IncomingDeleteObject(ctx context.Context, shardName string,
	id strfmt.UUID,
) error

func (*Index) IncomingDeleteObjectBatch 

func (i *Index) IncomingDeleteObjectBatch(ctx context.Context, shardName string,
	docIDs []uint64, dryRun bool,
) objects.BatchSimpleObjects

func (*Index) IncomingDigestObjects added in v1.18.0 

func (i *Index) IncomingDigestObjects(ctx context.Context,
	shardName string, ids []strfmt.UUID,
) (result []replica.RepairResponse, err error)

func (*Index) IncomingExists 

func (i *Index) IncomingExists(ctx context.Context, shardName string,
	id strfmt.UUID,
) (bool, error)

func (*Index) IncomingFilePutter 

func (i *Index) IncomingFilePutter(ctx context.Context, shardName,
	filePath string,
) (io.WriteCloser, error)

func (*Index) IncomingFindDocIDs 

func (i *Index) IncomingFindDocIDs(ctx context.Context, shardName string,
	filters *filters.LocalFilter,
) ([]uint64, error)

func (*Index) IncomingGetObject 

func (i *Index) IncomingGetObject(ctx context.Context, shardName string,
	id strfmt.UUID, props search.SelectProperties,
	additional additional.Properties,
) (*storobj.Object, error)

func (*Index) IncomingGetShardStatus 

func (i *Index) IncomingGetShardStatus(ctx context.Context, shardName string) (string, error)

func (*Index) IncomingMergeObject 

func (i *Index) IncomingMergeObject(ctx context.Context, shardName string,
	mergeDoc objects.MergeDocument,
) error

func (*Index) IncomingMultiGetObjects 

func (i *Index) IncomingMultiGetObjects(ctx context.Context, shardName string,
	ids []strfmt.UUID,
) ([]*storobj.Object, error)

func (*Index) IncomingOverwriteObjects added in v1.18.0 

func (i *Index) IncomingOverwriteObjects(ctx context.Context,
	shardName string, vobjects []*objects.VObject,
) ([]replica.RepairResponse, error)

func (*Index) IncomingPutObject 

func (i *Index) IncomingPutObject(ctx context.Context, shardName string,
	object *storobj.Object,
) error

func (*Index) IncomingReinitShard 

func (i *Index) IncomingReinitShard(ctx context.Context,
	shardName string,
) error

func (*Index) IncomingSearch 

func (i *Index) IncomingSearch(ctx context.Context, shardName string,
	searchVector []float32, distance float32, limit int, filters *filters.LocalFilter,
	keywordRanking *searchparams.KeywordRanking, sort []filters.Sort,
	cursor *filters.Cursor, additional additional.Properties,
) ([]*storobj.Object, []float32, error)

func (*Index) IncomingUpdateShardStatus 

func (i *Index) IncomingUpdateShardStatus(ctx context.Context, shardName, targetStatus string) error

func (*Index) IterateObjects 

func (i *Index) IterateObjects(ctx context.Context, cb func(index *Index, shard *Shard, object *storobj.Object) error) error

func (*Index) ReleaseBackup 

func (i *Index) ReleaseBackup(ctx context.Context, id string) error

ReleaseBackup marks the specified backup as inactive and restarts all async background and maintenance processes. It errors if the backup does not exist or is already inactive.

func (*Index) ReplicateDeletion 

func (i *Index) ReplicateDeletion(ctx context.Context, shard, requestID string, uuid strfmt.UUID) replica.SimpleResponse

func (*Index) ReplicateDeletions 

func (i *Index) ReplicateDeletions(ctx context.Context, shard, requestID string, docIDs []uint64, dryRun bool) replica.SimpleResponse

func (*Index) ReplicateObject 

func (i *Index) ReplicateObject(ctx context.Context, shard, requestID string, object *storobj.Object) replica.SimpleResponse

func (*Index) ReplicateObjects 

func (i *Index) ReplicateObjects(ctx context.Context, shard, requestID string, objects []*storobj.Object) replica.SimpleResponse

func (*Index) ReplicateReferences 

func (i *Index) ReplicateReferences(ctx context.Context, shard, requestID string, refs []objects.BatchReference) replica.SimpleResponse

func (*Index) ReplicateUpdate 

func (i *Index) ReplicateUpdate(ctx context.Context, shard, requestID string, doc *objects.MergeDocument) replica.SimpleResponse

func (*Index) Shutdown 

func (i *Index) Shutdown(ctx context.Context) error

type IndexConfig 

type IndexConfig struct {
	RootPath                  string
	ClassName                 schema.ClassName
	QueryMaximumResults       int64
	ResourceUsage             config.ResourceUsage
	MemtablesFlushIdleAfter   int
	MemtablesInitialSizeMB    int
	MemtablesMaxSizeMB        int
	MemtablesMinActiveSeconds int
	MemtablesMaxActiveSeconds int
	ReplicationFactor         int64

	TrackVectorDimensions bool
}

type KnnAggregator 

type KnnAggregator struct {
	// contains filtered or unexported fields
}

TODO: this is business logic, move out of here

func NewKnnAggregator 

func NewKnnAggregator(input search.Results, sourceVector []float32) *KnnAggregator

func (*KnnAggregator) Aggregate 

func (a *KnnAggregator) Aggregate(k int, properties []string) ([]classification.NeighborRef, error)

type Metrics 

type Metrics struct {
	// contains filtered or unexported fields
}

func NewMetrics 

func NewMetrics(logger logrus.FieldLogger, prom *monitoring.PrometheusMetrics,
	className, shardName string,
) *Metrics

func (*Metrics) BatchDelete 

func (m *Metrics) BatchDelete(start time.Time, op string)

func (*Metrics) BatchObject 

func (m *Metrics) BatchObject(start time.Time, size int)

func (*Metrics) FilteredVectorFilter added in v1.18.0 

func (m *Metrics) FilteredVectorFilter(dur time.Duration)

func (*Metrics) FilteredVectorObjects added in v1.18.0 

func (m *Metrics) FilteredVectorObjects(dur time.Duration)

func (*Metrics) FilteredVectorSort added in v1.18.0 

func (m *Metrics) FilteredVectorSort(dur time.Duration)

func (*Metrics) FilteredVectorVector added in v1.18.0 

func (m *Metrics) FilteredVectorVector(dur time.Duration)

func (*Metrics) InvertedDeleteDelta 

func (m *Metrics) InvertedDeleteDelta(start time.Time)

func (*Metrics) InvertedDeleteOld 

func (m *Metrics) InvertedDeleteOld(start time.Time)

func (*Metrics) InvertedExtend 

func (m *Metrics) InvertedExtend(start time.Time, propCount int)

func (*Metrics) ObjectStore 

func (m *Metrics) ObjectStore(start time.Time)

func (*Metrics) PutObject 

func (m *Metrics) PutObject(start time.Time)

func (*Metrics) PutObjectDetermineStatus 

func (m *Metrics) PutObjectDetermineStatus(start time.Time)

func (*Metrics) PutObjectUpdateInverted 

func (m *Metrics) PutObjectUpdateInverted(start time.Time)

func (*Metrics) PutObjectUpsertObject 

func (m *Metrics) PutObjectUpsertObject(start time.Time)

func (*Metrics) ShardStartup 

func (m *Metrics) ShardStartup(start time.Time)

func (*Metrics) VectorIndex 

func (m *Metrics) VectorIndex(start time.Time)

type Migrator 

type Migrator struct {
	// contains filtered or unexported fields
}

func NewMigrator 

func NewMigrator(db *DB, logger logrus.FieldLogger) *Migrator

func (*Migrator) AddClass 

func (m *Migrator) AddClass(ctx context.Context, class *models.Class,
	shardState *sharding.State,
) error

func (*Migrator) AddProperty 

func (m *Migrator) AddProperty(ctx context.Context, className string, prop *models.Property) error

func (*Migrator) DropClass 

func (m *Migrator) DropClass(ctx context.Context, className string) error

func (*Migrator) DropProperty 

func (m *Migrator) DropProperty(ctx context.Context, className string, propertyName string) error

DropProperty is ignored, API compliant change

func (*Migrator) GetShardsStatus 

func (m *Migrator) GetShardsStatus(ctx context.Context, className string) (map[string]string, error)

func (*Migrator) InvertedReindex added in v1.18.0 

func (m *Migrator) InvertedReindex(ctx context.Context, taskNames ...string) error

func (*Migrator) RecalculateVectorDimensions 

func (m *Migrator) RecalculateVectorDimensions(ctx context.Context) error

func (*Migrator) UpdateClass 

func (m *Migrator) UpdateClass(ctx context.Context, className string, newClassName *string) error

func (*Migrator) UpdateInvertedIndexConfig 

func (m *Migrator) UpdateInvertedIndexConfig(ctx context.Context, className string,
	updated *models.InvertedIndexConfig,
) error

func (*Migrator) UpdateProperty 

func (m *Migrator) UpdateProperty(ctx context.Context, className string, propName string, newName *string) error

func (*Migrator) UpdateShardStatus 

func (m *Migrator) UpdateShardStatus(ctx context.Context, className, shardName, targetStatus string) error

func (*Migrator) UpdateVectorIndexConfig 

func (m *Migrator) UpdateVectorIndexConfig(ctx context.Context,
	className string, updated schema.VectorIndexConfig,
) error

func (*Migrator) ValidateInvertedIndexConfigUpdate 

func (m *Migrator) ValidateInvertedIndexConfigUpdate(ctx context.Context,
	old, updated *models.InvertedIndexConfig,
) error

func (*Migrator) ValidateVectorIndexConfigUpdate 

func (m *Migrator) ValidateVectorIndexConfigUpdate(ctx context.Context,
	old, updated schema.VectorIndexConfig,
) error

type PropertyIndexType added in v1.18.0 

type PropertyIndexType uint8
const (
	IndexTypePropValue PropertyIndexType = iota + 1
	IndexTypePropLength
	IndexTypePropNull
	IndexTypeHashPropValue
	IndexTypeHashPropLength
	IndexTypeHashPropNull
)

func GetPropNameAndIndexTypeFromBucketName added in v1.18.0 

func GetPropNameAndIndexTypeFromBucketName(bucketName string) (string, PropertyIndexType)

type ReindexableProperty added in v1.18.0 

type ReindexableProperty struct {
	PropertyName    string
	IndexType       PropertyIndexType
	DesiredStrategy string
	BucketOptions   []lsmkv.BucketOption
}

type Replicator 

type Replicator interface {
	ReplicateObject(ctx context.Context, shardName, requestID string,
		object *storobj.Object) replica.SimpleResponse
	ReplicateObjects(ctx context.Context, shardName, requestID string,
		objects []*storobj.Object) replica.SimpleResponse
	ReplicateUpdate(ctx context.Context, shard, requestID string,
		doc *objects.MergeDocument) replica.SimpleResponse
	ReplicateDeletion(ctx context.Context, shardName, requestID string,
		uuid strfmt.UUID) replica.SimpleResponse
	ReplicateDeletions(ctx context.Context, shardName, requestID string,
		docIDs []uint64, dryRun bool) replica.SimpleResponse
	ReplicateReferences(ctx context.Context, shard, requestID string,
		refs []objects.BatchReference) replica.SimpleResponse
	CommitReplication(shard,
		requestID string) interface{}
	AbortReplication(shardName,
		requestID string) interface{}
}

type Shard 

type Shard struct {
	// contains filtered or unexported fields
}

Shard is the smallest completely-contained index unit. A shard manages database files for all the objects it owns. How a shard is determined for a target object (e.g. Murmur hash, etc.) is still open at this point

func NewShard 

func NewShard(ctx context.Context, promMetrics *monitoring.PrometheusMetrics,
	shardName string, index *Index, class *models.Class, jobQueueCh chan job,
) (*Shard, error)

func (*Shard) DBPathLSM 

func (s *Shard) DBPathLSM() string

func (*Shard) Dimensions 

func (s *Shard) Dimensions() int

func (*Shard) ID 

func (s *Shard) ID() string

type ShardInvertedReindexTask added in v1.18.0 

type ShardInvertedReindexTask interface {
	GetPropertiesToReindex(ctx context.Context, store *lsmkv.Store, indexConfig IndexConfig,
		invertedIndexConfig schema.InvertedIndexConfig, logger logrus.FieldLogger,
	) ([]ReindexableProperty, error)
}

type ShardInvertedReindexTaskSetToRoaringSet added in v1.18.0 

type ShardInvertedReindexTaskSetToRoaringSet struct{}

func (*ShardInvertedReindexTaskSetToRoaringSet) GetPropertiesToReindex added in v1.18.0 

func (t *ShardInvertedReindexTaskSetToRoaringSet) GetPropertiesToReindex(ctx context.Context,
	store *lsmkv.Store, indexConfig IndexConfig, invertedIndexConfig schema.InvertedIndexConfig,
	logger logrus.FieldLogger,
) ([]ReindexableProperty, error)

type ShardInvertedReindexer added in v1.18.0 

type ShardInvertedReindexer struct {
	// contains filtered or unexported fields
}

func NewShardInvertedReindexer added in v1.18.0 

func NewShardInvertedReindexer(shard *Shard, logger logrus.FieldLogger) *ShardInvertedReindexer

func (*ShardInvertedReindexer) AddTask added in v1.18.0 

func (r *ShardInvertedReindexer) AddTask(task ShardInvertedReindexTask)

func (*ShardInvertedReindexer) Do added in v1.18.0 

func (r *ShardInvertedReindexer) Do(ctx context.Context) error

type VectorIndex 

type VectorIndex interface {
	Dump(labels ...string)
	Add(id uint64, vector []float32) error
	Delete(id ...uint64) error
	SearchByVector(vector []float32, k int, allow helpers.AllowList) ([]uint64, []float32, error)
	SearchByVectorDistance(vector []float32, dist float32,
		maxLimit int64, allow helpers.AllowList) ([]uint64, []float32, error)
	UpdateUserConfig(updated schema.VectorIndexConfig, callback func()) error
	Drop(ctx context.Context) error
	Shutdown(ctx context.Context) error
	Flush() error
	PauseMaintenance(ctx context.Context) error
	SwitchCommitLogs(ctx context.Context) error
	ListFiles(ctx context.Context) ([]string, error)
	ResumeMaintenance(ctx context.Context) error
	PostStartup()
	ValidateBeforeInsert(vector []float32) error
}

VectorIndex is anything that indexes vectors efficiently. For an example look at ./vector/hnsw/index.go

Source Files

View all Source files

Directories

Path Synopsis
clusterintegrationtest acts as a test package to provide a compoenent test spanning multiple parts of the application, including everything that's required for a distributed setup.
 clusterintegrationtest acts as a test package to provide a compoenent test spanning multiple parts of the application, including everything that's required for a distributed setup.
stopwords
entities
ent contains common types used throughout various lsmkv (sub-)packages
 ent contains common types used throughout various lsmkv (sub-)packages
rbtree
roaringset
The "roaringset" package contains all the LSM business logic that is unique to the "RoaringSet" strategy
 The "roaringset" package contains all the LSM business logic that is unique to the "RoaringSet" strategy
segmentindex
vector
geo
hnsw
hnsw/commitlog
hnsw/distancer
hnsw/distancer/asm
asm only has amd64 specific implementations at the moment
 asm only has amd64 specific implementations at the moment
hnsw/priorityqueue
hnsw/visited
noop
ssdhelpers
testinghelpers

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.