db

package
v1.21.9 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 1, 2023 License: BSD-3-Clause Imports: 69 Imported by: 1

Documentation

Index

Constants

View Source 
const (
	ShardCodeBaseVersion                  = uint16(2)
	ShardCodeBaseMinimumVersionForStartup = uint16(1)
)

ShardCodeBaseVersion must be increased whenever there are breaking changes - including those that we can handle in a non-breaking way the version checker can then decide on init if it should prevent startup completely. If it does not prevent startup, but there is still a version mismatch, the version can be used to make specific decisions

CHANGELOG

  • Version 1 - Everything up until Weaviate v1.10.1 inclusive
  • Version 2 - Inverted Index is now stored in an always sorted fashion and doc ids are stored as BigEndian. To make this backward-compatible with v1, doc ids need to be read and written as Little Endian. In addition, an additional sort step is required in three places: during a MapList call, during a Map Cursor and during Map Compactions. BM25 is entirely disabled prior to this version
View Source 
const IdLockPoolSize = 128

Variables

This section is empty.

Functions

This section is empty.

Types

type BackupState 

type BackupState struct {
	BackupID   string
	InProgress bool
}

type Config 

type Config struct {
	RootPath                  string
	QueryLimit                int64
	QueryMaximumResults       int64
	QueryNestedRefLimit       int64
	ResourceUsage             config.ResourceUsage
	MaxImportGoroutinesFactor float64
	MemtablesFlushIdleAfter   int
	MemtablesInitialSizeMB    int
	MemtablesMaxSizeMB        int
	MemtablesMinActiveSeconds int
	MemtablesMaxActiveSeconds int
	TrackVectorDimensions     bool
	ServerVersion             string
	GitHash                   string
	AvoidMMap                 bool
	Replication               replication.GlobalConfig
}

type DB 

type DB struct {
	// contains filtered or unexported fields
}

func New 

func New(logger logrus.FieldLogger, config Config,
	remoteIndex sharding.RemoteIndexClient, nodeResolver nodeResolver,
	remoteNodesClient sharding.RemoteNodeClient, replicaClient replica.Client,
	promMetrics *monitoring.PrometheusMetrics,
) (*DB, error)

func (*DB) AbortReplication 

func (db *DB) AbortReplication(class,
	shard, requestID string,
) interface{}

func (*DB) AddBatchReferences 

func (db *DB) AddBatchReferences(ctx context.Context, references objects.BatchReferences,
	repl *additional.ReplicationProperties,
) (objects.BatchReferences, error)

func (*DB) AddReference 

func (db *DB) AddReference(ctx context.Context, source *crossref.RefSource, target *crossref.Ref,
	repl *additional.ReplicationProperties, tenant string,
) error

func (*DB) Aggregate 

func (db *DB) Aggregate(ctx context.Context,
	params aggregation.Params,
) (*aggregation.Result, error)

func (*DB) AggregateNeighbors 

func (db *DB) AggregateNeighbors(ctx context.Context, vector []float32,
	class string, properties []string, k int,
	filter *libfilters.LocalFilter,
) ([]classification.NeighborRef, error)

TODO: why is this logic in the persistence package? This is business-logic, move out of here!

func (*DB) BackupDescriptors 

func (db *DB) BackupDescriptors(ctx context.Context, bakid string, classes []string,
) <-chan backup.ClassDescriptor

BackupDescriptors returns a channel of class descriptors. Class descriptor records everything needed to restore a class If an error happens a descriptor with an error will be written to the channel just before closing it.

func (*DB) Backupable 

func (db *DB) Backupable(ctx context.Context, classes []string) error

Backupable returns whether all given class can be backed up.

func (*DB) BatchDeleteObjects 

func (db *DB) BatchDeleteObjects(ctx context.Context, params objects.BatchDeleteParams,
	repl *additional.ReplicationProperties, tenant string,
) (objects.BatchDeleteResult, error)

func (*DB) BatchPutObjects 

func (db *DB) BatchPutObjects(ctx context.Context, objs objects.BatchObjects,
	repl *additional.ReplicationProperties,
) (objects.BatchObjects, error)

func (*DB) ClassExists 

func (db *DB) ClassExists(name string) bool

func (*DB) CommitReplication 

func (db *DB) CommitReplication(class,
	shard, requestID string,
) interface{}

func (*DB) CrossClassVectorSearch added in v1.20.0 

func (db *DB) CrossClassVectorSearch(ctx context.Context, vector []float32, offset, limit int,
	filters *filters.LocalFilter,
) ([]search.Result, error)

func (*DB) DeleteIndex 

func (db *DB) DeleteIndex(className schema.ClassName) error

DeleteIndex deletes the index

func (*DB) DeleteObject 

func (db *DB) DeleteObject(ctx context.Context, class string, id strfmt.UUID,
	repl *additional.ReplicationProperties, tenant string,
) error

DeleteObject from of a specific class giving its ID

func (*DB) DenseObjectSearch added in v1.20.0 

func (db *DB) DenseObjectSearch(ctx context.Context, class string, vector []float32,
	offset int, limit int, filters *filters.LocalFilter, addl additional.Properties,
	tenant string,
) ([]*storobj.Object, []float32, error)

DenseObjectSearch is used to perform a vector search on the db

Earlier use cases required only []search.Result as a return value from the db, and the Class VectorSearch method fit this need. Later on, other use cases presented the need for the raw storage objects, such as hybrid search.

func (*DB) DigestObjects added in v1.18.0 

func (db *DB) DigestObjects(ctx context.Context,
	class, shardName string, ids []strfmt.UUID,
) (result []replica.RepairResponse, err error)

func (*DB) Exists 

func (db *DB) Exists(ctx context.Context, class string, id strfmt.UUID,
	repl *additional.ReplicationProperties, tenant string,
) (bool, error)

func (*DB) FetchObject added in v1.18.0 

func (db *DB) FetchObject(ctx context.Context,
	class, shardName string, id strfmt.UUID,
) (objects.Replica, error)

func (*DB) FetchObjects added in v1.18.0 

func (db *DB) FetchObjects(ctx context.Context,
	class, shardName string, ids []strfmt.UUID,
) ([]objects.Replica, error)

func (*DB) GetIndex 

func (db *DB) GetIndex(className schema.ClassName) *Index

GetIndex returns the index if it exists or nil if it doesn't

func (*DB) GetIndexForIncoming 

func (db *DB) GetIndexForIncoming(className schema.ClassName) sharding.RemoteIndexIncomingRepo

GetIndexForIncoming returns the index if it exists or nil if it doesn't

func (*DB) GetNodeStatus added in v1.20.0 

func (db *DB) GetNodeStatus(ctx context.Context, className string) ([]*models.NodeStatus, error)

GetNodeStatus returns the status of all Weaviate nodes.

func (*DB) GetQueryMaximumResults 

func (db *DB) GetQueryMaximumResults() int

func (*DB) GetUnclassified 

func (db *DB) GetUnclassified(ctx context.Context, class string,
	properties []string, filter *libfilters.LocalFilter,
) ([]search.Result, error)

TODO: why is this logic in the persistence package? This is business-logic, move out of here!

func (*DB) IncomingGetNodeStatus 

func (db *DB) IncomingGetNodeStatus(ctx context.Context, className string) (*models.NodeStatus, error)

IncomingGetNodeStatus returns the index if it exists or nil if it doesn't

func (*DB) IndexExists added in v1.18.5 

func (d *DB) IndexExists(className schema.ClassName) bool

IndexExists returns if an index exists

func (*DB) ListBackupable 

func (db *DB) ListBackupable() []string

ListBackupable returns a list of all classes which can be backed up.

func (*DB) ListClasses 

func (db *DB) ListClasses(ctx context.Context) []string

func (*DB) Merge 

func (db *DB) Merge(ctx context.Context, merge objects.MergeDocument,
	repl *additional.ReplicationProperties, tenant string,
) error

func (*DB) MultiGet 

func (db *DB) MultiGet(ctx context.Context, query []multi.Identifier,
	additional additional.Properties, tenant string,
) ([]search.Result, error)

func (*DB) Object 

func (db *DB) Object(ctx context.Context, class string, id strfmt.UUID,
	props search.SelectProperties, addl additional.Properties,
	repl *additional.ReplicationProperties, tenant string,
) (*search.Result, error)

Object gets object with id from index of specified class.

func (*DB) ObjectByID 

func (db *DB) ObjectByID(ctx context.Context, id strfmt.UUID,
	props search.SelectProperties, additional additional.Properties,
	tenant string,
) (*search.Result, error)

ObjectByID checks every index of the particular kind for the ID

@warning: this function is deprecated by Object()

func (*DB) ObjectSearch 

func (db *DB) ObjectSearch(ctx context.Context, offset, limit int,
	filters *filters.LocalFilter, sort []filters.Sort,
	additional additional.Properties, tenant string,
) (search.Results, error)

ObjectSearch search each index. Deprecated by Query which searches a specific index

func (*DB) ObjectsByID 

func (db *DB) ObjectsByID(ctx context.Context, id strfmt.UUID,
	props search.SelectProperties, additional additional.Properties,
	tenant string,
) (search.Results, error)

ObjectsByID checks every index of the particular kind for the ID this method is only used for Explore queries where we don't have a class context

func (*DB) OverwriteObjects added in v1.18.0 

func (db *DB) OverwriteObjects(ctx context.Context,
	class, shardName string, vobjects []*objects.VObject,
) ([]replica.RepairResponse, error)

func (*DB) PutObject 

func (db *DB) PutObject(ctx context.Context, obj *models.Object,
	vector []float32, repl *additional.ReplicationProperties,
) error

func (*DB) Query 

func (db *DB) Query(ctx context.Context, q *objects.QueryInput) (search.Results, *objects.Error)

Query a specific class

func (*DB) ReleaseBackup 

func (db *DB) ReleaseBackup(ctx context.Context, bakID, class string) (err error)

ReleaseBackup release resources acquired by the index during backup

func (*DB) ReplicateDeletion 

func (db *DB) ReplicateDeletion(ctx context.Context, class,
	shard, requestID string, uuid strfmt.UUID,
) replica.SimpleResponse

func (*DB) ReplicateDeletions 

func (db *DB) ReplicateDeletions(ctx context.Context, class,
	shard, requestID string, docIDs []uint64, dryRun bool,
) replica.SimpleResponse

func (*DB) ReplicateObject 

func (db *DB) ReplicateObject(ctx context.Context, class,
	shard, requestID string, object *storobj.Object,
) replica.SimpleResponse

func (*DB) ReplicateObjects 

func (db *DB) ReplicateObjects(ctx context.Context, class,
	shard, requestID string, objects []*storobj.Object,
) replica.SimpleResponse

func (*DB) ReplicateReferences 

func (db *DB) ReplicateReferences(ctx context.Context, class,
	shard, requestID string, refs []objects.BatchReference,
) replica.SimpleResponse

func (*DB) ReplicateUpdate 

func (db *DB) ReplicateUpdate(ctx context.Context, class,
	shard, requestID string, mergeDoc *objects.MergeDocument,
) replica.SimpleResponse

func (*DB) ResolveReferences 

func (db *DB) ResolveReferences(ctx context.Context, objs search.Results,
	props search.SelectProperties, groupBy *searchparams.GroupBy,
	addl additional.Properties, tenant string,
) (search.Results, error)

ResolveReferences takes a list of search results and enriches them with any referenced objects

func (*DB) Search added in v1.20.0 

func (db *DB) Search(ctx context.Context, params dto.GetParams) ([]search.Result, error)

func (*DB) SetSchemaGetter 

func (db *DB) SetSchemaGetter(sg schemaUC.SchemaGetter)

func (*DB) Shards 

func (db *DB) Shards(ctx context.Context, class string) ([]string, error)

Returns the list of nodes where shards of class are contained. If there are no shards for the class, returns an empty list If there are shards for the class but no nodes are found, return an error

func (*DB) ShardsBackup 

func (db *DB) ShardsBackup(
	ctx context.Context, bakID, class string, shards []string,
) (_ backup.ClassDescriptor, err error)

func (*DB) Shutdown 

func (db *DB) Shutdown(ctx context.Context) error

func (*DB) SparseObjectSearch added in v1.20.0 

func (db *DB) SparseObjectSearch(ctx context.Context, params dto.GetParams) ([]*storobj.Object, []float32, error)

SparseObjectSearch is used to perform an inverted index search on the db

Earlier use cases required only []search.Result as a return value from the db, and the Class ClassSearch method fit this need. Later on, other use cases presented the need for the raw storage objects, such as hybrid search.

func (*DB) StartupComplete 

func (db *DB) StartupComplete() bool

func (*DB) VectorSearch 

func (db *DB) VectorSearch(ctx context.Context,
	params dto.GetParams,
) ([]search.Result, error)

func (*DB) WaitForStartup 

func (db *DB) WaitForStartup(ctx context.Context) error

func (*DB) ZeroShotSearch 

func (db *DB) ZeroShotSearch(ctx context.Context, vector []float32,
	class string, properties []string,
	filter *libfilters.LocalFilter,
) ([]search.Result, error)

TODO: why is this logic in the persistence package? This is business-logic, move out of here!

type Index 

type Index struct {
	Config IndexConfig
	// contains filtered or unexported fields
}

Index is the logical unit which contains all the data for one particular class. An index can be further broken up into self-contained units, called Shards, to allow for easy distribution across Nodes

func NewIndex 

func NewIndex(ctx context.Context, cfg IndexConfig,
	shardState *sharding.State, invertedIndexConfig schema.InvertedIndexConfig,
	vectorIndexUserConfig schema.VectorIndexConfig, sg schemaUC.SchemaGetter,
	cs inverted.ClassSearcher, logger logrus.FieldLogger,
	nodeResolver nodeResolver, remoteClient sharding.RemoteIndexClient,
	replicaClient replica.Client,
	promMetrics *monitoring.PrometheusMetrics, class *models.Class, jobQueueCh chan job,
) (*Index, error)

NewIndex creates an index with the specified amount of shards, using only the shards that are local to a node

func (*Index) AbortReplication 

func (i *Index) AbortReplication(shard, requestID string) interface{}

func (*Index) CommitReplication 

func (i *Index) CommitReplication(shard, requestID string) interface{}

func (*Index) ForEachShard added in v1.19.6 

func (i *Index) ForEachShard(f func(name string, shard *Shard) error) error

func (*Index) ForEachShardConcurrently added in v1.21.8 

func (i *Index) ForEachShardConcurrently(f func(name string, shard *Shard) error) error

func (*Index) ID 

func (i *Index) ID() string

func (*Index) IncomingAggregate 

func (i *Index) IncomingAggregate(ctx context.Context, shardName string,
	params aggregation.Params,
) (*aggregation.Result, error)

func (*Index) IncomingBatchAddReferences 

func (i *Index) IncomingBatchAddReferences(ctx context.Context, shardName string,
	refs objects.BatchReferences,
) []error

func (*Index) IncomingBatchPutObjects 

func (i *Index) IncomingBatchPutObjects(ctx context.Context, shardName string,
	objects []*storobj.Object,
) []error

func (*Index) IncomingCreateShard 

func (i *Index) IncomingCreateShard(ctx context.Context,
	shardName string,
) error

func (*Index) IncomingDeleteObject 

func (i *Index) IncomingDeleteObject(ctx context.Context, shardName string,
	id strfmt.UUID,
) error

func (*Index) IncomingDeleteObjectBatch 

func (i *Index) IncomingDeleteObjectBatch(ctx context.Context, shardName string,
	docIDs []uint64, dryRun bool,
) objects.BatchSimpleObjects

func (*Index) IncomingDigestObjects added in v1.18.0 

func (i *Index) IncomingDigestObjects(ctx context.Context,
	shardName string, ids []strfmt.UUID,
) (result []replica.RepairResponse, err error)

func (*Index) IncomingExists 

func (i *Index) IncomingExists(ctx context.Context, shardName string,
	id strfmt.UUID,
) (bool, error)

func (*Index) IncomingFilePutter 

func (i *Index) IncomingFilePutter(ctx context.Context, shardName,
	filePath string,
) (io.WriteCloser, error)

func (*Index) IncomingFindDocIDs 

func (i *Index) IncomingFindDocIDs(ctx context.Context, shardName string,
	filters *filters.LocalFilter,
) ([]uint64, error)

func (*Index) IncomingGetObject 

func (i *Index) IncomingGetObject(ctx context.Context, shardName string,
	id strfmt.UUID, props search.SelectProperties,
	additional additional.Properties,
) (*storobj.Object, error)

func (*Index) IncomingGetShardStatus 

func (i *Index) IncomingGetShardStatus(ctx context.Context, shardName string) (string, error)

func (*Index) IncomingMergeObject 

func (i *Index) IncomingMergeObject(ctx context.Context, shardName string,
	mergeDoc objects.MergeDocument,
) error

func (*Index) IncomingMultiGetObjects 

func (i *Index) IncomingMultiGetObjects(ctx context.Context, shardName string,
	ids []strfmt.UUID,
) ([]*storobj.Object, error)

func (*Index) IncomingOverwriteObjects added in v1.18.0 

func (i *Index) IncomingOverwriteObjects(ctx context.Context,
	shardName string, vobjects []*objects.VObject,
) ([]replica.RepairResponse, error)

func (*Index) IncomingPutObject 

func (i *Index) IncomingPutObject(ctx context.Context, shardName string,
	object *storobj.Object,
) error

func (*Index) IncomingReinitShard 

func (i *Index) IncomingReinitShard(ctx context.Context,
	shardName string,
) error

func (*Index) IncomingSearch 

func (i *Index) IncomingSearch(ctx context.Context, shardName string,
	searchVector []float32, distance float32, limit int, filters *filters.LocalFilter,
	keywordRanking *searchparams.KeywordRanking, sort []filters.Sort,
	cursor *filters.Cursor, groupBy *searchparams.GroupBy,
	additional additional.Properties,
) ([]*storobj.Object, []float32, error)

func (*Index) IncomingUpdateShardStatus 

func (i *Index) IncomingUpdateShardStatus(ctx context.Context, shardName, targetStatus string) error

func (*Index) IterateObjects 

func (i *Index) IterateObjects(ctx context.Context, cb func(index *Index, shard *Shard, object *storobj.Object) error) (err error)

Iterate over all objects in the index, applying the callback function to each one. Adding or removing objects during iteration is not supported.

func (*Index) IterateShards added in v1.19.0 

func (i *Index) IterateShards(ctx context.Context, cb func(index *Index, shard *Shard) error) (err error)

Iterate over all objects in the shard, applying the callback function to each one. Adding or removing objects during iteration is not supported.

func (*Index) ReleaseBackup 

func (i *Index) ReleaseBackup(ctx context.Context, id string) error

ReleaseBackup marks the specified backup as inactive and restarts all async background and maintenance processes. It errors if the backup does not exist or is already inactive.

func (*Index) ReplicateDeletion 

func (i *Index) ReplicateDeletion(ctx context.Context, shard, requestID string, uuid strfmt.UUID) replica.SimpleResponse

func (*Index) ReplicateDeletions 

func (i *Index) ReplicateDeletions(ctx context.Context, shard, requestID string, docIDs []uint64, dryRun bool) replica.SimpleResponse

func (*Index) ReplicateObject 

func (i *Index) ReplicateObject(ctx context.Context, shard, requestID string, object *storobj.Object) replica.SimpleResponse

func (*Index) ReplicateObjects 

func (i *Index) ReplicateObjects(ctx context.Context, shard, requestID string, objects []*storobj.Object) replica.SimpleResponse

func (*Index) ReplicateReferences 

func (i *Index) ReplicateReferences(ctx context.Context, shard, requestID string, refs []objects.BatchReference) replica.SimpleResponse

func (*Index) ReplicateUpdate 

func (i *Index) ReplicateUpdate(ctx context.Context, shard, requestID string, doc *objects.MergeDocument) replica.SimpleResponse

func (*Index) Shutdown 

func (i *Index) Shutdown(ctx context.Context) error

type IndexConfig 

type IndexConfig struct {
	RootPath                  string
	ClassName                 schema.ClassName
	QueryMaximumResults       int64
	QueryNestedRefLimit       int64
	ResourceUsage             config.ResourceUsage
	MemtablesFlushIdleAfter   int
	MemtablesInitialSizeMB    int
	MemtablesMaxSizeMB        int
	MemtablesMinActiveSeconds int
	MemtablesMaxActiveSeconds int
	ReplicationFactor         int64
	AvoidMMap                 bool

	TrackVectorDimensions bool
}

type KnnAggregator 

type KnnAggregator struct {
	// contains filtered or unexported fields
}

TODO: this is business logic, move out of here

func NewKnnAggregator 

func NewKnnAggregator(input search.Results, sourceVector []float32) *KnnAggregator

func (*KnnAggregator) Aggregate 

func (a *KnnAggregator) Aggregate(k int, properties []string) ([]classification.NeighborRef, error)

type Metrics 

type Metrics struct {
	// contains filtered or unexported fields
}

func NewMetrics 

func NewMetrics(
	logger logrus.FieldLogger, prom *monitoring.PrometheusMetrics,
	className, shardName string,
) *Metrics

func (*Metrics) BatchDelete 

func (m *Metrics) BatchDelete(start time.Time, op string)

func (*Metrics) BatchObject 

func (m *Metrics) BatchObject(start time.Time, size int)

func (*Metrics) DeleteShardLabels added in v1.21.7 

func (m *Metrics) DeleteShardLabels(class, shard string)

func (*Metrics) FilteredVectorFilter added in v1.18.0 

func (m *Metrics) FilteredVectorFilter(dur time.Duration)

func (*Metrics) FilteredVectorObjects added in v1.18.0 

func (m *Metrics) FilteredVectorObjects(dur time.Duration)

func (*Metrics) FilteredVectorSort added in v1.18.0 

func (m *Metrics) FilteredVectorSort(dur time.Duration)

func (*Metrics) FilteredVectorVector added in v1.18.0 

func (m *Metrics) FilteredVectorVector(dur time.Duration)

func (*Metrics) InvertedDeleteDelta 

func (m *Metrics) InvertedDeleteDelta(start time.Time)

func (*Metrics) InvertedDeleteOld 

func (m *Metrics) InvertedDeleteOld(start time.Time)

func (*Metrics) InvertedExtend 

func (m *Metrics) InvertedExtend(start time.Time, propCount int)

func (*Metrics) ObjectStore 

func (m *Metrics) ObjectStore(start time.Time)

func (*Metrics) PutObject 

func (m *Metrics) PutObject(start time.Time)

func (*Metrics) PutObjectDetermineStatus 

func (m *Metrics) PutObjectDetermineStatus(start time.Time)

func (*Metrics) PutObjectUpdateInverted 

func (m *Metrics) PutObjectUpdateInverted(start time.Time)

func (*Metrics) PutObjectUpsertObject 

func (m *Metrics) PutObjectUpsertObject(start time.Time)

func (*Metrics) ShardStartup 

func (m *Metrics) ShardStartup(start time.Time)

func (*Metrics) VectorIndex 

func (m *Metrics) VectorIndex(start time.Time)

type Migrator 

type Migrator struct {
	// contains filtered or unexported fields
}

func NewMigrator 

func NewMigrator(db *DB, logger logrus.FieldLogger) *Migrator

func (*Migrator) AddClass 

func (m *Migrator) AddClass(ctx context.Context, class *models.Class,
	shardState *sharding.State,
) error

func (*Migrator) AddProperty 

func (m *Migrator) AddProperty(ctx context.Context, className string, prop *models.Property) error

func (*Migrator) AdjustFilterablePropSettings added in v1.19.0 

func (m *Migrator) AdjustFilterablePropSettings(ctx context.Context) error

As of v1.19 property's IndexInverted setting is replaced with IndexFilterable and IndexSearchable Filterable buckets use roaring set strategy and searchable ones use map strategy (therefore are applicabe just for text/text[]) Since both type of buckets can coexist for text/text[] props they need to be distinguished by their name: searchable bucket has "searchable" suffix. Up until v1.19 default text/text[]/string/string[] (string/string[] deprecated since v1.19) strategy for buckets was map, migrating from pre v1.19 to v1.19 needs to properly handle existing text/text[] buckets of map strategy having filterable bucket name.

Enabled InvertedIndex translates in v1.19 to both InvertedFilterable and InvertedSearchable enabled, but since only searchable bucket exist (with filterable name), it has to be renamed to searchable bucket. Though IndexFilterable setting is enabled filterable index does not exists, therefore shards are switched into fallback mode, to use searchable buckets instead of filterable ones whenever filtered are expected. Fallback mode efectivelly sets IndexFilterable to false, although it stays enabled according to schema.

If filterable indexes will be created (that is up to user to decide whether missing indexes should be created later on), shards will not be working in fallback mode, and actual filterable index will be used when needed.

func (*Migrator) DeleteTenants added in v1.20.0 

func (m *Migrator) DeleteTenants(ctx context.Context, class *models.Class, tenants []string) (commit func(success bool), err error)

DeleteTenants deletes tenants and returns a commit func that can be used to either commit or rollback deletion

func (*Migrator) DropClass 

func (m *Migrator) DropClass(ctx context.Context, className string) error

func (*Migrator) DropProperty 

func (m *Migrator) DropProperty(ctx context.Context, className string, propertyName string) error

DropProperty is ignored, API compliant change

func (*Migrator) GetShardsStatus 

func (m *Migrator) GetShardsStatus(ctx context.Context, className string) (map[string]string, error)

func (*Migrator) InvertedReindex added in v1.18.0 

func (m *Migrator) InvertedReindex(ctx context.Context, taskNames ...string) error

func (*Migrator) NewTenants added in v1.20.0 

func (m *Migrator) NewTenants(ctx context.Context, class *models.Class, creates []*migrate.CreateTenantPayload) (commit func(success bool), err error)

NewTenants creates new partitions and returns a commit func that can be used to either commit or rollback the partitions

func (*Migrator) RecalculateVectorDimensions 

func (m *Migrator) RecalculateVectorDimensions(ctx context.Context) error

func (*Migrator) RecountProperties added in v1.19.0 

func (m *Migrator) RecountProperties(ctx context.Context) error

func (*Migrator) UpdateClass 

func (m *Migrator) UpdateClass(ctx context.Context, className string, newClassName *string) error

func (*Migrator) UpdateInvertedIndexConfig 

func (m *Migrator) UpdateInvertedIndexConfig(ctx context.Context, className string,
	updated *models.InvertedIndexConfig,
) error

func (*Migrator) UpdateProperty 

func (m *Migrator) UpdateProperty(ctx context.Context, className string, propName string, newName *string) error

func (*Migrator) UpdateShardStatus 

func (m *Migrator) UpdateShardStatus(ctx context.Context, className, shardName, targetStatus string) error

func (*Migrator) UpdateTenants added in v1.21.0 

func (m *Migrator) UpdateTenants(ctx context.Context, class *models.Class, updates []*migrate.UpdateTenantPayload) (commit func(success bool), err error)

UpdateTenans activates or deactivates tenant partitions and returns a commit func that can be used to either commit or rollback the changes

func (*Migrator) UpdateVectorIndexConfig 

func (m *Migrator) UpdateVectorIndexConfig(ctx context.Context,
	className string, updated schema.VectorIndexConfig,
) error

func (*Migrator) ValidateInvertedIndexConfigUpdate 

func (m *Migrator) ValidateInvertedIndexConfigUpdate(ctx context.Context,
	old, updated *models.InvertedIndexConfig,
) error

func (*Migrator) ValidateVectorIndexConfigUpdate 

func (m *Migrator) ValidateVectorIndexConfigUpdate(ctx context.Context,
	old, updated schema.VectorIndexConfig,
) error

type PropertyIndexType added in v1.18.0 

type PropertyIndexType uint8
const (
	IndexTypePropValue PropertyIndexType = iota + 1
	IndexTypePropLength
	IndexTypePropNull
	IndexTypePropSearchableValue
)

func GetPropNameAndIndexTypeFromBucketName added in v1.18.0 

func GetPropNameAndIndexTypeFromBucketName(bucketName string) (string, PropertyIndexType)

type ReindexableProperty added in v1.18.0 

type ReindexableProperty struct {
	PropertyName    string
	IndexType       PropertyIndexType
	NewIndex        bool // is new index, there is no bucket to replace with
	DesiredStrategy string
	BucketOptions   []lsmkv.BucketOption
}

type Replicator 

type Replicator interface {
	ReplicateObject(ctx context.Context, shardName, requestID string,
		object *storobj.Object) replica.SimpleResponse
	ReplicateObjects(ctx context.Context, shardName, requestID string,
		objects []*storobj.Object) replica.SimpleResponse
	ReplicateUpdate(ctx context.Context, shard, requestID string,
		doc *objects.MergeDocument) replica.SimpleResponse
	ReplicateDeletion(ctx context.Context, shardName, requestID string,
		uuid strfmt.UUID) replica.SimpleResponse
	ReplicateDeletions(ctx context.Context, shardName, requestID string,
		docIDs []uint64, dryRun bool) replica.SimpleResponse
	ReplicateReferences(ctx context.Context, shard, requestID string,
		refs []objects.BatchReference) replica.SimpleResponse
	CommitReplication(shard,
		requestID string) interface{}
	AbortReplication(shardName,
		requestID string) interface{}
}

type Shard 

type Shard struct {
	// contains filtered or unexported fields
}

Shard is the smallest completely-contained index unit. A shard manages database files for all the objects it owns. How a shard is determined for a target object (e.g. Murmur hash, etc.) is still open at this point

func NewShard 

func NewShard(ctx context.Context, promMetrics *monitoring.PrometheusMetrics,
	shardName string, index *Index, class *models.Class, jobQueueCh chan job,
) (*Shard, error)

func (*Shard) DBPathLSM 

func (s *Shard) DBPathLSM() string

func (*Shard) Dimensions 

func (s *Shard) Dimensions() int

func (*Shard) ID 

func (s *Shard) ID() string

type ShardInvertedReindexTask added in v1.18.0 

type ShardInvertedReindexTask interface {
	GetPropertiesToReindex(ctx context.Context, shard *Shard,
	) ([]ReindexableProperty, error)
	// right now only OnResume is needed, but in the future more
	// callbacks could be added
	// (like OnPrePauseStore, OnPostPauseStore, OnPreResumeStore, etc)
	OnPostResumeStore(ctx context.Context, shard *Shard) error
}

type ShardInvertedReindexTaskSetToRoaringSet added in v1.18.0 

type ShardInvertedReindexTaskSetToRoaringSet struct{}

func (*ShardInvertedReindexTaskSetToRoaringSet) GetPropertiesToReindex added in v1.18.0 

func (t *ShardInvertedReindexTaskSetToRoaringSet) GetPropertiesToReindex(ctx context.Context,
	shard *Shard,
) ([]ReindexableProperty, error)

func (*ShardInvertedReindexTaskSetToRoaringSet) OnPostResumeStore added in v1.19.0 

func (t *ShardInvertedReindexTaskSetToRoaringSet) OnPostResumeStore(ctx context.Context, shard *Shard) error

type ShardInvertedReindexer added in v1.18.0 

type ShardInvertedReindexer struct {
	// contains filtered or unexported fields
}

func NewShardInvertedReindexer added in v1.18.0 

func NewShardInvertedReindexer(shard *Shard, logger logrus.FieldLogger) *ShardInvertedReindexer

func (*ShardInvertedReindexer) AddTask added in v1.18.0 

func (r *ShardInvertedReindexer) AddTask(task ShardInvertedReindexTask)

func (*ShardInvertedReindexer) Do added in v1.18.0 

func (r *ShardInvertedReindexer) Do(ctx context.Context) error

type VectorIndex 

type VectorIndex interface {
	Dump(labels ...string)
	Add(id uint64, vector []float32) error
	Delete(id ...uint64) error
	SearchByVector(vector []float32, k int, allow helpers.AllowList) ([]uint64, []float32, error)
	SearchByVectorDistance(vector []float32, dist float32,
		maxLimit int64, allow helpers.AllowList) ([]uint64, []float32, error)
	UpdateUserConfig(updated schema.VectorIndexConfig, callback func()) error
	Drop(ctx context.Context) error
	Shutdown(ctx context.Context) error
	Flush() error
	SwitchCommitLogs(ctx context.Context) error
	ListFiles(ctx context.Context) ([]string, error)
	PostStartup()
	ValidateBeforeInsert(vector []float32) error
}

VectorIndex is anything that indexes vectors efficiently. For an example look at ./vector/hnsw/index.go

Source Files

View all Source files

Directories

Path Synopsis
clusterintegrationtest acts as a test package to provide a compoenent test spanning multiple parts of the application, including everything that's required for a distributed setup.
 clusterintegrationtest acts as a test package to provide a compoenent test spanning multiple parts of the application, including everything that's required for a distributed setup.
stopwords
entities
ent contains common types used throughout various lsmkv (sub-)packages
 ent contains common types used throughout various lsmkv (sub-)packages
rbtree
roaringset
The "roaringset" package contains all the LSM business logic that is unique to the "RoaringSet" strategy
 The "roaringset" package contains all the LSM business logic that is unique to the "RoaringSet" strategy
segmentindex
vector
geo
hnsw
hnsw/commitlog
hnsw/distancer
hnsw/distancer/asm
asm only has amd64 specific implementations at the moment
 asm only has amd64 specific implementations at the moment
hnsw/priorityqueue
hnsw/visited
noop
ssdhelpers
testinghelpers

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.