schema

package
v1.26.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 23, 2024 License: BSD-3-Clause Imports: 36 Imported by: 0

Documentation

Overview

Package migrate provides a simple composer tool, which implements the Migrator interface and can take in any number of migrators which themselves have to implement the interface

Index

Constants

View Source 
const (
	ErrMsgMaxAllowedTenants = "" /* 136-byte string literal not displayed */
)

Variables

View Source 
var ErrNotFound = errors.New("not found")

Functions

func Diff added in v1.17.4 

func Diff(
	leftLabel string, left *State,
	rightLabel string, right *State,
) []string

Diff creates human-readable information about the difference in two schemas, returns a len=0 slice if schemas are identical

func IsLocalActiveTenant added in v1.25.0 

func IsLocalActiveTenant(phys *sharding.Physical, localNode string) bool

IsLocalActiveTenant determines whether a given physical partition represents a tenant that is expected to be active

func NewExecutor added in v1.25.0 

func NewExecutor(migrator Migrator, sr SchemaReader,
	logger logrus.FieldLogger, classBackupDir func(string) error,
) *executor

NewManager creates a new manager

Types

type ClassPayload added in v1.20.0 

type ClassPayload struct {
	Name          string
	Metadata      []byte
	ShardingState []byte
	Shards        []KeyValuePair
	ReplaceShards bool
	Error         error
}

ClassPayload is used to serialize class updates

type CreateTenantPayload added in v1.25.0 

type CreateTenantPayload struct {
	Name   string
	Status string
}

type Handler added in v1.25.0 

type Handler struct {
	Authorizer authorizer
	// contains filtered or unexported fields
}

The handler manages API requests for manipulating class schemas. This separation of responsibilities helps decouple these tasks from the Manager class, which combines many unrelated functions. By delegating these clear responsibilities to the handler, it maintains a clean separation from the manager, enhancing code modularity and maintainability.

func NewHandler added in v1.25.0 

func NewHandler(
	schemaReader SchemaReader,
	schemaManager SchemaManager,
	validator validator,
	logger logrus.FieldLogger, authorizer authorizer, config config.Config,
	configParser VectorConfigParser, vectorizerValidator VectorizerValidator,
	invertedConfigValidator InvertedConfigValidator,
	moduleConfig ModuleConfig, clusterState clusterState,
	scaleoutManager scaleOut,
) (Handler, error)

NewHandler creates a new handler

func (*Handler) AddClass added in v1.25.0 

func (h *Handler) AddClass(ctx context.Context, principal *models.Principal,
	cls *models.Class,
) (*models.Class, uint64, error)

AddClass to the schema

func (*Handler) AddClassProperty added in v1.25.0 

func (h *Handler) AddClassProperty(ctx context.Context, principal *models.Principal,
	class *models.Class, merge bool, newProps ...*models.Property,
) (*models.Class, uint64, error)

AddClassProperty it is upsert operation. it adds properties to a class and updates existing properties if the merge bool passed true.

func (*Handler) AddTenants added in v1.25.0 

func (h *Handler) AddTenants(ctx context.Context,
	principal *models.Principal,
	class string,
	tenants []*models.Tenant,
) (uint64, error)

AddTenants is used to add new tenants to a class Class must exist and has partitioning enabled

func (*Handler) ConsistentTenantExists added in v1.25.0 

func (h *Handler) ConsistentTenantExists(ctx context.Context, principal *models.Principal, class string, consistency bool, tenant string) error

TenantExists is used to check if the tenant exists of a class

Class must exist and has partitioning enabled

func (*Handler) DeleteClass added in v1.25.0 

func (h *Handler) DeleteClass(ctx context.Context, principal *models.Principal, class string) error

DeleteClass from the schema

func (*Handler) DeleteClassProperty added in v1.25.0 

func (h *Handler) DeleteClassProperty(ctx context.Context, principal *models.Principal,
	class string, property string,
) error

DeleteClassProperty from existing Schema

func (*Handler) DeleteTenants added in v1.25.0 

func (h *Handler) DeleteTenants(ctx context.Context, principal *models.Principal, class string, tenants []string) error

DeleteTenants is used to delete tenants of a class.

Class must exist and has partitioning enabled

func (*Handler) GetCachedClass added in v1.25.0 

func (h *Handler) GetCachedClass(ctxWithClassCache context.Context,
	principal *models.Principal, names ...string,
) (map[string]versioned.Class, error)

func (*Handler) GetClass added in v1.25.0 

func (h *Handler) GetClass(ctx context.Context, principal *models.Principal, name string) (*models.Class, error)

func (*Handler) GetConsistentClass added in v1.25.0 

func (h *Handler) GetConsistentClass(ctx context.Context, principal *models.Principal,
	name string, consistency bool,
) (*models.Class, uint64, error)

func (*Handler) GetConsistentSchema added in v1.25.0 

func (h *Handler) GetConsistentSchema(principal *models.Principal, consistency bool) (schema.Schema, error)

GetSchema retrieves a locally cached copy of the schema

func (*Handler) GetConsistentTenants added in v1.25.0 

func (h *Handler) GetConsistentTenants(ctx context.Context, principal *models.Principal, class string, consistency bool, tenants []string) ([]*models.Tenant, error)

func (*Handler) GetSchema added in v1.25.0 

func (h *Handler) GetSchema(principal *models.Principal) (schema.Schema, error)

GetSchema retrieves a locally cached copy of the schema

func (*Handler) GetSchemaSkipAuth added in v1.25.0 

func (h *Handler) GetSchemaSkipAuth() schema.Schema

GetSchemaSkipAuth can never be used as a response to a user request as it could leak the schema to an unauthorized user, is intended to be used for non-user triggered processes, such as regular updates / maintenance / etc

func (*Handler) GetTenants added in v1.25.0 

func (h *Handler) GetTenants(ctx context.Context, principal *models.Principal, class string) ([]*models.Tenant, error)

GetTenants is used to get tenants of a class.

Class must exist and has partitioning enabled

func (*Handler) JoinNode added in v1.25.0 

func (h *Handler) JoinNode(ctx context.Context, node string, nodePort string, voter bool) error

JoinNode adds the given node to the cluster. Node needs to reachable via memberlist/gossip. If nodePort is an empty string, nodePort will be the default raft port. If the node is not reachable using memberlist, an error is returned If joining the node fails, an error is returned.

func (*Handler) NodeName added in v1.25.0 

func (h *Handler) NodeName() string

func (*Handler) Nodes added in v1.25.0 

func (h *Handler) Nodes() []string

func (*Handler) RemoveNode added in v1.25.0 

func (h *Handler) RemoveNode(ctx context.Context, node string) error

RemoveNode removes the given node from the cluster.

func (*Handler) RestoreClass added in v1.25.0 

func (h *Handler) RestoreClass(ctx context.Context, d *backup.ClassDescriptor, m map[string]string) error

func (*Handler) ShardsStatus added in v1.25.0 

func (h *Handler) ShardsStatus(ctx context.Context,
	principal *models.Principal, class, tenant string,
) (models.ShardStatusList, error)

func (*Handler) Statistics added in v1.25.0 

func (h *Handler) Statistics() map[string]any

Statistics is used to return a map of various internal stats. This should only be used for informative purposes or debugging.

func (*Handler) StoreSchemaV1 added in v1.25.0 

func (h *Handler) StoreSchemaV1() error

func (*Handler) UpdateClass added in v1.25.0 

func (h *Handler) UpdateClass(ctx context.Context, principal *models.Principal,
	className string, updated *models.Class,
) error

func (*Handler) UpdateShardStatus added in v1.25.0 

func (h *Handler) UpdateShardStatus(ctx context.Context,
	principal *models.Principal, class, shard, status string,
) (uint64, error)

func (*Handler) UpdateTenants added in v1.25.0 

func (h *Handler) UpdateTenants(ctx context.Context, principal *models.Principal,
	class string, tenants []*models.Tenant,
) ([]*models.Tenant, error)

UpdateTenants is used to set activity status of tenants of a class.

Class must exist and has partitioning enabled

type InvertedConfigValidator 

type InvertedConfigValidator func(in *models.InvertedIndexConfig) error

type KeyValuePair added in v1.20.0 

type KeyValuePair struct {
	Key   string
	Value []byte
}

KeyValuePair is used to serialize shards updates

type Manager 

type Manager struct {
	Authorizer authorizer

	sync.RWMutex
	// The handler is responsible for well-defined tasks and should be decoupled from the manager.
	// This enables API requests to be directed straight to the handler without the need to pass through the manager.
	// For more context, refer to the handler's definition.
	Handler

	SchemaReader
	// contains filtered or unexported fields
}

Manager Manages schema changes at a use-case level, i.e. agnostic of underlying databases or storage providers

func NewManager 

func NewManager(validator validator,
	schemaManager SchemaManager,
	schemaReader SchemaReader,
	repo SchemaStore,
	logger logrus.FieldLogger, authorizer authorizer, config config.Config,
	configParser VectorConfigParser, vectorizerValidator VectorizerValidator,
	invertedConfigValidator InvertedConfigValidator,
	moduleConfig ModuleConfig, clusterState clusterState,
	scaleoutManager scaleOut,
) (*Manager, error)

NewManager creates a new manager

func (*Manager) AllowImplicitTenantActivation added in v1.25.2 

func (m *Manager) AllowImplicitTenantActivation(class string) bool

func (*Manager) ClusterHealthScore 

func (m *Manager) ClusterHealthScore() int

func (*Manager) OptimisticTenantStatus added in v1.25.0 

func (m *Manager) OptimisticTenantStatus(class string, tenant string) (map[string]string, error)

OptimisticTenantStatus tries to query the local state first. It is optimistic that the state has already propagated correctly. If the state is unexpected, i.e. either the tenant is not found at all or the status is COLD, it will double-check with the leader.

This way we accept false positives (for HOT tenants), but guarantee that there will never be false negatives (i.e. tenants labelled as COLD that the leader thinks should be HOT).

This means:

  • If a tenant is HOT locally (true positive), we proceed normally
  • If a tenant is HOT locally, but should be COLD (false positive), we still proceed. This is a conscious decision to keep the happy path free from (expensive) leader lookups.
  • If a tenant is not found locally, we assume it was recently created, but the state hasn't propagated yet. To verify, we check with the leader.
  • If a tenant is found locally, but is marked as COLD, we assume it was recently turned HOT, but the state hasn't propagated yet. To verify, we check with the leader

Overall, we keep the (very common) happy path, free from expensive leader-lookups and only fall back to the leader if the local result implies an unhappy path.

func (*Manager) ResolveParentNodes added in v1.18.0 

func (m *Manager) ResolveParentNodes(class, shardName string) (map[string]string, error)

ResolveParentNodes gets all replicas for a specific class shard and resolves their names

it returns map[node_name] node_address where node_address = "" if can't resolve node_name

func (*Manager) ShardOwner added in v1.20.0 

func (m *Manager) ShardOwner(class, shard string) (string, error)

func (*Manager) TenantsShards added in v1.25.0 

func (m *Manager) TenantsShards(class string, tenants ...string) (map[string]string, error)

type Migrator added in v1.25.0 

type Migrator interface {
	AddClass(ctx context.Context, class *models.Class, shardingState *sharding.State) error
	DropClass(ctx context.Context, className string, hasFrozen bool) error
	// UpdateClass(ctx context.Context, className string,newClassName *string) error
	GetShardsQueueSize(ctx context.Context, className, tenant string) (map[string]int64, error)

	AddProperty(ctx context.Context, className string,
		props ...*models.Property) error
	UpdateProperty(ctx context.Context, className string,
		propName string, newName *string) error
	UpdateIndex(ctx context.Context, class *models.Class, shardingState *sharding.State) error

	NewTenants(ctx context.Context, class *models.Class, creates []*CreateTenantPayload) error
	UpdateTenants(ctx context.Context, class *models.Class, updates []*UpdateTenantPayload) error
	DeleteTenants(ctx context.Context, class string, tenants []string) error

	GetShardsStatus(ctx context.Context, className, tenant string) (map[string]string, error)
	UpdateShardStatus(ctx context.Context, className, shardName, targetStatus string, schemaVersion uint64) error

	UpdateVectorIndexConfig(ctx context.Context, className string, updated schemaConfig.VectorIndexConfig) error
	ValidateVectorIndexConfigsUpdate(old, updated map[string]schemaConfig.VectorIndexConfig) error
	UpdateVectorIndexConfigs(ctx context.Context, className string,
		updated map[string]schemaConfig.VectorIndexConfig) error
	ValidateInvertedIndexConfigUpdate(old, updated *models.InvertedIndexConfig) error
	UpdateInvertedIndexConfig(ctx context.Context, className string,
		updated *models.InvertedIndexConfig) error
	UpdateReplicationConfig(ctx context.Context, className string,
		updated *models.ReplicationConfig) error
	WaitForStartup(context.Context) error
	Shutdown(context.Context) error
}

Migrator represents both the input and output interface of the Composer

type ModuleConfig 

type ModuleConfig interface {
	SetClassDefaults(class *models.Class)
	SetSinglePropertyDefaults(class *models.Class, props ...*models.Property)
	ValidateClass(ctx context.Context, class *models.Class) error
	GetByName(name string) modulecapabilities.Module
}

type Parser added in v1.25.0 

type Parser struct {
	// contains filtered or unexported fields
}

func NewParser added in v1.25.0 

func NewParser(cs clusterState, vCfg VectorConfigParser, v validator) *Parser

func (*Parser) ParseClass added in v1.25.0 

func (m *Parser) ParseClass(class *models.Class) error

func (*Parser) ParseClassUpdate added in v1.25.0 

func (p *Parser) ParseClassUpdate(class, update *models.Class) (*models.Class, error)

ParseClassUpdate parses a class after unmarshaling by setting concrete types for the fields

type RefFinder 

type RefFinder struct {
	// contains filtered or unexported fields
}

RefFinder is a helper that lists classes and their possible paths to to a desired target class.

For example if the target class is "car". It might list: - Person, drives, Car - Person, owns, Car - Person, friendsWith, Person, drives, Car etc.

It will stop at a preconfigured depth limit, to avoid infinite results, such as: - Person, friendsWith, Person, friendsWith, Person, ..., drives Car

func NewRefFinder 

func NewRefFinder(getter schemaGetterForRefFinder, depthLimit int) *RefFinder

NewRefFinder with SchemaGetter and depth limit

func (*RefFinder) Find 

func (r *RefFinder) Find(className libschema.ClassName) []filters.Path

type SchemaGetter 

type SchemaGetter interface {
	GetSchemaSkipAuth() schema.Schema
	ReadOnlyClass(string) *models.Class
	Nodes() []string
	NodeName() string
	ClusterHealthScore() int
	ResolveParentNodes(string, string) (map[string]string, error)
	Statistics() map[string]any

	CopyShardingState(class string) *sharding.State
	ShardOwner(class, shard string) (string, error)
	TenantsShards(class string, tenants ...string) (map[string]string, error)
	OptimisticTenantStatus(class string, tenants string) (map[string]string, error)
	ShardFromUUID(class string, uuid []byte) string
	ShardReplicas(class, shard string) ([]string, error)
}

type SchemaManager added in v1.25.5 

type SchemaManager interface {
	// Schema writes operation.
	AddClass(cls *models.Class, ss *sharding.State) (uint64, error)
	RestoreClass(cls *models.Class, ss *sharding.State) (uint64, error)
	UpdateClass(cls *models.Class, ss *sharding.State) (uint64, error)
	DeleteClass(name string) (uint64, error)
	AddProperty(class string, p ...*models.Property) (uint64, error)
	UpdateShardStatus(class, shard, status string) (uint64, error)
	AddTenants(class string, req *command.AddTenantsRequest) (uint64, error)
	UpdateTenants(class string, req *command.UpdateTenantsRequest) (uint64, error)
	DeleteTenants(class string, req *command.DeleteTenantsRequest) (uint64, error)

	// Cluster related operations
	Join(_ context.Context, nodeID, raftAddr string, voter bool) error
	Remove(_ context.Context, nodeID string) error
	Stats() map[string]any
	StoreSchemaV1() error

	// Strongly consistent schema read. These endpoints will emit a query to the leader to ensure that the data is read
	// from an up to date schema.
	QueryReadOnlyClasses(names ...string) (map[string]versioned.Class, error)
	QuerySchema() (models.Schema, error)
	QueryTenants(class string, tenants []string) ([]*models.Tenant, uint64, error)
	QueryShardOwner(class, shard string) (string, uint64, error)
	QueryTenantsShards(class string, tenants ...string) (map[string]string, uint64, error)
	QueryShardingState(class string) (*sharding.State, uint64, error)
}

SchemaManager is responsible for consistent schema operations. It allows reading and writing the schema while directly talking to the leader, no matter which node it is. It also allows cluster related operations that can only be done on the leader (join/remove/stats/etc...) For details about each endpoint see github.com/weaviate/weaviate/cluster.Raft. For local schema lookup where eventual consistency is acceptable, see SchemaReader.

type SchemaReader added in v1.25.5 

type SchemaReader interface {
	// WaitForUpdate ensures that the local schema has caught up to version.
	WaitForUpdate(ctx context.Context, version uint64) error

	// These schema reads function reads the metadata immediately present in the local schema and can be eventually
	// consistent.
	// For details about each endpoint see [github.com/weaviate/weaviate/cluster/schema.SchemaReader].
	ClassEqual(name string) string
	MultiTenancy(class string) models.MultiTenancyConfig
	ClassInfo(class string) (ci clusterSchema.ClassInfo)
	ReadOnlyClass(name string) *models.Class
	ReadOnlySchema() models.Schema
	CopyShardingState(class string) *sharding.State
	ShardReplicas(class, shard string) ([]string, error)
	ShardFromUUID(class string, uuid []byte) string
	ShardOwner(class, shard string) (string, error)
	Read(class string, reader func(*models.Class, *sharding.State) error) error
	GetShardsStatus(class, tenant string) (models.ShardStatusList, error)

	// These schema reads function (...WithVersion) return the metadata once the local schema has caught up to the
	// version parameter. If version is 0 is behaves exactly the same as eventual consistent reads.
	// For details about each endpoint see [github.com/weaviate/weaviate/cluster/schema.VersionedSchemaReader].
	ClassInfoWithVersion(ctx context.Context, class string, version uint64) (clusterSchema.ClassInfo, error)
	MultiTenancyWithVersion(ctx context.Context, class string, version uint64) (models.MultiTenancyConfig, error)
	ReadOnlyClassWithVersion(ctx context.Context, class string, version uint64) (*models.Class, error)
	ShardOwnerWithVersion(ctx context.Context, lass, shard string, version uint64) (string, error)
	ShardFromUUIDWithVersion(ctx context.Context, class string, uuid []byte, version uint64) (string, error)
	ShardReplicasWithVersion(ctx context.Context, class, shard string, version uint64) ([]string, error)
	TenantsShardsWithVersion(ctx context.Context, version uint64, class string, tenants ...string) (map[string]string, error)
	CopyShardingStateWithVersion(ctx context.Context, class string, version uint64) (*sharding.State, error)
}

SchemaReader allows reading the local schema with or without using a schema version.

type SchemaStore added in v1.20.0 

type SchemaStore interface {
	// Save saves the complete schema to the persistent storage
	Save(ctx context.Context, schema State) error

	// Load loads the complete schema from the persistent storage
	Load(context.Context) (State, error)
}

SchemaStore is responsible for persisting the schema by providing support for both partial and complete schema updates Deprecated: instead schema now is persistent via RAFT see : usecase/schema/handler.go & cluster/store/store.go Load and save are left to support backward compatibility

type State 

type State struct {
	ObjectSchema  *models.Schema `json:"object"`
	ShardingState map[string]*sharding.State
}

State is a cached copy of the schema that can also be saved into a remote storage, as specified by Repo

func NewState added in v1.20.0 

func NewState(nClasses int) State

NewState returns a new state with room for nClasses classes

func (State) EqualEnough added in v1.24.9 

func (s State) EqualEnough(other *State) bool

type UpdateTenantPayload added in v1.25.0 

type UpdateTenantPayload struct {
	Name   string
	Status string
}

type VectorConfigParser 

type VectorConfigParser func(in interface{}, vectorIndexType string) (schemaConfig.VectorIndexConfig, error)

type VectorizerValidator 

type VectorizerValidator interface {
	ValidateVectorizer(moduleName string) error
}

Source Files

View all Source files

Directories

Path Synopsis
migrate
fs

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.