metadata

package
v1.0.0-alpha.33 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 24, 2022 License: Apache-2.0 Imports: 21 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	// DefaultNamespaceName is for "default" namespace in the cluster which means all the databases created are under a single
	// namespace.
	// It is totally fine for a deployment to choose this and just have one namespace. The default assigned value for
	// this namespace is 1.
	DefaultNamespaceName string = "default_namespace"

	DefaultNamespaceId = uint32(1)
)

Variables

View Source 
var (
	// VersionKey is the metadata version key whose value is returned to the clients in the transaction
	VersionKey = []byte{0xff, '/', 'm', 'e', 't', 'a', 'd', 'a', 't', 'a', 'V', 'e', 'r', 's', 'i', 'o', 'n'}
	// VersionValue is the value set when calling setVersionstampedValue, any value other than this is rejected.
	VersionValue = []byte{0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00}
)

Functions

func NewCollectionHolder 

func NewCollectionHolder(id uint32, name string, collection *schema.DefaultCollection, idxNameToId map[string]uint32) *collectionHolder

Types

type CacheTracker 

type CacheTracker struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

CacheTracker is to track if tenant cache is stale and trigger reloads on it. CacheTracker is used by the session manager to identify during the running transaction whether there is a need to fill tenant state from the database or whether the cached version is up-to-date.

func NewCacheTracker 

func NewCacheTracker(tenantMgr *TenantManager, txMgr *transaction.Manager) *CacheTracker

NewCacheTracker creates and returns the cache tracker. It uses tenant manager state to populate in-memory version tracking for each tenant.

func (*CacheTracker) DeferredTracking 

func (cacheTracker *CacheTracker) DeferredTracking(ctx context.Context, tx transaction.Tx, tenant *Tenant) (*Tracker, error)

DeferredTracking returns a tracker that has a future attached to the caller's transaction.

func (*CacheTracker) InstantTracking 

func (cacheTracker *CacheTracker) InstantTracking(ctx context.Context, tx transaction.Tx, tenant *Tenant) (*Tracker, error)

InstantTracking is when a tracker is needed outside the caller’s transaction. This is used by DDL transactions that are also bumping up the metadata version.

type Database 

type Database struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

Database is to manage the collections for this database. Check the Clone method before changing this struct.

func NewDatabase 

func NewDatabase(id uint32, name string) *Database

func (*Database) Clone 

func (d *Database) Clone() *Database

Clone is used to stage the database

func (*Database) GetCollection 

func (d *Database) GetCollection(cname string) *schema.DefaultCollection

GetCollection returns the collection object, or null if the collection map contains no mapping for the database. At this point collection is fully formed and safe to use.

func (*Database) Id 

func (d *Database) Id() uint32

Id returns the dictionary encoded value of this collection.

func (*Database) ListCollection 

func (d *Database) ListCollection() []*schema.DefaultCollection

ListCollection returns the collection object of all the collections in this database.

func (*Database) Name 

func (d *Database) Name() string

Name returns the database name.

type DefaultNamespace 

type DefaultNamespace struct{}

DefaultNamespace is for "default" namespace in the cluster. This is useful when there is no need to logically group databases. All databases will be created under a single namespace. It is totally fine for a deployment to choose this and just have one namespace. The default assigned value for this namespace is 1.

func NewDefaultNamespace 

func NewDefaultNamespace() *DefaultNamespace

func (*DefaultNamespace) Id 

func (n *DefaultNamespace) Id() uint32

Id returns id assigned to the namespace

func (*DefaultNamespace) Name 

func (n *DefaultNamespace) Name() string

type DictKeyEncoder 

type DictKeyEncoder struct {
}

func (*DictKeyEncoder) DecodeIndexName 

func (d *DictKeyEncoder) DecodeIndexName(indexName []byte) uint32

func (*DictKeyEncoder) DecodeTableName 

func (d *DictKeyEncoder) DecodeTableName(tableName []byte) (uint32, uint32, uint32, bool)

func (*DictKeyEncoder) EncodeIndexName 

func (d *DictKeyEncoder) EncodeIndexName(idx *schema.Index) []byte

func (*DictKeyEncoder) EncodeKey 

func (d *DictKeyEncoder) EncodeKey(encodedTable []byte, idx *schema.Index, idxParts []interface{}) (keys.Key, error)

func (*DictKeyEncoder) EncodeTableName 

func (d *DictKeyEncoder) EncodeTableName(ns Namespace, db *Database, coll *schema.DefaultCollection) ([]byte, error)

EncodeTableName creates storage friendly table name from namespace, database and collection ids Database and collection objects can be omitted to get table name prefix. If the collection is ommitted then result name includes all the collections in the database If both database and collections are omitted then result name includes all databases in the namespace

type Encoder 

type Encoder interface {
	// EncodeTableName returns encoded bytes which are formed by combining namespace, database, and collection.
	EncodeTableName(ns Namespace, db *Database, coll *schema.DefaultCollection) ([]byte, error)
	// EncodeIndexName returns encoded bytes for the index name
	EncodeIndexName(idx *schema.Index) []byte
	// EncodeKey returns encoded bytes of the key which will be used to store the values in fdb. The Key return by this
	// method has two parts,
	//   - tableName: This is set with an encoding of namespace, database and collection id.
	//   - IndexParts: This has the index identifier and value(s) associated with a single or composite index. This is appended
	//	   to the table name to form the Key. The first element of this list is the dictionary encoding of index type key
	//	   information i.e. whether the index is pkey, etc. The remaining elements are values for this index.
	EncodeKey(encodedTable []byte, idx *schema.Index, idxParts []interface{}) (keys.Key, error)

	// DecodeTableName is used to decode the key stored in FDB and extract namespace name, database name and collection ids.
	DecodeTableName(tableName []byte) (uint32, uint32, uint32, bool)
	DecodeIndexName(indexName []byte) uint32
}

Encoder is used to encode/decode values of the Key.

func NewEncoder 

func NewEncoder() Encoder

NewEncoder creates Dictionary metaStore to encode keys.

type Namespace 

type Namespace interface {
	// Id for the namespace is used by the cluster to append as the first element in the key.
	Id() uint32
	// Name is the name used for the lookup.
	Name() string
}

A Namespace is a logical grouping of databases.

type NamespaceType 

type NamespaceType string

type TableKeyGenerator 

type TableKeyGenerator struct{}

TableKeyGenerator is used to generated keys that may need persistence like counter.

func NewTableKeyGenerator 

func NewTableKeyGenerator() *TableKeyGenerator

func (*TableKeyGenerator) GenerateCounter 

func (g *TableKeyGenerator) GenerateCounter(ctx context.Context, txMgr *transaction.Manager, table []byte) (int32, error)

GenerateCounter is used to generate an id in a transaction for int32 field only. This is mainly used to guarantee uniqueness with auto-incremented ids, so what we are doing is reserving this id in storage before returning to the caller so that only one id is assigned to one caller.

type Tenant 

type Tenant struct {
	sync.RWMutex

	Encoder Encoder

	TableKeyGenerator *TableKeyGenerator
	// contains filtered or unexported fields
}

Tenant is a logical grouping of databases. The tenant is used to manage all the databases that belongs to this tenant and the corresponding collections for these databases. Operations performed on the tenant object are thread-safe.

func NewTenant 

func NewTenant(namespace Namespace, kvStore kv.KeyValueStore, searchStore search.Store, dict *encoding.MetadataDictionary, schemaStore *encoding.SchemaSubspace, encoder Encoder, versionH *VersionHandler, currentVersion Version, tableKeyGenerator *TableKeyGenerator) *Tenant

func (*Tenant) CollectionSize 

func (tenant *Tenant) CollectionSize(ctx context.Context, db *Database, coll *schema.DefaultCollection) (int64, error)

CollectionSize returns approximate data size on disk for all the collections for the database provided by the caller.

func (*Tenant) CreateCollection 

func (tenant *Tenant) CreateCollection(ctx context.Context, tx transaction.Tx, database *Database, schFactory *schema.Factory) error

CreateCollection is to create a collection inside tenant namespace.

func (*Tenant) CreateDatabase 

func (tenant *Tenant) CreateDatabase(ctx context.Context, tx transaction.Tx, dbName string) (bool, error)

CreateDatabase is responsible for creating a dictionary encoding of the database name. This method is not adding the entry to the tenant because the outer layer may still roll back the transaction. The session manager is bumping the metadata version once the commit is successful so reloading happens at the next call when a transaction sees a stale tenant version. This applies to the reloading mechanism on all the servers. It returns "true" If the database already exists, else "false" and the error

func (*Tenant) DatabaseSize 

func (tenant *Tenant) DatabaseSize(ctx context.Context, db *Database) (int64, error)

DatabaseSize returns approximate data size on disk for all the database for this tenant.

func (*Tenant) DropCollection 

func (tenant *Tenant) DropCollection(ctx context.Context, tx transaction.Tx, db *Database, collectionName string) error

DropCollection is to drop a collection and its associated indexes. It removes the "created" entry from the encoding subspace and adds a "dropped" entry for the same collection key.

func (*Tenant) DropDatabase 

func (tenant *Tenant) DropDatabase(ctx context.Context, tx transaction.Tx, dbName string) (bool, error)

DropDatabase is responsible for first dropping a dictionary encoding of the database and then adding a corresponding dropped encoding entry in the encoding table. Drop returns "false" if the database doesn't exist so that caller can reason about it. DropDatabase is more involved than CreateDatabase as with Drop we also need to iterate over all the collections present in this database and call drop collection on each one of them.

func (*Tenant) GetCollection 

func (tenant *Tenant) GetCollection(db string, collection string) *schema.DefaultCollection

func (*Tenant) GetDatabase 

func (tenant *Tenant) GetDatabase(_ context.Context, dbName string) (*Database, error)

GetDatabase returns the database object, or null if there is no database existing with the name passed in the param. As reloading of tenant state is happening at the session manager layer so GetDatabase calls assume that the caller just needs the state from the cache.

func (*Tenant) GetNamespace 

func (tenant *Tenant) GetNamespace() Namespace

GetNamespace returns the namespace of this tenant.

func (*Tenant) ListDatabases 

func (tenant *Tenant) ListDatabases(_ context.Context) []string

ListDatabases is used to list all database available for this tenant.

func (*Tenant) Reload 

func (tenant *Tenant) Reload(ctx context.Context, tx transaction.Tx, version Version) error

Reload is used to reload this tenant. The reload method compares the currently attached version to the tenant to the version passed in the API call to detect whether reloading is needed. This check is needed to ensure only a single thread will actually perform reload. This is a blocking API which means if most of the requests detected that the tenant state is stale then they all will block till one of them will reload the tenant state from the database. All the blocking transactions will be restarted to ensure they see the latest view of the tenant.

func (*Tenant) Size 

func (tenant *Tenant) Size(ctx context.Context) (int64, error)

Size returns approximate data size on disk for all the collections, databases for this tenant.

func (*Tenant) String 

func (tenant *Tenant) String() string

type TenantManager 

type TenantManager struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

TenantManager is to manage all the tenants ToDo: start a background thread to reload the mapping

func NewTenantManager 

func NewTenantManager(kvStore kv.KeyValueStore, searchStore search.Store) *TenantManager

func NewTestTenantMgr 

func NewTestTenantMgr(kvStore kv.KeyValueStore) (*TenantManager, context.Context, context.CancelFunc)

NewTestTenantMgr creates new TenantManager for tests

func (*TenantManager) CreateOrGetTenant 

func (m *TenantManager) CreateOrGetTenant(ctx context.Context, txMgr *transaction.Manager, namespace Namespace) (tenant *Tenant, err error)

CreateOrGetTenant is a thread safe implementation of creating a new tenant. It returns the tenant if it already exists. This is mainly returning the tenant to avoid calling "Get" again after creating the tenant. This method is expensive as it reloads the existing tenants from the disk if it sees the tenant is not present in the cache.

func (*TenantManager) CreateTenant 

func (m *TenantManager) CreateTenant(ctx context.Context, tx transaction.Tx, namespace Namespace) (Namespace, error)

CreateTenant is a thread safe implementation of creating a new tenant. It returns an error if it already exists.

func (*TenantManager) DecodeTableName 

func (m *TenantManager) DecodeTableName(tableName []byte) (string, string, string, bool)

func (*TenantManager) EnsureDefaultNamespace 

func (m *TenantManager) EnsureDefaultNamespace(txMgr *transaction.Manager) error

func (*TenantManager) GetDatabaseAndCollectionId 

func (m *TenantManager) GetDatabaseAndCollectionId(db string, c string) (uint32, uint32)

GetDatabaseAndCollectionId returns the id of db and c in the default namespace. This is just a temporary API for the streams to know if database and collection exists at the start of streaming and their corresponding IDs.

func (*TenantManager) GetEncoder 

func (m *TenantManager) GetEncoder() Encoder

func (*TenantManager) GetNamespaceNames 

func (m *TenantManager) GetNamespaceNames() []string

func (*TenantManager) GetTableNameFromIds 

func (m *TenantManager) GetTableNameFromIds(tenantId uint32, dbId uint32, collId uint32) (string, string, string, bool)

GetTableNameFromIds returns tenant name, database name, collection name corresponding to their encoded ids.

func (*TenantManager) GetTenant 

func (m *TenantManager) GetTenant(ctx context.Context, namespaceName string, txMgr *transaction.Manager) (tenant *Tenant, err error)

GetTenant is responsible for returning the tenant from the cache. If the tenant is not available in the cache then this method will attempt to load it from the database and will update the tenant manager cache accordingly.

func (*TenantManager) ListNamespaces 

func (m *TenantManager) ListNamespaces(ctx context.Context, tx transaction.Tx) ([]Namespace, error)

ListNamespaces returns all the namespaces(tenants) exist in this cluster.

func (*TenantManager) Reload 

func (m *TenantManager) Reload(ctx context.Context, tx transaction.Tx) error

Reload reads all the tenants exist in the database and builds an in-memory view of the manager to track the tenants. As this is an expensive call, the reloading happens only during the start of the server. It is possible that reloading fails during start time then we rely on each transaction to detect it and trigger reload. The consistency shouldn’t be impacted if we fail to load the in-memory view.

type TenantNamespace 

type TenantNamespace struct {
	// contains filtered or unexported fields
}

TenantNamespace is used when there is a finer isolation of databases is needed. The caller provides a unique name and unique id to this namespace which is used by the cluster to create a namespace.

func NewTenantNamespace 

func NewTenantNamespace(name string, id uint32) *TenantNamespace

func (*TenantNamespace) Id 

func (n *TenantNamespace) Id() uint32

Id returns assigned id for the namespace

func (*TenantNamespace) Name 

func (n *TenantNamespace) Name() string

type Tracker 

type Tracker struct {
	// contains filtered or unexported fields
}

Tracker is an object attached to a transaction so that a transaction can identify if metadata is changed and whether it needs to restart the transaction. Tracker is not thread-safe and should be used only in a single session.

func (*Tracker) Stop 

func (tracker *Tracker) Stop(ctx context.Context) (bool, error)

Stop is needed to stop the tracker and reload the tenant if needed.

type Version 

type Version []byte

type VersionFuture 

type VersionFuture kv.Future

type VersionHandler 

type VersionHandler struct{}

VersionHandler is used to maintain a version for each schema change. Using this we can implement transactional DDL APIs. This will also be used to provide a strongly consistent Cache lookup on the schemas i.e. anytime version changes we know that a DDL operation is performed which means we can invalidate the cache and reload from the disk.

func (*VersionHandler) Increment 

func (m *VersionHandler) Increment(ctx context.Context, tx transaction.Tx) error

Increment is used to increment the metadata version

func (*VersionHandler) Read 

func (m *VersionHandler) Read(ctx context.Context, tx transaction.Tx, isSnapshot bool) (Version, error)

Read is blocking and returns the latest metadata version.

func (*VersionHandler) ReadFuture 

func (m *VersionHandler) ReadFuture(ctx context.Context, tx transaction.Tx, isSnapshot bool) (VersionFuture, error)

ReadFuture is a non-blocking API to return the future corresponding to the latest metadata version

func (*VersionHandler) ReadInOwnTxn 

func (m *VersionHandler) ReadInOwnTxn(ctx context.Context, txMgr *transaction.Manager, isSnapshot bool) (version Version, err error)

ReadInOwnTxn creates a transaction and then reads the version. This is useful when a transaction is also changing the metadata then it is better to read the metadata version in its own transaction as the read-write-read or write-read metadata version is not allowed in a transaction.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.