dax

package
v3.33.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 2, 2023 License: Apache-2.0, Apache-2.0 Imports: 17 Imported by: 0

README

DAX

DAX encapsulates anything which covers all of the services which make up the "disaggregation of storage and compute" project. Initially, this will include integration tests which pull in things like Metadata Services (MDS), including the Controller, as well as FeatureBase and IDK-based ingesters.

Setting up the tests

The DAX test currently requires docker images for: featurebase and datagen.

If at any point you run into problems with go mod failing to reference a private repo, make sure that you have gitlab.com/featurebasedb in your GOPRIVATE environment variable.

Note that during the docker image build step, go mod vendor is run, which creates a vendor directory in the root directory, and copies that to docker during the build stage. Just be aware of this; you may want to remove that vendor directory after you're done building docker images.

Possible Configuruation

The following were relevent when the dax code was in a separate repository. These may no longer be relevant.

I needed to but this in my ~/.profile file:

export GOPRIVATE=github.com/featurebasedb,gitlab.com/featurebasedb

And this in my ~/.gitconfig

[url "ssh://git@github.com/"]
	insteadOf = https://github.com/
[url "ssh://git@gitlab.com/"]
	insteadOf = https://gitlab.com/

Then make docker ran successfully.

Build the FeatureBase docker image
  • Check out the dax branch of the featurebase repository.
  • Run make docker-image-featurebase to build the docker image
  • You should now have an image in docker named dax/featurebase with the tag latest.
Build the Datagen docker image
  • cd <featurebase_repo_root>/idk
  • Run make docker-image-datagen to build the docker image
  • You should now have an image in docker named dax/datagen with the tag latest.

Running the tests

  • Check out the dax branch of the featurebase repository.
  • Change into the dax directory: cd dax
  • Run make test-integration.

Documentation

Overview

Package dax defines DAX domain level types.

Index

Constants

View Source
const (
	ServicePrefixComputer    = "computer"
	ServicePrefixController  = "controller"
	ServicePrefixQueryer     = "queryer"
	ServicePrefixSnapshotter = "snapshotter"
	ServicePrefixWritelogger = "writelogger"
)

ServicePrefixes are used as the service prefix value in http handlers.

View Source
const (
	ErrOrganizationIDDoesNotExist errors.Code = "OrganizationIDDoesNotExist"

	ErrDatabaseIDExists         errors.Code = "DatabaseIDExists"
	ErrDatabaseIDDoesNotExist   errors.Code = "DatabaseIDDoesNotExist"
	ErrDatabaseNameDoesNotExist errors.Code = "DatabaseNameDoesNotExist"

	ErrTableIDExists         errors.Code = "TableIDExists"
	ErrTableKeyExists        errors.Code = "TableKeyExists"
	ErrTableNameExists       errors.Code = "TableNameExists"
	ErrTableIDDoesNotExist   errors.Code = "TableIDDoesNotExist"
	ErrTableKeyDoesNotExist  errors.Code = "TableKeyDoesNotExist"
	ErrTableNameDoesNotExist errors.Code = "TableNameDoesNotExist"

	ErrFieldExists       errors.Code = "FieldExists"
	ErrFieldDoesNotExist errors.Code = "FieldDoesNotExist"

	ErrInvalidTransaction errors.Code = "InvalidTransaction"

	ErrUnimplemented errors.Code = "Unimplemented"
)
View Source
const (
	BaseTypeBool       = "bool"       //
	BaseTypeDecimal    = "decimal"    //
	BaseTypeID         = "id"         // non-keyed mutex
	BaseTypeIDSet      = "idset"      // non-keyed set
	BaseTypeIDSetQ     = "idsetq"     // non-keyed set timequantum
	BaseTypeInt        = "int"        //
	BaseTypeString     = "string"     // keyed mutex
	BaseTypeStringSet  = "stringset"  // keyed set
	BaseTypeStringSetQ = "stringsetq" // keyed set timequantum
	BaseTypeTimestamp  = "timestamp"  //

	DefaultPartitionN = 256

	PrimaryKeyFieldName = FieldName("_id")
)

Base types.

View Source
const (
	DatabaseOptionWorkersMin = "workers-min"
	DatabaseOptionWorkersMax = "workers-max"
)
View Source
const (
	ErrNodeDoesNotExist errors.Code = "NodeDoesNotExist"
)
View Source
const PrefixDatabase = "db"

PrefixDatabase is used as a prefix to DatabaseKey strings because FeatureBase indexes must start with an alpha (a-z) character. Because the string representation of a uuid (i.e. the OrganizationID value) can start with a numeric value, we can't have OrganizationId (or any of the other ID values which make up the DatabaseKey) be at the beginning of the DatabaseKey.

View Source
const PrefixTable = "tbl"

PrefixTable is used as a prefix to TableKey strings because FeatureBase indexes must start with an alpha (a-z) character. Because the string representation of a uuid (i.e. the OrganizationID value) can start with a numeric value, we can't have OrganizationId (or any of the other ID values which make up the TableKey) be at the beginning of the TableKey.

View Source
const TableKeyDelimiter = "__"

TableKeyDelimiter is used to delimit the qualifier elements in the TableKey. While it might make more sense to use a pipe ("|") here, we instead use a double underscore because underscore is one of the few characters allowed by the FeatureBase index name restrictions, and we double it in a lame attempt to distinquish it from FeatureBase index names which contain a single underscore.

Variables

This section is empty.

Functions

func NewErrDatabaseIDDoesNotExist added in v3.29.0

func NewErrDatabaseIDDoesNotExist(qdbid QualifiedDatabaseID) error

func NewErrDatabaseIDExists added in v3.29.0

func NewErrDatabaseIDExists(qdbid QualifiedDatabaseID) error

func NewErrDatabaseNameDoesNotExist added in v3.29.0

func NewErrDatabaseNameDoesNotExist(dbName DatabaseName) error

func NewErrFieldDoesNotExist

func NewErrFieldDoesNotExist(fieldName FieldName) error

func NewErrFieldExists

func NewErrFieldExists(fieldName FieldName) error

func NewErrInvalidTransaction added in v3.29.0

func NewErrInvalidTransaction() error

func NewErrNodeDoesNotExist

func NewErrNodeDoesNotExist(addr Address) error

func NewErrOrganizationIDDoesNotExist added in v3.33.0

func NewErrOrganizationIDDoesNotExist(orgID OrganizationID) error

func NewErrTableIDDoesNotExist

func NewErrTableIDDoesNotExist(qtid QualifiedTableID) error

func NewErrTableIDExists

func NewErrTableIDExists(qtid QualifiedTableID) error

func NewErrTableKeyDoesNotExist

func NewErrTableKeyDoesNotExist(tkey TableKey) error

func NewErrTableKeyExists

func NewErrTableKeyExists(tkey TableKey) error

func NewErrTableNameDoesNotExist

func NewErrTableNameDoesNotExist(tableName TableName) error

func NewErrTableNameExists

func NewErrTableNameExists(tableName TableName) error

func NewNopController added in v3.29.0

func NewNopController() *nopController

func NewNopNodeService added in v3.29.0

func NewNopNodeService() *nopNodeService

func NewNopNoder added in v3.27.0

func NewNopNoder() *nopNoder

Types

type Address

type Address string

Address is a string of the form [scheme]://[host]:[port]/path

func (Address) Host

func (a Address) Host() string

Host returns the [host] portion of the Address.

func (Address) HostPort

func (a Address) HostPort() string

HostPort returns the [host]:[port] portion of the Address; in other words, the Address stripped of any scheme and path.

func (Address) OverrideScheme

func (a Address) OverrideScheme(scheme string) string

OverrideScheme overrides Address's current scheme with the one provided. If an empty scheme is provided, OverrideScheme will return just the host:port/path.

func (Address) Path

func (a Address) Path() string

Path returns the path portion of the Address.

func (Address) Port

func (a Address) Port() uint16

Port returns the [port] portion of the Address. If the port values is invalid or does not exist, the returned value will default to 0.

func (Address) Scheme

func (a Address) Scheme() string

Scheme returns the [scheme] portion of the Address. This may be an empty string if Address does not contain a scheme.

func (Address) String

func (a Address) String() string

String returns the Address as a string type.

func (Address) WithScheme

func (a Address) WithScheme(dflt string) string

WithScheme ensures that the string returned contains the scheme portion of a URL. Because an Address may not have a scheme (for example, it could be just "host:80"), this method can be applied to an address when it needs to be used as a URL. If the address's existing scheme is blank, the default scheme provided will be used. If address is blank, the default scheme will not be added; i.e. address will remain blank.

type AddressManager

type AddressManager interface {
	AddAddresses(context.Context, ...Address) error
	RemoveAddresses(context.Context, ...Address) error
}

AddressManager is an interface for any service which needs to maintain a list of addresses, and receive add/remove address requests from other services.

type Addresses added in v3.29.0

type Addresses []Address

Addresses is a sortable slice of Address.

func (Addresses) Len added in v3.29.0

func (a Addresses) Len() int

func (Addresses) Less added in v3.29.0

func (a Addresses) Less(i, j int) bool

func (Addresses) Swap added in v3.29.0

func (a Addresses) Swap(i, j int)

type AssignedNode

type AssignedNode struct {
	Address Address `json:"address"`
	Role    Role    `json:"role"`
}

AssignedNode represents a Worker which has been assigned a role. Note that the worker which it represents might be responsible for multiple roles, but AssignedNode only ever represents one of those roles at a time. This is because it is always the response of a RoleType-specific request.

type BaseType

type BaseType string

BaseType is a typed string used for field types.

func BaseTypeFromString

func BaseTypeFromString(s string) (BaseType, error)

BaseTypeFromString converts a string to one of the defined BaseTypes. If the string does not match a BaseType, then an error is returned.

type ComputeNode added in v3.27.0

type ComputeNode struct {
	Address Address   `json:"address"`
	Table   TableKey  `json:"table"`
	Shards  ShardNums `json:"shards"`
}

ComputeNode represents a compute node and the table/shards for which it is responsible.

type ComputeRole

type ComputeRole struct {
	TableKey TableKey  `json:"table-key"`
	Shards   ShardNums `json:"shards"`
}

ComputeRole is a role specific to compute nodes.

func (*ComputeRole) Type

func (cr *ComputeRole) Type() RoleType

Type returns the type for ComputeRole. This is mainly to implement the Role interface.

type ComputerService

type ComputerService interface {
	MultiService

	SetController(Address) error
}

type Controller added in v3.29.0

type Controller interface {
	Noder
	Schemar
}

type ControllerService added in v3.30.0

type ControllerService interface {
	Service
}

type Database added in v3.29.0

type Database struct {
	ID      DatabaseID      `json:"id"`
	Name    DatabaseName    `json:"name"`
	Options DatabaseOptions `json:"options"`

	Description string `json:"description,omitempty"`
	Owner       string `json:"owner,omitempty"`
	CreatedAt   int64  `json:"createdAt,omitempty"`
	UpdatedAt   int64  `json:"updatedAt,omitempty"`
	UpdatedBy   string `json:"updatedBy,omitempty"`
}

Database represents a database and its configuration.

func (*Database) CreateID added in v3.32.0

func (d *Database) CreateID() (DatabaseID, error)

CreateID generates a unique identifier for Database. If Database has already been assigned an ID, then this no-ops. The reason for this is that the cloud implementation of FeatureBase may allocate an ID before calling CreateDatabase on the controller.

type DatabaseID

type DatabaseID string

DatabaseID is the unique database identifier, currently generated by the Control Plane in a FeatureBase cloud implementation. In that implementation, its value is a uuid as a string, but there's nothing enforcing that; the value could be any string.

type DatabaseIDs added in v3.29.0

type DatabaseIDs []DatabaseID

DatabaseIDs is a sortable slice of DatabaseID.

func (DatabaseIDs) Len added in v3.29.0

func (s DatabaseIDs) Len() int

func (DatabaseIDs) Less added in v3.29.0

func (s DatabaseIDs) Less(i, j int) bool

func (DatabaseIDs) Swap added in v3.29.0

func (s DatabaseIDs) Swap(i, j int)

type DatabaseKey added in v3.29.0

type DatabaseKey string

DatabaseKey is a globally unique identifier for a database; it is effectively the compound key: (org, database). This is (hopefully) the value that will be used when interfacing with services which are unaware of qualifiers.

func (DatabaseKey) QualifiedDatabaseID added in v3.29.0

func (dk DatabaseKey) QualifiedDatabaseID() QualifiedDatabaseID

QualifiedDatabaseID returns the QualifiedDatabaseID based on the key. If DatabaseKey can't be parsed into a valid (i.e. complete) QualifiedDatabaseID, then blank values are used where necessary.

type DatabaseName added in v3.29.0

type DatabaseName string

DatabaseName is a human-friendly string.

type DatabaseOption added in v3.32.0

type DatabaseOption string

DatabaseOption is a string key representing a database option.

type DatabaseOptions added in v3.29.0

type DatabaseOptions struct {
	WorkersMin int `json:"workers-min"`
	WorkersMax int `json:"workers-max"`
}

DatabaseOptions are used to configure a database.

func (*DatabaseOptions) Set added in v3.32.0

func (opts *DatabaseOptions) Set(option string, value string) error

Set sets the specified option to the provided value.

type Directive

type Directive struct {
	Address Address `json:"address"`

	// Method describes how the compute node should handle the Directive. See
	// the different constants of type DirectiveMethod for how this value is
	// handled.
	Method DirectiveMethod `json:"method"`

	Tables []*QualifiedTable `json:"schema"`

	ComputeRoles   []ComputeRole   `json:"compute-roles"`
	TranslateRoles []TranslateRole `json:"translate-roles"`

	Version uint64 `json:"version"`
}

Directive contains the instructions, sent from the Controller, which a compute node is to follow. A Directive is typically JSON-encoded and POSTed to a compute node's `/directive` endpoint.

func (*Directive) ComputeShards

func (d *Directive) ComputeShards(tbl TableKey) ShardNums

ComputeShards returns the list of shards, for the given table, for which this compute node is responsible. It assumes that the Directive does not contain more than one ComputeRole for the same table; in that case, we would need to return the union of Shards.

func (*Directive) ComputeShardsMap

func (d *Directive) ComputeShardsMap() map[TableKey]ShardNums

ComputeShardsMap returns a map of table to shards. It assumes that the Directive does not contain more than one ComputeRole for the same table; in that case, we would need to return the union of Shards.

func (*Directive) IsEmpty

func (d *Directive) IsEmpty() bool

IsEmpty tells whether a directive is assigning actual responsibilty to a node or not. If the directive does not assign responsibility for any shard or partition then it is considered empty. This is used to determine whether we can ignore an error received from applying this directive (an empty directive is often sent to a node which is already down).

func (*Directive) Table

func (d *Directive) Table(qtid QualifiedTableID) (*QualifiedTable, error)

Table returns the ID'd table from the Directive's Tables list. If it's not found, it returns nil and a non-nil error. A nil error guarantees that the returned table is non-nil.

func (*Directive) TranslateFieldsMap

func (d *Directive) TranslateFieldsMap() map[TableKey][]FieldName

TranslateFieldsMap returns a map of table to fields. It assumes that the Directive does not contain more than one TranslateRole for the same table; in that case, we would need to return the union of FieldValues.

func (*Directive) TranslatePartitions

func (d *Directive) TranslatePartitions(tbl TableKey) PartitionNums

TranslatePartitions returns the list of partitions, for the given table, for which this translate node is responsible. It assumes that the Directive does not contain more than one TranslateRole for the same table; in that case, we would need to return the union of Shards.

func (*Directive) TranslatePartitionsMap

func (d *Directive) TranslatePartitionsMap() map[TableKey]PartitionNums

TranslatePartitionsMap returns a map of table to partitions. It assumes that the Directive does not contain more than one TranslateRole for the same table; in that case, we would need to return the union of Partitions.

type DirectiveMethod

type DirectiveMethod string

DirectiveMethod is used to tell the compute node how it should handle the Directive.

const (
	// DirectiveMethodDiff tells the compute node to diff the Directive with its
	// local, cached Directive and only apply the differences.
	DirectiveMethodDiff DirectiveMethod = "diff"

	// DirectiveMethodReset tells the compute node to delete all of its existing
	// data before applying the directive.
	DirectiveMethodReset DirectiveMethod = "reset"

	// DirectiveMethodSnapshot tells the compute node that the incoming
	// Directive should only contain data version updates related to a snapshot
	// request.
	DirectiveMethodSnapshot DirectiveMethod = "snapshot"
)

type DirectiveVersion

type DirectiveVersion interface {
	Increment(tx Transaction, delta uint64) (uint64, error)
}

type Directives

type Directives []*Directive

Directives is a sortable slice of Directive.

func (Directives) Len

func (d Directives) Len() int

func (Directives) Less

func (d Directives) Less(i, j int) bool

func (Directives) Swap

func (d Directives) Swap(i, j int)

type Field

type Field struct {
	Name    FieldName    `json:"name"`
	Type    BaseType     `json:"type"`
	Options FieldOptions `json:"options"`

	CreatedAt int64 `json:"createdAt,omitempty"`
}

Field represents a field and its configuration.

func (*Field) CreateSQL

func (f *Field) CreateSQL() string

CreateSQL returns the SQL representation of the field to be used in a CREATE TABLE statement.

func (*Field) IsPrimaryKey

func (f *Field) IsPrimaryKey() bool

IsPrimaryKey returns true if the field is the primary key field (of either type ID or STRING).

func (*Field) String

func (f *Field) String() string

String returns the field name as a string.

func (*Field) StringKeys

func (f *Field) StringKeys() bool

StringKeys returns true if the field uses string keys.

type FieldName

type FieldName string

FieldName is a typed string used for field names.

type FieldOptions

type FieldOptions struct {
	Min            pql.Decimal   `json:"min,omitempty"`
	Max            pql.Decimal   `json:"max,omitempty"`
	Scale          int64         `json:"scale,omitempty"`
	NoStandardView bool          `json:"no-standard-view,omitempty"` // TODO: we should remove this
	CacheType      string        `json:"cache-type,omitempty"`
	CacheSize      uint32        `json:"cache-size,omitempty"`
	TimeUnit       string        `json:"time-unit,omitempty"`
	Epoch          time.Time     `json:"epoch,omitempty"`
	TimeQuantum    TimeQuantum   `json:"time-quantum,omitempty"`
	TTL            time.Duration `json:"ttl,omitempty"`
	ForeignIndex   string        `json:"foreign-index,omitempty"`
}

FieldOptions represents options to set when initializing a field.

type Job

type Job string

Job is a generic identifier used to represent a specific role assigned to a worker.

func (Job) Job added in v3.29.0

func (j Job) Job() Job

Job allows a Job to implement the Jobber interface.

type Jobber added in v3.29.0

type Jobber interface {
	Job() Job
}

type Jobs

type Jobs []Job

Jobs is a slice of Job.

type MultiService

type MultiService interface {
	Service

	SetKey(ServiceKey)
	Key() ServiceKey
}

MultiService is a service type which can have multiple instances within ServicesManager.

type Node

type Node struct {
	Address Address `json:"address"`

	RoleTypes []RoleType `json:"role-types"`
}

Node is used in API requests, like RegisterNode (before being assigned roles).

type NodeService

type NodeService interface {
	CreateNode(context.Context, Address, *Node) error
	ReadNode(context.Context, Address) (*Node, error)
	DeleteNode(context.Context, Address) error
	Nodes(context.Context) ([]*Node, error)
}

NodeService represents a service for managing Nodes.

type Noder added in v3.27.0

type Noder interface {
	ComputeNodes(ctx context.Context, qtid QualifiedTableID, shards ...ShardNum) ([]ComputeNode, error)
	TranslateNodes(ctx context.Context, qtid QualifiedTableID, partitions ...PartitionNum) ([]TranslateNode, error)

	// IngestPartition is effectively the "write" version of TranslateNodes. Its
	// implementations will return the same Address that TranslateNodes would,
	// but it includes the logic to create/assign the partition if it is not
	// already being handled by a computer.
	IngestPartition(ctx context.Context, qtid QualifiedTableID, partition PartitionNum) (Address, error)

	// IngestShard is effectively the "write" version of ComputeNodes. Its
	// implementations will return the same Address that ComputeNodes would, but
	// it includes the logic to create/assign the shard if it is not already
	// being handled by a computer.
	IngestShard(ctx context.Context, qtid QualifiedTableID, shard ShardNum) (Address, error)
}

type Nodes added in v3.29.0

type Nodes []*Node

Nodes is a slice of *Node. It's useful for printing the nodes as a list of node.Addresses via its String() method.

func (Nodes) String added in v3.29.0

func (n Nodes) String() string

String prints the slice of node addresses in Nodes.

type NopAddressManager

type NopAddressManager struct{}

NopAddressManager is a no-op implementation of the AddressManager interface.

func NewNopAddressManager

func NewNopAddressManager() *NopAddressManager

func (*NopAddressManager) AddAddresses

func (a *NopAddressManager) AddAddresses(ctx context.Context, addrs ...Address) error

func (*NopAddressManager) RemoveAddresses

func (a *NopAddressManager) RemoveAddresses(ctx context.Context, addrs ...Address) error

type NopSchemar added in v3.27.0

type NopSchemar struct{}

NopSchemar is a no-op implementation of the Schemar interface.

func NewNopSchemar added in v3.27.0

func NewNopSchemar() *NopSchemar

func (*NopSchemar) CreateDatabase added in v3.29.0

func (s *NopSchemar) CreateDatabase(context.Context, *QualifiedDatabase) error

func (*NopSchemar) CreateField added in v3.27.0

func (s *NopSchemar) CreateField(ctx context.Context, qtid QualifiedTableID, fld *Field) error

func (*NopSchemar) CreateTable added in v3.27.0

func (s *NopSchemar) CreateTable(ctx context.Context, qtbl *QualifiedTable) error

func (*NopSchemar) DatabaseByID added in v3.29.0

func (s *NopSchemar) DatabaseByID(ctx context.Context, qdbid QualifiedDatabaseID) (*QualifiedDatabase, error)

func (*NopSchemar) DatabaseByName added in v3.29.0

func (s *NopSchemar) DatabaseByName(ctx context.Context, orgID OrganizationID, dbname DatabaseName) (*QualifiedDatabase, error)

func (*NopSchemar) Databases added in v3.29.0

func (*NopSchemar) DropDatabase added in v3.29.0

func (*NopSchemar) DropField added in v3.27.0

func (s *NopSchemar) DropField(ctx context.Context, qtid QualifiedTableID, fld FieldName) error

func (*NopSchemar) DropTable added in v3.27.0

func (s *NopSchemar) DropTable(ctx context.Context, qtid QualifiedTableID) error

func (*NopSchemar) SetDatabaseOption added in v3.32.0

func (s *NopSchemar) SetDatabaseOption(ctx context.Context, qdbid QualifiedDatabaseID, option string, value string) error

func (*NopSchemar) TableByID added in v3.27.0

func (s *NopSchemar) TableByID(ctx context.Context, qtid QualifiedTableID) (*QualifiedTable, error)

func (*NopSchemar) TableByName added in v3.27.0

func (*NopSchemar) Tables added in v3.27.0

func (s *NopSchemar) Tables(ctx context.Context, qdbid QualifiedDatabaseID, tids ...TableID) ([]*QualifiedTable, error)

type OrganizationID

type OrganizationID string

OrganizationID is the unique organization identifier, currently generated by the Control Plane in a FeatureBase cloud implementation. In that implementation, its value is a uuid as a string, but there's nothing enforcing that; the value could be any string.

type PartitionNum

type PartitionNum int

PartitionNum is the numerical (int) partition value.

func (PartitionNum) String

func (p PartitionNum) String() string

String returns the PartitionNum as a string.

type PartitionNums

type PartitionNums []PartitionNum

PartitionNums is a slice of PartitionNum.

func NewPartitionNums added in v3.27.0

func NewPartitionNums(nums ...uint64) PartitionNums

func (PartitionNums) Len

func (p PartitionNums) Len() int

func (PartitionNums) Less

func (p PartitionNums) Less(i, j int) bool

func (PartitionNums) Swap

func (p PartitionNums) Swap(i, j int)

type QualifiedDatabase added in v3.29.0

type QualifiedDatabase struct {
	OrganizationID OrganizationID `json:"org-id"`
	Database
}

QualifiedDatabase is a Database along with its OrganizationID.

func NewQualifiedDatabase added in v3.32.0

func NewQualifiedDatabase(orgID OrganizationID, db *Database) *QualifiedDatabase

NewQualifiedDatabase returns the db as a QualifiedDatabase with the provided OrganizationID.

func (QualifiedDatabase) Key added in v3.29.0

func (qdb QualifiedDatabase) Key() DatabaseKey

Key returns the string-encoded (delimited by DatabaseKeyDelimiter) globally unique DatabaseKey.

func (*QualifiedDatabase) QualifiedID added in v3.29.0

func (qdb *QualifiedDatabase) QualifiedID() QualifiedDatabaseID

QualifiedID returns the QualifiedDatabaseID for the database.

func (QualifiedDatabase) String added in v3.29.0

func (qdb QualifiedDatabase) String() string

String returns a human-friendly version of the QualifiedDatabase. It is only used for display purposes; it is not used as any kind of key.

type QualifiedDatabaseID added in v3.29.0

type QualifiedDatabaseID struct {
	OrganizationID OrganizationID `json:"org-id"`
	DatabaseID     DatabaseID     `json:"db-id"`
}

QualifiedDatabaseID is a DatabaseID along with its OrganizationID.

func NewQualifiedDatabaseID added in v3.29.0

func NewQualifiedDatabaseID(orgID OrganizationID, dbID DatabaseID) QualifiedDatabaseID

NewQualifiedDatabaseID is a helper function used to create a QualifiedDatabaseID from the provided arguments.

func QualifiedDatabaseIDFromKey added in v3.29.0

func QualifiedDatabaseIDFromKey(key string) (QualifiedDatabaseID, error)

QualifiedDatabaseIDFromKey decodes a string key into a QualifiedDatabaseID. The key is assumed to have been encoded using the QualifiedDatabaseID.Key() method.

func (QualifiedDatabaseID) Key added in v3.29.0

func (qdbid QualifiedDatabaseID) Key() DatabaseKey

Key returns the string-encoded (delimited by TableKeyDelimiter) globally unique DatabaseKey. The key has a prefix because FeatureBase index name restrictions require the name to start with a non-numeric value, and since a uuid can contain a number as its first character, we have to prefix it with something.

func (QualifiedDatabaseID) String added in v3.29.0

func (qdbid QualifiedDatabaseID) String() string

String returns a human-friendly version of the QualifiedDatabaseID. It is only used for display purposes; it is not used as any kind of key. For that, see the QualifiedDatabaseID.Key() method and the DatabaseKey type.

type QualifiedDatabaseIDs added in v3.29.0

type QualifiedDatabaseIDs []QualifiedDatabaseID

QualifiedDatabaseIDs is a list of QualifiedDatabaseID.

func (QualifiedDatabaseIDs) Len added in v3.29.0

func (s QualifiedDatabaseIDs) Len() int

func (QualifiedDatabaseIDs) Less added in v3.29.0

func (s QualifiedDatabaseIDs) Less(i, j int) bool

func (QualifiedDatabaseIDs) Swap added in v3.29.0

func (s QualifiedDatabaseIDs) Swap(i, j int)

type QualifiedDatabases added in v3.29.0

type QualifiedDatabases []*QualifiedDatabase

type QualifiedTable

type QualifiedTable struct {
	QualifiedDatabaseID
	Table
}

QualifiedTable wraps Table and includes a QualifiedDatabaseID.

func NewQualifiedTable

func NewQualifiedTable(qdbid QualifiedDatabaseID, tbl *Table) *QualifiedTable

NewQualifiedTable returns the tbl as a QualifiedTable with the provided QualifiedDatabaseID.

func (QualifiedTable) Key

func (qt QualifiedTable) Key() TableKey

Key returns the string-encoded (delimited by TableKeyDelimiter) globally unique TableKey.

func (*QualifiedTable) QualifiedID

func (qt *QualifiedTable) QualifiedID() QualifiedTableID

QualifiedID returns the QualifiedTableID for the table.

func (*QualifiedTable) Qualifier

func (qt *QualifiedTable) Qualifier() QualifiedDatabaseID

Qualifier returns the QualifiedDatabaseID portion of the QualifiedTable.

func (QualifiedTable) String

func (qt QualifiedTable) String() string

String returns a human-friendly version of the QualifiedTable. It is only used for display purposes; it is not used as any kind of key.

type QualifiedTableID

type QualifiedTableID struct {
	QualifiedDatabaseID
	ID   TableID   `json:"id"`
	Name TableName `json:"name"`
}

QualifiedTableID is a globally unique table identifier. It is a sub-set of a QualifiedTable (i.e. it's just the identification portion). Most things will take a Name or an ID and do the right thing™.

func NewQualifiedTableID

func NewQualifiedTableID(qdbid QualifiedDatabaseID, tid TableID) QualifiedTableID

NewQualifiedTableID is a helper function used to create a QualifiedTableID from the provided arguments.

func QualifiedTableIDFromKey

func QualifiedTableIDFromKey(key string) (QualifiedTableID, error)

QualifiedTableIDFromKey decodes a string key into a QualifiedTableID. The key is assumed to have been encoded using the QualifiedTableID.Key() method.

func (QualifiedTableID) Equals

func (qtid QualifiedTableID) Equals(other QualifiedTableID) bool

Equals returns true if `other` is the same as qtid. Note: the `Name` value is ignored in this comparison; only `QualifiedDatabaseID` and `ID` are considered.

func (QualifiedTableID) Key

func (qtid QualifiedTableID) Key() TableKey

Key returns the string-encoded (delimited by TableKeyDelimiter) globally unique TableKey. The key has a prefix because FeatureBase index name restrictions require the name to start with a non-numeric value, and since a uuid can contain a number as its first character, we have to prefix it with something.

func (QualifiedTableID) Qualifier added in v3.29.0

func (qtid QualifiedTableID) Qualifier() QualifiedDatabaseID

Qualifier returns the QualifiedDatabaseID (qdbid) portion of the QualifiedTableID (qtid).

func (QualifiedTableID) String

func (qtid QualifiedTableID) String() string

String returns a human-friendly version of the QualifiedDatabaseID. It is only used for display purposes; it is not used as any kind of key. For that, see the QualifiedDatabaseID.Key() method.

type QualifiedTables

type QualifiedTables []*QualifiedTable

QualifiedTables is a sortable slice of QualifiedTable.

func (QualifiedTables) Len

func (o QualifiedTables) Len() int

func (QualifiedTables) Less

func (o QualifiedTables) Less(i, j int) bool

func (QualifiedTables) Swap

func (o QualifiedTables) Swap(i, j int)

type QueryerService

type QueryerService interface {
	Service

	SetController(Address) error
}

type Role

type Role interface {
	Type() RoleType
}

Role is an interface for any role which a worker node can assume.

type RoleType

type RoleType string

RoleType represents a role type which a worker node can act as.

const (
	RoleTypeCompute   RoleType = "compute"
	RoleTypeTranslate RoleType = "translate"
)

type RoleTypes

type RoleTypes []RoleType

RoleTypes is a list of RoleType, used primarily to introduce helper methods on the list.

func (RoleTypes) Contains

func (rt RoleTypes) Contains(t RoleType) bool

Contains returns true if the given RoleType is in RoleTypes.

type Schema

type Schema struct {
	Tables []*Table
}

Schema contains a list of Tables.

func (*Schema) Table

func (s *Schema) Table(name TableName) (*Table, bool)

Table returns the table with the provided name. If a table with that name does not exist, the returned boolean will be false.

type Schemar added in v3.27.0

type Schemar interface {
	CreateDatabase(context.Context, *QualifiedDatabase) error
	DropDatabase(context.Context, QualifiedDatabaseID) error

	DatabaseByName(ctx context.Context, orgID OrganizationID, dbname DatabaseName) (*QualifiedDatabase, error)
	DatabaseByID(ctx context.Context, qdbid QualifiedDatabaseID) (*QualifiedDatabase, error)

	SetDatabaseOption(ctx context.Context, qdbid QualifiedDatabaseID, option string, value string) error

	// Databases returns a list of databases. If the list of DatabaseIDs is
	// empty, all databases will be returned. If greater than zero DatabaseIDs
	// are passed in the second argument, only databases matching those IDs will
	// be returned.
	Databases(context.Context, OrganizationID, ...DatabaseID) ([]*QualifiedDatabase, error)

	CreateTable(ctx context.Context, qtbl *QualifiedTable) error
	DropTable(ctx context.Context, qtid QualifiedTableID) error

	TableByName(ctx context.Context, qdbid QualifiedDatabaseID, tname TableName) (*QualifiedTable, error)
	TableByID(ctx context.Context, qtid QualifiedTableID) (*QualifiedTable, error)

	// Tables returns a list of tables. If the qualifiers DatabaseID is empty,
	// all tables in the org will be returned. If the OrganizationID is empty,
	// all tables will be returned. If both are populated, only tables in that
	// database will be returned. If greater than zero table IDs are passed in
	// the third argument, only tables matching those IDs will be returned.
	Tables(ctx context.Context, qdbid QualifiedDatabaseID, tids ...TableID) ([]*QualifiedTable, error)

	CreateField(ctx context.Context, qtid QualifiedTableID, fld *Field) error
	DropField(ctx context.Context, qtid QualifiedTableID, fname FieldName) error
}

Schemar is similar to the pilosa.SchemaAPI interface, but it takes QualifiedDatabaseIDs into account. Note that it is also similar to the schemar.Schemar interface, but that is used internally, typically within the Controller, and it takes Transactions rather than a Context, because its methods are assumed to be used as part of a larger request. TODO(tlt): clean up the mds/controller/schemar Schemar interface confusion.

type Service

type Service interface {
	Start() error
	Stop() error
	Address() Address
	HTTPHandler() http.Handler
}

Service is an interface implemented by any service which is part of ServiceManager.

type ServiceKey

type ServiceKey string

ServiceKey is a unique key used to identify one service managed by the ServiceManager. These typically align with the ServicePrefix* values.

type ServiceManager

type ServiceManager struct {

	// Controller
	Controller ControllerService

	// Queryer
	Queryer QueryerService

	Logger logger.Logger
	// contains filtered or unexported fields
}

ServiceManager manages the various services running in process. It is used to do things like start/stop services, and it dynamically builds the http router depending on the state of all services.

func NewServiceManager

func NewServiceManager() *ServiceManager

NewServiceManager returns a new ServiceManager with default values.

func (*ServiceManager) AddComputer

func (s *ServiceManager) AddComputer(cs ComputerService) ServiceKey

AddComputer adds the provided ComputerService to ServiceManager. It assigns the service a unique ServiceKey.

func (*ServiceManager) Computer

func (s *ServiceManager) Computer(key ServiceKey) ComputerService

Computer returns the ComputerService specified by the provided key.

func (*ServiceManager) ComputerStart

func (s *ServiceManager) ComputerStart(key ServiceKey) error

ComputerStart starts the Computer service specified by the provided key.

func (*ServiceManager) ComputerStop

func (s *ServiceManager) ComputerStop(key ServiceKey) error

ComputerStop stops the Computer service specified by the provided key.

func (*ServiceManager) Computers

func (s *ServiceManager) Computers() map[ServiceKey]ComputerService

Computers returns a map (keyed by ServiceKey) of all computers registered with ServiceManager.

func (*ServiceManager) ControllerStart added in v3.30.0

func (s *ServiceManager) ControllerStart() error

ControllerStart starts the Controller service.

func (*ServiceManager) ControllerStop added in v3.30.0

func (s *ServiceManager) ControllerStop() error

ControllerStop stops the Controller service.

func (*ServiceManager) HTTPHandler

func (s *ServiceManager) HTTPHandler() http.Handler

HTTPHandler returns the current http.Handler for ServiceManager based on the state of its services.

func (*ServiceManager) QueryerStart

func (s *ServiceManager) QueryerStart() error

QueryerStart starts the Queryer service.

func (*ServiceManager) QueryerStop

func (s *ServiceManager) QueryerStop() error

QueryerStop stops the Queryer service.

func (*ServiceManager) RemoveComputer

func (s *ServiceManager) RemoveComputer(key ServiceKey) bool

RemoveComputer removes the ComputerService specified by the provided key.

func (*ServiceManager) StartAll

func (s *ServiceManager) StartAll() error

StartAll starts all services which have been added to ServiceManager.

func (*ServiceManager) StopAll added in v3.27.0

func (s *ServiceManager) StopAll() error

type Set

type Set[K ~string] map[K]struct{}

Set is a set of stringy items.

func NewSet

func NewSet[K ~string](stuff ...K) Set[K]

func (Set[K]) Add

func (s Set[K]) Add(k K)

Add adds k to the set.

func (Set[K]) Contains

func (s Set[K]) Contains(k K) bool

Contains returns true if k is in the set.

func (Set[K]) Copy

func (s Set[K]) Copy() Set[K]

Copy creates a copy of the set.

func (Set[K]) Count

func (s Set[K]) Count() int

Count returns the number of items in the set.

func (Set[K]) Merge

func (s Set[K]) Merge(s2 Set[K])

Merge adds the members of s2 to s.

func (Set[K]) Minus

func (s Set[K]) Minus(s2 Set[K]) Set[K]

Minus returns the a copy of s without any members which are also in s2.

func (Set[K]) Plus

func (s Set[K]) Plus(s2 Set[K]) Set[K]

Plus returns a copy of s that also contains all members of s2.

func (Set[K]) Remove

func (s Set[K]) Remove(k K)

Remove removes k from the set.

func (Set[K]) RemoveByPrefix added in v3.29.0

func (s Set[K]) RemoveByPrefix(prefix string) []K

RemoveByPrefix removes all items from Set that have the given prefix.

func (Set[K]) Slice

func (s Set[K]) Slice() []K

Slice returns a slice containing each member of the set in an undefined order.

func (Set[K]) Sorted

func (s Set[K]) Sorted() []K

Sorted returns Set[K] as a sorted slice of K.

type ShardNum

type ShardNum uint64

ShardNum is the numerical (uint64) shard value.

func (ShardNum) String

func (s ShardNum) String() string

type ShardNums

type ShardNums []ShardNum

ShardNums is a slice of ShardNum.

func NewShardNums added in v3.27.0

func NewShardNums(nums ...uint64) ShardNums

func (ShardNums) Len added in v3.27.0

func (s ShardNums) Len() int

func (ShardNums) Less added in v3.27.0

func (s ShardNums) Less(i, j int) bool

func (ShardNums) Swap added in v3.27.0

func (s ShardNums) Swap(i, j int)

type SnapshotFieldKeysRequest

type SnapshotFieldKeysRequest struct {
	Address Address `json:"address"`

	TableKey TableKey  `json:"table-key"`
	Field    FieldName `json:"field"`
}

type SnapshotShardDataRequest

type SnapshotShardDataRequest struct {
	Address Address `json:"address"`

	TableKey TableKey `json:"table-key"`
	ShardNum ShardNum `json:"shard"`
}

type SnapshotTableKeysRequest

type SnapshotTableKeysRequest struct {
	Address Address `json:"address"`

	TableKey     TableKey     `json:"table-key"`
	PartitionNum PartitionNum `json:"partition"`
}

type StringTableKeyer added in v3.27.0

type StringTableKeyer string

StringTableKeyer is a helper type which can wrap a string, making it a TableKeyer. This is useful for certain calls to Execute() which take a string index name.

func (StringTableKeyer) Key added in v3.27.0

func (s StringTableKeyer) Key() TableKey

type Table

type Table struct {
	ID         TableID   `json:"id,omitempty"`
	Name       TableName `json:"name,omitempty"`
	Fields     []*Field  `json:"fields"`
	PartitionN int       `json:"partitionN"`

	Description string `json:"description,omitempty"`
	Owner       string `json:"owner,omitempty"`
	CreatedAt   int64  `json:"createdAt,omitempty"`
	UpdatedAt   int64  `json:"updatedAt,omitempty"`
	UpdatedBy   string `json:"updatedBy,omitempty"`
}

Table represents a table and its configuration.

func NewTable

func NewTable(name TableName) *Table

NewTable returns a new instance of table with a pseudo-random ID which is assumed to be unique within the scope of a QualifiedDatabaseID.

func (*Table) CreateID

func (t *Table) CreateID() (TableID, error)

CreateID generates a unique identifier for Table. If Table has already been assigned an ID, then an error is returned.

func (*Table) CreateSQL

func (t *Table) CreateSQL() string

CreateSQL returns the SQL CREATE TABLE string necessary to create the table.

func (*Table) Field

func (t *Table) Field(name FieldName) (*Field, bool)

Field returns the field with the provided name. If a field with that name does not exist, the returned boolean will be false.

func (*Table) FieldNames

func (t *Table) FieldNames() []FieldName

FieldNames returns the list of field names associated with the table.

func (*Table) HasValidPrimaryKey

func (t *Table) HasValidPrimaryKey() bool

HasValidPrimaryKey returns false if the table does not contain a primary key field (which is required), or if the primary key field is not a valid type.

func (*Table) Key added in v3.27.0

func (t *Table) Key() TableKey

func (*Table) RemoveField

func (t *Table) RemoveField(name FieldName) bool

RemoveField removes the given field by name. It returns true if the field was removed.

func (*Table) StringKeys

func (t *Table) StringKeys() bool

StringKeys returns true if the table's primary key is either a string or a concatenation of fields.

type TableID

type TableID string

TableID is a table identifier. It is unique within the scope of a QualifiedDatabaseID. Coupled with a QualifiedDatabaseID, it makes up a QualifiedTableID and, when encoded as a string, a TableKey.

type TableIDs

type TableIDs []TableID

TableIDs is a sortable slice of TableID.

func (TableIDs) Len

func (s TableIDs) Len() int

func (TableIDs) Less

func (s TableIDs) Less(i, j int) bool

func (TableIDs) Swap

func (s TableIDs) Swap(i, j int)

type TableKey

type TableKey string

TableKey is a globally unique identifier for a table; it is effectively the compound key: (org, database, table). This is (hopefully) the value that will be used when interfacing with services which are unaware of table qualifiers. For example, the FeatureBase server has no notion of organization or database; its top level type is index/indexName/table. So in this case, until and unless we introduce table qualifiers into FeatureBase, we will use TableKey as the value for index.Name.

func (TableKey) Key added in v3.27.0

func (tk TableKey) Key() TableKey

func (TableKey) QualifiedTableID

func (tk TableKey) QualifiedTableID() QualifiedTableID

QualifiedTableID returns the QualifiedTableID based on the key. If TableKey can't be parsed into a valid (i.e. complete) QualifiedTableID, then blank values are used where necessary.

type TableKeyer added in v3.27.0

type TableKeyer interface {
	Key() TableKey
}

TableKeyer is an interface implemented by any type which can produce, and be represented by, a TableKey. In the case of a QualifiedTable, its TableKey might be something like `tbl__org__db__tableid`, while a general pilosa implemenation might represent a table as a basic table name `foo`.

type TableKeys

type TableKeys []TableKey

TableKeys is a sortable slice of TableKey.

func (TableKeys) Len

func (s TableKeys) Len() int

func (TableKeys) Less

func (s TableKeys) Less(i, j int) bool

func (TableKeys) Swap

func (s TableKeys) Swap(i, j int)

type TableName

type TableName string

TableName is a human-friendly string. While it is not used as a primary key, uniqueness is generally enforced within the scope of a QualifiedDatabaseID.

type TableNames

type TableNames []TableName

TableNames is a sortable slice of TableName.

func (TableNames) Len

func (s TableNames) Len() int

func (TableNames) Less

func (s TableNames) Less(i, j int) bool

func (TableNames) Swap

func (s TableNames) Swap(i, j int)

type Tables

type Tables []*Table

Tables is a sortable slice of Table.

func (Tables) Len

func (o Tables) Len() int

func (Tables) Less

func (o Tables) Less(i, j int) bool

func (Tables) Swap

func (o Tables) Swap(i, j int)

type TimeQuantum

type TimeQuantum string

TimeQuantum represents a time granularity for time-based bitmaps.

func (TimeQuantum) Granularity

func (q TimeQuantum) Granularity() rune

func (TimeQuantum) HasDay

func (q TimeQuantum) HasDay() bool

HasDay returns true if the quantum contains a 'D' unit.

func (TimeQuantum) HasHour

func (q TimeQuantum) HasHour() bool

HasHour returns true if the quantum contains a 'H' unit.

func (TimeQuantum) HasMonth

func (q TimeQuantum) HasMonth() bool

HasMonth returns true if the quantum contains a 'M' unit.

func (TimeQuantum) HasYear

func (q TimeQuantum) HasYear() bool

HasYear returns true if the quantum contains a 'Y' unit.

func (TimeQuantum) IsEmpty

func (q TimeQuantum) IsEmpty() bool

IsEmpty returns true if the quantum is empty.

func (*TimeQuantum) Set

func (q *TimeQuantum) Set(value string) error

Set sets the time quantum value.

func (TimeQuantum) String

func (q TimeQuantum) String() string

String returns the TimeQuantum as a string type.

func (TimeQuantum) Type

func (q TimeQuantum) Type() string

Type returns the type of a time quantum value.

func (TimeQuantum) Valid

func (q TimeQuantum) Valid() bool

Valid returns true if q is a valid time quantum value.

type Transaction added in v3.29.0

type Transaction interface {
	Commit() error
	Context() context.Context
	Rollback() error
}

type TranslateNode added in v3.27.0

type TranslateNode struct {
	Address    Address       `json:"address"`
	Table      TableKey      `json:"table"`
	Partitions PartitionNums `json:"partitions"`
}

TranslateNode represents a translate node and the table/partitions for which it is responsible.

type TranslateRole

type TranslateRole struct {
	TableKey   TableKey      `json:"table-key"`
	Partitions PartitionNums `json:"partitions"`
	Fields     []FieldName   `json:"fields"`
}

TranslateRole is a role specific to translate nodes.

func (*TranslateRole) Type

func (cr *TranslateRole) Type() RoleType

Type returns the type for TranslateRole. This is mainly to implement the Role interface.

type WorkerDiff

type WorkerDiff struct {
	Address     Address
	AddedJobs   []Job
	RemovedJobs []Job
}

WorkerDiff represents the changes made to a Worker following the latest event.

func (*WorkerDiff) Add

func (w *WorkerDiff) Add(w2 WorkerDiff)

Add adds w2 to w. It panics of w and w2 don't have teh same worker ID. Any job that is added and then removed or removed and then added cancels out and won't be present after add is called.

type WorkerDiffs

type WorkerDiffs []WorkerDiff

WorkerDiffs is a sortable slice of WorkerDiff.

func (WorkerDiffs) Len

func (w WorkerDiffs) Len() int

func (WorkerDiffs) Less

func (w WorkerDiffs) Less(i, j int) bool

func (WorkerDiffs) Swap

func (w WorkerDiffs) Swap(i, j int)

type WorkerInfo

type WorkerInfo struct {
	Address Address
	Jobs    []Job
}

WorkerInfo represents a Worker and the Jobs to which it has been assigned.

type WorkerInfos

type WorkerInfos []WorkerInfo

WorkerInfos is a sortable slice of WorkerInfo.

func (WorkerInfos) Len

func (w WorkerInfos) Len() int

func (WorkerInfos) Less

func (w WorkerInfos) Less(i, j int) bool

func (WorkerInfos) Swap

func (w WorkerInfos) Swap(i, j int)

Directories

Path Synopsis
Package boltdb contains the boltdb implementations of the DAX interfaces.
Package boltdb contains the boltdb implementations of the DAX interfaces.
Package computer contains the compute-specific portions of the DAX architecture.
Package computer contains the compute-specific portions of the DAX architecture.
Package controller provides the core Controller struct.
Package controller provides the core Controller struct.
balancer
Package balancer is an implementation of the controller.Balancer interface.
Package balancer is an implementation of the controller.Balancer interface.
balancer/boltdb
Package boltdb contains the boltdb implementation of the Balancer interface.
Package boltdb contains the boltdb implementation of the Balancer interface.
client
Package client is an HTTP client for Controller.
Package client is an HTTP client for Controller.
http
Package http provides the http implementation of the Director interface.
Package http provides the http implementation of the Director interface.
partitioner
Package partitioner provides the Partitioner type, which provides helper methods for determining partitions based on string keys.
Package partitioner provides the Partitioner type, which provides helper methods for determining partitions based on string keys.
poller
Package poller provides the core Poller struct.
Package poller provides the core Poller struct.
schemar
Package schemar provides the core Schemar interface.
Package schemar provides the core Schemar interface.
schemar/boltdb
Package boltdb contains the boltdb implementation of the Schemar interfaces.
Package boltdb contains the boltdb implementation of the Schemar interfaces.
client
Package client is an HTTP client for the Queryer.
Package client is an HTTP client for the Queryer.
Package snapshotter provides the core snapshotter structs.
Package snapshotter provides the core snapshotter structs.
Package test include external test apps, helper functions, and test data.
Package test include external test apps, helper functions, and test data.
Package writelogger provides the writelogger structs.
Package writelogger provides the writelogger structs.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL