disco

package
v3.35.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 30, 2023 License: Apache-2.0, Apache-2.0 Imports: 11 Imported by: 0

Documentation

Overview

Copyright 2022 Molecula Corp. (DBA FeatureBase). SPDX-License-Identifier: Apache-2.0

Copyright 2022 Molecula Corp. (DBA FeatureBase). SPDX-License-Identifier: Apache-2.0

Copyright 2022 Molecula Corp. (DBA FeatureBase). SPDX-License-Identifier: Apache-2.0

Copyright 2022 Molecula Corp. (DBA FeatureBase). SPDX-License-Identifier: Apache-2.0

Copyright 2022 Molecula Corp. (DBA FeatureBase). SPDX-License-Identifier: Apache-2.0

Index

Constants

View Source 
const (
	InitialClusterStateNew      InitialClusterState = "new"
	InitialClusterStateExisting InitialClusterState = "existing"

	ClusterStateUnknown  ClusterState = "UNKNOWN"  // default cluster state. It is returned when we are not able to get the real actual state.
	ClusterStateDegraded ClusterState = "DEGRADED" // cluster is running but we've lost some # of hosts >0 but < replicaN. Only read queries are allowed.
	ClusterStateNormal   ClusterState = "NORMAL"   // cluster is up and running.
	ClusterStateDown     ClusterState = "DOWN"     // cluster is unable to serve queries.
)
View Source 
const (
	// DefaultPartitionN is the default number of partitions in a cluster.
	DefaultPartitionN = 256

	// ShardWidth is the number of column IDs in a shard. It must be a power of 2 greater than or equal to 16.
	// shardWidthExponent = 20 // set in shardwidthNN.go files
	ShardWidth = 1 << shardwidth.Exponent
)

Variables

View Source 
var (
	ErrTooManyResults    error = fmt.Errorf("too many results")
	ErrNoResults         error = fmt.Errorf("no results")
	ErrKeyDeleted        error = fmt.Errorf("key deleted")
	ErrIndexExists       error = fmt.Errorf("index already exists")
	ErrIndexDoesNotExist error = fmt.Errorf("index does not exist")
	ErrFieldExists       error = fmt.Errorf("field already exists")
	ErrFieldDoesNotExist error = fmt.Errorf("field does not exist")
	ErrViewExists        error = fmt.Errorf("view already exists")
	ErrViewDoesNotExist  error = fmt.Errorf("view does not exist")
	ErrKeyDoesNotExist   error = fmt.Errorf("key does not exist")
)

Functions

func NewEmptyLocalNoder 

func NewEmptyLocalNoder() *localNoder

NewEmptyLocalNoder is an empty Noder used for testing.

func NewIDNoder 

func NewIDNoder(ids []string) *localNoder

NewIDNoder is a helper function for wrapping an existing slice of Node IDs with something which implements Noder.

func NewInMemDisCo 

func NewInMemDisCo(id string) *inMemDisCo

func NewInMemSchemator 

func NewInMemSchemator() *inMemSchemator

NewInMemSchemator instantiates an InMemSchemator this allows new holders to have thier own, and not rely on a shared instance

func NewInMemSharder added in v3.33.0 

func NewInMemSharder() *inMemSharder

func NewLocalNoder 

func NewLocalNoder(nodes []*Node) *localNoder

NewLocalNoder is a helper function for wrapping an existing slice of Nodes with something which implements Noder.

func NodePositionByID 

func NodePositionByID(nodes []*Node, nodeID string) int

NodePositionByID returns the position of the node in slice nodes. TODO: this is exported because it's used in noder.go. Because that's the same package, it doesn't need to be exported, but ideally we could put this snapshot code into its own package. I tried to do that (by putting it into a package called `topology`), but that created an import loop. So what we really need to do is do a better job of creating sub-packages under pilosa (for things like `Noder` and `Nodes`).

func PrimaryNodeID 

func PrimaryNodeID(nodeIDs []string, hasher Hasher) string

PrimaryNodeID returns the ID of the primary node, given a list of node IDs and a hasher. The order of the node IDs provided does not matter because this function will re-order them in a deterministic way.

func ShardToShardPartition 

func ShardToShardPartition(index string, shard uint64, partitionN int) int

ShardToShardParition ...

Types

type ByID 

type ByID []*Node

ByID implements sort.Interface for []Node based on the ID field.

func (ByID) Len 

func (h ByID) Len() int

func (ByID) Less 

func (h ByID) Less(i, j int) bool

func (ByID) Swap 

func (h ByID) Swap(i, j int)

type ClusterSnapshot 

type ClusterSnapshot struct {
	Nodes []*Node

	// Hashing algorithm used to assign partitions to nodes.
	Hasher Hasher

	// The number of partitions in the cluster.
	PartitionN int

	// The number of replicas a partition has.
	ReplicaN int

	PartitionAssignment string
}

ClusterSnapshot is a static representation of a cluster and its nodes. It is used to calculate things like partition location and data distribution.

func NewClusterSnapshot 

func NewClusterSnapshot(noder Noder, hasher Hasher, partitionAssignment string, replicas int) *ClusterSnapshot

NewClusterSnapshot returns a new instance of ClusterSnapshot.

func (*ClusterSnapshot) ContainsShards 

func (c *ClusterSnapshot) ContainsShards(index string, availableShards *roaring.Bitmap, node *Node) []uint64

ContainsShards is like OwnsShards, but it includes replicas.

func (*ClusterSnapshot) IDToShardPartition 

func (c *ClusterSnapshot) IDToShardPartition(index string, id uint64) int

IDToShardPartition returns the shard-partition that an id belongs to.

func (*ClusterSnapshot) IsPrimary 

func (c *ClusterSnapshot) IsPrimary(nodeID string, partition int) bool

IsPrimary returns true if the given node is the primary for the given partition.

func (*ClusterSnapshot) IsPrimaryFieldTranslationNode 

func (c *ClusterSnapshot) IsPrimaryFieldTranslationNode(nodeID string) bool

IsPrimaryFieldTranslationNode returns true if nodeID represents the primary node responsible for field translation.

func (*ClusterSnapshot) KeyNodes 

func (c *ClusterSnapshot) KeyNodes(index, key string) []*Node

KeyNodes returns a list of nodes that own a key.

func (*ClusterSnapshot) KeyToKeyPartition 

func (c *ClusterSnapshot) KeyToKeyPartition(index, key string) int

KeyToKeyPartition returns the key-partition that the given key belongs to. NOTE: The key-partition is DIFFERENT from the shard-partition.

func (*ClusterSnapshot) NonPrimaryReplicas 

func (c *ClusterSnapshot) NonPrimaryReplicas(partition int) (nonPrimaryReplicas []string)

NonPrimaryReplicas returns the list of node IDs which are replicas for the given partition.

func (*ClusterSnapshot) OwnsShard 

func (c *ClusterSnapshot) OwnsShard(nodeID string, index string, shard uint64) (ret bool)

OwnsShard returns true if a host owns a fragment.

func (*ClusterSnapshot) PartitionNodes 

func (c *ClusterSnapshot) PartitionNodes(partitionID int) []*Node

PartitionNodes returns a list of nodes that own the given partition.

func (*ClusterSnapshot) PrimaryFieldTranslationNode 

func (c *ClusterSnapshot) PrimaryFieldTranslationNode() *Node

PrimaryFieldTranslationNode is the primary node responsible for translating field keys. The primary could be any node in the cluster, but we arbitrarily define it to be the node responsible for partition 0.

func (*ClusterSnapshot) PrimaryForColKeyTranslation 

func (c *ClusterSnapshot) PrimaryForColKeyTranslation(index, key string) (primary int)

TODO: update this comment The boltdb key translation stores are partitioned, designated by partitionIDs. These are shared between replicas, and one node is the primary for replication. So with 4 nodes and 3-way replication, each node has 3/4 of the translation stores on it.

func (*ClusterSnapshot) PrimaryForShardReplication 

func (c *ClusterSnapshot) PrimaryForShardReplication(index string, shard uint64) int

TODO: update this comment

func (*ClusterSnapshot) PrimaryNodeIndex 

func (c *ClusterSnapshot) PrimaryNodeIndex(partition int) int

PrimaryNodeIndex returns the index (position in the cluster) of the primary node for the given partition.

func (*ClusterSnapshot) PrimaryPartitionNode 

func (c *ClusterSnapshot) PrimaryPartitionNode(partitionID int) *Node

PrimaryPartitionNode returns the primary node of the given partition.

func (*ClusterSnapshot) PrimaryReplicaNode 

func (c *ClusterSnapshot) PrimaryReplicaNode(nodeID string) *Node

PrimaryReplicaNode returns the node listed before the current node in Nodes(). This is different than "previous node" as the first node always returns nil.

func (*ClusterSnapshot) ReplicasForPrimary 

func (c *ClusterSnapshot) ReplicasForPrimary(primary int) (replicaNodeIDs, nonReplicas map[string]bool)

ReplicasForPrimary returns the map replicaNodeIDs[nodeID] which will have a true value for the primary nodeID, and false for others.

func (*ClusterSnapshot) ShardNodes 

func (c *ClusterSnapshot) ShardNodes(index string, shard uint64) []*Node

ShardNodes returns a list of nodes that own a shard.

func (*ClusterSnapshot) ShardToShardPartition 

func (c *ClusterSnapshot) ShardToShardPartition(index string, shard uint64) int

ShardToShardPartition returns the shard-partition that the given shard belongs to. NOTE: This is DIFFERENT from the key-partition.

type ClusterState 

type ClusterState string

ClusterState represents the state returned in the /status endpoint.

type DisCo 

type DisCo interface {
	io.Closer

	Start(ctx context.Context) (InitialClusterState, error)
	IsLeader() bool
	ID() string
	Leader() *Peer
	Peers() []*Peer
	DeleteNode(ctx context.Context, id string) error
}
var NopDisCo DisCo = &nopDisCo{}

NopDisCo represents a DisCo that doesn't do anything.

type Field 

type Field struct {
	Data  []byte
	Views map[string]struct{}
}

Field is a struct which contains the data encoded for the field as well as for each of its views.

type Hasher 

type Hasher interface {
	// Hashes the key into a number between [0,N).
	Hash(key uint64, n int) int
	Name() string
}

Hasher represents an interface to hash integers into buckets.

type Index 

type Index struct {
	Data   []byte
	Fields map[string]*Field
}

Index is a struct which contains the data encoded for the index as well as for each of its fields.

type InitialClusterState 

type InitialClusterState string

type Jmphasher 

type Jmphasher struct{}

Jmphasher represents an implementation of jmphash. Implements Hasher.

func (*Jmphasher) Hash 

func (h *Jmphasher) Hash(key uint64, n int) int

Hash returns the integer hash for the given key.

func (*Jmphasher) Name 

func (h *Jmphasher) Name() string

Name returns the name of this hash.

type Node 

type Node struct {
	ID        string    `json:"id"`
	URI       net.URI   `json:"uri"`
	GRPCURI   net.URI   `json:"grpc-uri"`
	IsPrimary bool      `json:"isPrimary"`
	State     NodeState `json:"state"`
}

Node represents a node in the cluster.

func PrimaryNode 

func PrimaryNode(nodes []*Node, hasher Hasher) *Node

PrimaryNode yields the node that would be selected as the primary from a list, for a given ID. It assumes the list is already in the expected order, as from Noder.Nodes().

func (*Node) Clone 

func (n *Node) Clone() *Node

func (*Node) String 

func (n *Node) String() string

type NodeState 

type NodeState string
const (
	NodeStateUnknown  NodeState = "UNKNOWN"
	NodeStateStarting NodeState = "STARTING"
	NodeStateStarted  NodeState = "STARTED"
)

type Noder 

type Noder interface {
	Nodes() []*Node // Remember: this has to be sorted correctly!!
	PrimaryNodeID(hasher Hasher) string

	// SetMetadata records the local node's metadata.
	SetMetadata(ctx context.Context, node *Node) error

	// SetState changes a node to a given state.
	SetState(ctx context.Context, state NodeState) error

	// ClusterState considers the state of all nodes and gives
	// a general cluster state. The output calculation is as follows:
	// - If any of the nodes are still starting: "STARTING"
	// - If all nodes are up and running: "NORMAL"
	// - If number of DOWN nodes is lower than number of replicas: "DEGRADED"
	// - If number of unresponsive nodes is greater than (or equal to) the number of replicas: "DOWN"
	ClusterState(context.Context) (ClusterState, error)
}

Noder is an interface which abstracts the Node slice so that the list of nodes in a cluster can be maintained outside of the cluster struct.

type Nodes 

type Nodes []*Node

Nodes represents a list of nodes.

func (Nodes) Clone 

func (a Nodes) Clone() []*Node

Clone returns a shallow copy of nodes.

func (Nodes) Contains 

func (a Nodes) Contains(n *Node) bool

Contains returns true if a node exists in the list.

func (Nodes) ContainsID 

func (a Nodes) ContainsID(id string) bool

ContainsID returns true if host matches one of the node's id.

func (Nodes) Filter 

func (a Nodes) Filter(n *Node) []*Node

Filter returns a new list of nodes with node removed.

func (Nodes) FilterID 

func (a Nodes) FilterID(id string) []*Node

FilterID returns a new list of nodes with ID removed.

func (Nodes) FilterURI 

func (a Nodes) FilterURI(uri net.URI) []*Node

FilterURI returns a new list of nodes with URI removed.

func (Nodes) IDs 

func (a Nodes) IDs() []string

IDs returns a list of all node IDs.

func (Nodes) NodeByID 

func (a Nodes) NodeByID(id string) *Node

NodeByID returns the node for an ID. If the ID is not found, it returns nil.

func (Nodes) URIs 

func (a Nodes) URIs() []net.URI

URIs returns a list of all uris.

type Peer 

type Peer struct {
	URL string
	ID  string
}

func (*Peer) String 

func (p *Peer) String() string

type Schema 

type Schema map[string]*Index

Schema is a map of all indexes, each of those being a map of fields, then views.

type Schemator 

type Schemator interface {

	// Schema return the actual pilosa schema. If the schema is not present, an error is returned.
	Schema(ctx context.Context) (Schema, error)

	// Index gets a specific index data by name.
	Index(ctx context.Context, name string) ([]byte, error)

	CreateIndex(ctx context.Context, name string, val []byte) error
	DeleteIndex(ctx context.Context, name string) error
	Field(ctx context.Context, index, field string) ([]byte, error)
	CreateField(ctx context.Context, index, field string, fieldVal []byte) error
	UpdateField(ctx context.Context, index, field string, fieldVal []byte) error
	DeleteField(ctx context.Context, index, field string) error
	View(ctx context.Context, index, field, view string) (bool, error)
	CreateView(ctx context.Context, index, field, view string) error
	DeleteView(ctx context.Context, index, field, view string) error
}

Schemator is the source of truth for different schema elements. All nodes will store and retrieve information from the same source, having the same information at the same time. 

var NopSchemator Schemator = &nopSchemator{}

NopSchemator represents a Schemator that doesn't do anything.

type Sharder 

type Sharder interface {
	Shards(ctx context.Context, index, field string) ([][]byte, error)
	SetShards(ctx context.Context, index, field string, shards []byte) error
}

Sharder is an interface used to maintain the set of availableShards bitmaps per field. 

var NopSharder Sharder = &nopSharder{}

NopSharder represents a Sharder that doesn't do anything.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.