sharding

Documentation

Index

• type Shard
  • func NewShard(id, weight int) Shard
• type ShardFunc
  • func NewIntSharder(max int64) ShardFunc
  • func NewStringSharder(max int) ShardFunc
• type ShardMap
  • func NewShardMap() ShardMap

Types

type Shard

type Shard interface {
	// ID is the shard ID. The shard ID is calculated through a shard function
	// which will map an identifier to a particular shard.
	ID() int

	// Weight represents the relative work for the shard. Some shards might
	// require more work than others, depending on the work distribution.
	// Initially this can be set to 1 for all shards but if you have hotspots
	// with higher resource requirements (like more CPU or memory) you can
	// increase the weight of a shard to balance the load across the cluster.
	Weight() int

	// NodeID returns the node responsible for the shard.
	NodeID() string

	// SetNodeID sets the node ID for the shard
	SetNodeID(nodeID string)
}

Shard represents a partition that a single node is responsible for.

func NewShard

func NewShard(id, weight int) Shard

NewShard creates a new shard with a weight

type ShardFunc

type ShardFunc func(interface{}) int

ShardFunc is a sharding function

func NewIntSharder

func NewIntSharder(max int64) ShardFunc

NewIntSharder shards on integers using a simple mod operation. The returned integer is the shard.

func NewStringSharder

func NewStringSharder(max int) ShardFunc

NewStringSharder hashes a string and calculates a shard based on the hash value. The returned integer is the shard id.

type ShardMap

type ShardMap interface {
	// Init reinitializes the (shard) manager. This can be called one and only
	// once. Performance critical since this is part of the node
	// onboarding process. The weights parameter may be set to nil. In that
	// case the shards gets a weight of 1. If the weights parameter is specfied
	// the lenght of the weights parameter must match the maxShards parameter.
	// Shard IDs are assigned from 0...maxShards-1
	Init(maxShards int, weights []int) error

	// UpdateNodes syncs the nodes internally in the cluster and reshards if
	// necessary.
	UpdateNodes(nodeID ...string)

	// GetNode returns the node (ID) responsible for the shards. Performance
	// critical since this will be used in every single call to determine
	// the home location for mutations.
	// TBD: Panic if the shard ID is > maxShards?
	MapToNode(shardID int) Shard

	// Shards returns a copy of all of the shards. Not performance critical. This
	// is typically used for diagnostics.
	Shards() []Shard

	// TotalWeight is the total weight of all shards. Not performance critical
	// directly but it will be used when calculating the distribution of shards
	// so the manager should cache this value and update when a shard changes
	// its weight.
	TotalWeight() int

	// ShardCount returns the number of shards
	ShardCount() int

	// NodeList returns a list of nodes in the shard map
	NodeList() []string

	// ShardCountForNode returns the number of shards allocated to a particular node.
	ShardCountForNode(nodeid string) int

	encoding.BinaryMarshaler
	encoding.BinaryUnmarshaler
}

ShardMap is a type that manages shards. The number of shards are immutable, ie no new shards will be added for the lifetime. (shards can be added or removed between invocations of the leader)

func NewShardMap

func NewShardMap() ShardMap

NewShardMap creates a new shard mapper instance.