sharding

Documentation

Index

• Constants
• Variables
• func GetOrUpdateShardFromConfigMap(kubeClient *kubernetes.Clientset, settingsMgr *settings.SettingsManager, ...) (int, error)
• func InferShard() (int, error)
• type ClusterFilterFunction
  • func GetClusterFilter(db db.ArgoDB, distributionFunction DistributionFunction, shard int) ClusterFilterFunction
• type DistributionFunction
  • func GetDistributionFunction(db db.ArgoDB, shardingAlgorithm string) DistributionFunction
  • func LegacyDistributionFunction(db db.ArgoDB) DistributionFunction
  • func RoundRobinDistributionFunction(db db.ArgoDB) DistributionFunction

Constants

const ShardControllerMappingKey = "shardControllerMapping"

Variables

var (
	HeartbeatDuration = env.ParseNumFromEnv(common.EnvControllerHeartbeatTime, 10, 10, 60)
	HeartbeatTimeout  = 3 * HeartbeatDuration
)

Functions

func GetOrUpdateShardFromConfigMap

func GetOrUpdateShardFromConfigMap(kubeClient *kubernetes.Clientset, settingsMgr *settings.SettingsManager, replicas, shard int) (int, error)

GetOrUpdateShardFromConfigMap finds the shard number from the shard mapping configmap. If the shard mapping configmap does not exist, the function creates the shard mapping configmap. The function takes the shard number from the environment variable (default value -1, if not set) and passes it to this function. If the shard value passed to this function is -1, that is, the shard was not set as an environment variable, we default the shard number to 0 for computing the default config map.

func InferShard

func InferShard() (int, error)

InferShard extracts the shard index based on its hostname.

Types

type ClusterFilterFunction

type ClusterFilterFunction func(c *v1alpha1.Cluster) bool

func GetClusterFilter

func GetClusterFilter(db db.ArgoDB, distributionFunction DistributionFunction, shard int) ClusterFilterFunction

GetClusterFilter returns a ClusterFilterFunction which is a function taking a cluster as a parameter and returns wheter or not the cluster should be processed by a given shard. It calls the distributionFunction to determine which shard will process the cluster, and if the given shard is equal to the calculated shard the function will return true.

type DistributionFunction

type DistributionFunction func(c *v1alpha1.Cluster) int

func GetDistributionFunction

func GetDistributionFunction(db db.ArgoDB, shardingAlgorithm string) DistributionFunction

GetDistributionFunction returns which DistributionFunction should be used based on the passed algorithm and the current datas.

func LegacyDistributionFunction

func LegacyDistributionFunction(db db.ArgoDB) DistributionFunction

LegacyDistributionFunction returns a DistributionFunction using a stable distribution algorithm: for a given cluster the function will return the shard number based on the cluster id. This function is lightweight and can be distributed easily, however, it does not ensure an homogenous distribution as some shards may get assigned more clusters than others. It is the legacy function distribution that is kept for compatibility reasons

func RoundRobinDistributionFunction

func RoundRobinDistributionFunction(db db.ArgoDB) DistributionFunction

RoundRobinDistributionFunction returns a DistributionFunction using an homogeneous distribution algorithm: for a given cluster the function will return the shard number based on the modulo of the cluster rank in the cluster's list sorted by uid on the shard number. This function ensures an homogenous distribution: each shards got assigned the same number of clusters +/-1 , but with the drawback of a reshuffling of clusters accross shards in case of some changes in the cluster list