sharding

Documentation

Index

• Constants
• Variables
• func GetOrUpdateShardFromConfigMap(kubeClient *kubernetes.Clientset, settingsMgr *settings.SettingsManager, ...) (int, error)
• func InferShard() (int, error)
• type ClusterFilterFunction
  • func GetClusterFilter(db db.ArgoDB, distributionFunction DistributionFunction, replicas, shard int) ClusterFilterFunction
• type ClusterSharding
  • func (sharding *ClusterSharding) Add(c *v1alpha1.Cluster)
  • func (sharding *ClusterSharding) Delete(clusterServer string)
  • func (d *ClusterSharding) GetClusterAccessor() clusterAccessor
  • func (sharding *ClusterSharding) GetDistribution() map[string]int
  • func (sharding *ClusterSharding) Init(clusters *v1alpha1.ClusterList)
  • func (s *ClusterSharding) IsManagedCluster(c *v1alpha1.Cluster) bool
  • func (sharding *ClusterSharding) Update(c *v1alpha1.Cluster)
• type ClusterShardingCache
  • func NewClusterSharding(db db.ArgoDB, shard, replicas int, shardingAlgorithm string) ClusterShardingCache
• type DistributionFunction
  • func GetDistributionFunction(clusters clusterAccessor, shardingAlgorithm string, replicasCount int) DistributionFunction
  • func LegacyDistributionFunction(replicas int) DistributionFunction
  • func NoShardingDistributionFunction() DistributionFunction
  • func RoundRobinDistributionFunction(clusters clusterAccessor, replicas int) DistributionFunction

Constants

const ShardControllerMappingKey = "shardControllerMapping"

Variables

var (
	HeartbeatDuration = env.ParseNumFromEnv(common.EnvControllerHeartbeatTime, 10, 10, 60)
	HeartbeatTimeout  = 3 * HeartbeatDuration
)

Functions

func GetOrUpdateShardFromConfigMap

func GetOrUpdateShardFromConfigMap(kubeClient *kubernetes.Clientset, settingsMgr *settings.SettingsManager, replicas, shard int) (int, error)

GetOrUpdateShardFromConfigMap finds the shard number from the shard mapping configmap. If the shard mapping configmap does not exist, the function creates the shard mapping configmap. The function takes the shard number from the environment variable (default value -1, if not set) and passes it to this function. If the shard value passed to this function is -1, that is, the shard was not set as an environment variable, we default the shard number to 0 for computing the default config map.

func InferShard

func InferShard() (int, error)

InferShard extracts the shard index based on its hostname.

Types

type ClusterFilterFunction

type ClusterFilterFunction func(c *v1alpha1.Cluster) bool

func GetClusterFilter

func GetClusterFilter(db db.ArgoDB, distributionFunction DistributionFunction, replicas, shard int) ClusterFilterFunction

GetClusterFilter returns a ClusterFilterFunction which is a function taking a cluster as a parameter and returns wheter or not the cluster should be processed by a given shard. It calls the distributionFunction to determine which shard will process the cluster, and if the given shard is equal to the calculated shard the function will return true.

type ClusterSharding

type ClusterSharding struct {
	Shard    int
	Replicas int
	Shards   map[string]int
	Clusters map[string]*v1alpha1.Cluster
	// contains filtered or unexported fields
}

func (*ClusterSharding) Add

func (sharding *ClusterSharding) Add(c *v1alpha1.Cluster)

func (*ClusterSharding) Delete

func (sharding *ClusterSharding) Delete(clusterServer string)

func (*ClusterSharding) GetClusterAccessor

func (d *ClusterSharding) GetClusterAccessor() clusterAccessor

func (*ClusterSharding) GetDistribution

func (sharding *ClusterSharding) GetDistribution() map[string]int

func (*ClusterSharding) Init

func (sharding *ClusterSharding) Init(clusters *v1alpha1.ClusterList)

func (*ClusterSharding) IsManagedCluster

func (s *ClusterSharding) IsManagedCluster(c *v1alpha1.Cluster) bool

IsManagedCluster returns wheter or not the cluster should be processed by a given shard.

func (*ClusterSharding) Update

func (sharding *ClusterSharding) Update(c *v1alpha1.Cluster)

type ClusterShardingCache

type ClusterShardingCache interface {
	Init(clusters *v1alpha1.ClusterList)
	Add(c *v1alpha1.Cluster)
	Delete(clusterServer string)
	Update(c *v1alpha1.Cluster)
	IsManagedCluster(c *v1alpha1.Cluster) bool
	GetDistribution() map[string]int
}

func NewClusterSharding

func NewClusterSharding(db db.ArgoDB, shard, replicas int, shardingAlgorithm string) ClusterShardingCache

type DistributionFunction

type DistributionFunction func(c *v1alpha1.Cluster) int

func GetDistributionFunction

func GetDistributionFunction(clusters clusterAccessor, shardingAlgorithm string, replicasCount int) DistributionFunction

GetDistributionFunction returns which DistributionFunction should be used based on the passed algorithm and the current datas.

func LegacyDistributionFunction

func LegacyDistributionFunction(replicas int) DistributionFunction

LegacyDistributionFunction returns a DistributionFunction using a stable distribution algorithm: for a given cluster the function will return the shard number based on the cluster id. This function is lightweight and can be distributed easily, however, it does not ensure an homogenous distribution as some shards may get assigned more clusters than others. It is the legacy function distribution that is kept for compatibility reasons

func NoShardingDistributionFunction

func NoShardingDistributionFunction() DistributionFunction

NoShardingDistributionFunction returns a DistributionFunction that will process all cluster by shard 0 the function is created for API compatibility purposes and is not supposed to be activated.

func RoundRobinDistributionFunction

func RoundRobinDistributionFunction(clusters clusterAccessor, replicas int) DistributionFunction