shardstrategy

Documentation

Overview

Package shardstrategy defines a variety of sharding stratgies to distribute FlyteWorkflows over managed FlytePropeller instances.

Index

• Constants
• func ComputeKeyRange(keyspaceSize, podCount, podIndex int) (int, int)
• type EnvironmentShardStrategy
  • func (e *EnvironmentShardStrategy) GetPodCount() int
  • func (e *EnvironmentShardStrategy) HashCode() (uint32, error)
  • func (e *EnvironmentShardStrategy) UpdatePodSpec(pod *v1.PodSpec, containerName string, podIndex int) error
• type HashShardStrategy
  • func (h *HashShardStrategy) GetPodCount() int
  • func (h *HashShardStrategy) HashCode() (uint32, error)
  • func (h *HashShardStrategy) UpdatePodSpec(pod *v1.PodSpec, containerName string, podIndex int) error
• type ShardStrategy
  • func NewShardStrategy(ctx context.Context, shardConfig config.ShardConfig) (ShardStrategy, error)

Constants

const (
	Project environmentType = iota
	Domain
)

Functions

func ComputeKeyRange

func ComputeKeyRange(keyspaceSize, podCount, podIndex int) (int, int)

ComputeKeyRange computes a [startKey, endKey) pair denoting the key responsibilities for the provided pod index given the keyspaceSize and podCount parameters.

Types

type EnvironmentShardStrategy

type EnvironmentShardStrategy struct {
	EnvType     environmentType
	PerShardIDs [][]string
}

EnvironmentShardStrategy assigns either project or domain identifiers to individual FlytePropeller instances to determine FlyteWorkflow processing responsibility.

func (*EnvironmentShardStrategy) GetPodCount

func (e *EnvironmentShardStrategy) GetPodCount() int

func (*EnvironmentShardStrategy) HashCode

func (e *EnvironmentShardStrategy) HashCode() (uint32, error)

func (*EnvironmentShardStrategy) UpdatePodSpec

func (e *EnvironmentShardStrategy) UpdatePodSpec(pod *v1.PodSpec, containerName string, podIndex int) error

type HashShardStrategy

type HashShardStrategy struct {
	ShardCount int
}

HashShardStrategy evenly assigns disjoint keyspace responsibilities over a collection of pods. All FlyteWorkflows are assigned a shard-key using a hash of their executionID and are then processed by the FlytePropeller instance responsible for that keyspace range.

func (*HashShardStrategy) GetPodCount

func (h *HashShardStrategy) GetPodCount() int

func (*HashShardStrategy) HashCode

func (h *HashShardStrategy) HashCode() (uint32, error)

func (*HashShardStrategy) UpdatePodSpec

func (h *HashShardStrategy) UpdatePodSpec(pod *v1.PodSpec, containerName string, podIndex int) error

type ShardStrategy

type ShardStrategy interface {
	// GetPodCount returns the total number of pods for the sharding strategy.
	GetPodCount() int
	// HashCode generates a unique hash code to identify shard strategy updates.
	HashCode() (uint32, error)
	// UpdatePodSpec amends the PodSpec for the specified index to include label selectors.
	UpdatePodSpec(pod *v1.PodSpec, containerName string, podIndex int) error
}

ShardStrategy defines necessary functionality for a sharding strategy.

func NewShardStrategy

func NewShardStrategy(ctx context.Context, shardConfig config.ShardConfig) (ShardStrategy, error)

NewShardStrategy creates and validates a new ShardStrategy defined by the configuration.