manager

Documentation

Overview

Package manager introduces a KozmoPropeller Manager implementation that enables horizontal scaling of KozmoPropeller by sharding KozmoWorkflows.

The KozmoPropeller Manager manages a collection of KozmoPropeller instances to effectively distribute load. Each managed KozmoPropller instance is created as a k8s pod using a configurable k8s PodTemplate resource. The KozmoPropeller Manager use a control loop to periodically check the status of managed KozmoPropeller instances and creates, updates, or deletes pods as required. It is important to note that if the KozmoPropeller Manager fails, managed instances are left running. This is in effort to ensure progress continues in evaluating KozmoWorkflow CRDs.

KozmoPropeller Manager is configured at the root of the KozmoPropeller configuration. Below is an example of the variety of configuration options along with succinct associated descriptions for each field:

manager:
  pod-application: "kozmopropeller"             # application name for managed pods
  pod-template-container-name: "kozmopropeller" # the container name within the K8s PodTemplate name used to set KozmoWorkflow CRD labels selectors
  pod-template-name: "kozmopropeller-template"  # k8s PodTemplate name to use for starting KozmoPropeller pods
  pod-template-namespace: "kozmo"               # namespace where the k8s PodTemplate is located
  scan-interval: 10s                            # frequency to scan KozmoPropeller pods and start / restart if necessary
  shard:                                        # configure sharding strategy
    # shard configuration redacted

KozmoPropeller Manager handles dynamic updates to both the k8s PodTemplate and shard configuration. The k8s PodTemplate resource has an associated resource version which uniquely identifies changes. Additionally, shard configuration modifications may be tracked using a simple hash. Kozmo stores these values as annotations on managed KozmoPropeller instances. Therefore, if either of there values change the KozmoPropeller Manager instance will detect it and perform the necessary deployment updates.

Shard Strategies

Kozmo defines a variety of Shard Strategies for configuring how KozmoWorkflows are sharded. These options may include the shard type (ex. hash, project, or domain) along with the number of shards or the distribution of project / domain IDs over shards.

Internally, KozmoWorkflow CRDs are initialized with k8s labels for project, domain, and a shard-key. The project and domain label values are associated with the environment of the registered workflow. The shard-key value is a range-bounded hash over various components of the KozmoWorkflow metadata, currently the keyspace range is defined as [0,32). A sharded Kozmo deployment ensures deterministic KozmoWorkflow evaluations by setting disjoint k8s label selectors, based on the aforementioned labels, on each managed KozmoPropeller instance. This ensures that only a single KozmoPropeller instance is responsible for processing each KozmoWorkflow.

The Hash Shard Strategy, denoted by "type: hash" in the configuration below, uses consistent hashing to evenly distribute KozmoWorkflows over managed KozmoPropeller instances. This is achieved by partitioning the keyspace (i.e. [0,32)) into a collection of disjoint ranges and using label selectors to assign those ranges to managed KozmoPropeller instances. For example, with "shard-count: 4" the first instance is responsible for KozmoWorkflows with "shard-keys" in the range [0,8), the second [8,16), the third [16,24), and the fourth [24,32). It may be useful to note that the default shard type is "hash", so it will be implicitly defined if otherwise left out of the configuration. An example configuration for the Hash Shard Strategy is provided below:

# a configuration example using the "hash" shard type
manager:
  # pod and scanning configuration redacted
  shard:
    type: hash     # use the "hash" shard strategy
    shard-count: 4 # the total number of shards

The Project and Domain Shard Strategies, denoted by "type: project" and "type: domain" respectively, use the KozmoWorkflow project and domain metadata to distributed KozmoWorkflows over managed KozmoPropeller instances. These Shard Strategies are configured using a "per-shard-mapping" option, which is a list of ID lists. Each element in the "per-shard-mapping" list defines a new shard and the ID list assigns responsibility for the specified IDs to that shard. The assignment is performed using k8s label selectors, where each managed KozmoPropeller instance includes KozmoWorkflows with the specified project or domain labels.

A shard configured as a single wildcard ID (i.e. "*") is responsible for all IDs that are not covered by other shards. Only a single shard may be configured with a wildcard ID and on that shard their must be only one ID, namely the wildcard. In this case, the managed KozmoPropeller instance uses k8s label selectors to exclude KozmoWorkflows with project or domain IDs from other shards.

# a configuration example using the "project" shard type
manager:
  # pod and scanning configuration redacted
  shard:
    type: project       # use the "project" shard strategy
    per-shard-mapping:  # a list of per shard mappings - one shard is created for each element
      - ids:            # the list of ids to be managed by the first shard
        - kozmosnacks
      - ids:            # the list of ids to be managed by the second shard
        - kozmoexamples
        - kozmolabs
      - ids:            # the list of ids to be managed by the third shard
        - "*"           # use the wildcard to manage all ids not managed by other shards

# a configuration example using the "domain" shard type
manager:
  # pod and scanning configuration redacted
  shard:
    type: domain        # use the "domain" shard strategy
    per-shard-mapping:  # a list of per shard mappings - one shard is created for each element
      - ids:            # the list of ids to be managed by the first shard
        - production
      - ids:            # the list of ids to be managed by the second shard
        - "*"           # use the wildcard to manage all ids not managed by other shards

Index

• type Manager
  • func New(ctx context.Context, propellerCfg *propellerConfig.Config, ...) (*Manager, error)
  • func (m *Manager) Run(ctx context.Context) error

Types

type Manager

type Manager struct {
	// contains filtered or unexported fields
}

Manager periodically scans k8s to ensure liveness of multiple KozmoPropeller controller instances and rectifies state based on the configured sharding strategy.

func New

func New(ctx context.Context, propellerCfg *propellerConfig.Config, cfg *managerConfig.Config, podNamespace string, ownerReferences []metav1.OwnerReference, kubeClient kubernetes.Interface, scope promutils.Scope) (*Manager, error)

New creates a new KozmoPropeller Manager instance.

func (*Manager) Run

func (m *Manager) Run(ctx context.Context) error

Run starts the manager instance as either a k8s leader, if configured, or as a standalone process.