smith

README

Smith

Smith is a Kubernetes workflow engine / resource manager prototype. It's not complete yet, it's still under active development. It may or may not fulfil the requirements of https://github.com/kubernetes/kubernetes/issues/1704.

The idea

What if we build a service that allows us to manage Kubernetes' built-in resources and other Custom Resources (CRs) in a generic way? Similar to how AWS CloudFormation (or Google Deployment Manager) allows us to manage any AWS/GCE and custom resource. Then we could expose all the resources we need to integrate as Third Party Resources and manage them declaratively. This is an open architecture with Kubernetes as its core. Other microservices can create/update/watch CRs to co-ordinate their work/lifecycle.

Implementation

A group of resources is defined using a Bundle (just like a Stack for AWS CloudFormation). The Bundle itself is also a Kubernetes CR. Smith watches for new instances of a Bundle (and events to existing ones), picks them up and processes them.

Processing involves parsing the bundle, building a dependency graph (which is implicitly defined in the bundle), walking the graph, and creating/updating necessary resources. Each created/referenced resource gets an label pointing at the origin Bundle.

Example bundle

CR definitions:

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: bundles.smith.atlassian.com
spec:
  group: smith.atlassian.com
  version: v1
  names:
    kind: Bundle
    plural: bundles
    singular: bundle

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: postgresql-resources.smith.atlassian.com
spec:
  group: smith.atlassian.com
  version: v1
  names:
    kind: PostgresqlResource
    plural: postgresqlresources
    singular: postgresqlresource

Bundle:

apiVersion: smith.atlassian.com/v1
kind: Bundle
description: "Sample resource bundle"
metadata:
  name: bundle1
spec:
  resources:
  - name: db1
    spec:
      apiVersion: smith.atlassian.com/v1
      kind: PostgresqlResource
      metadata:
        name: db1
      spec:
        disk: 100GiB
  - name: app1
    dependsOn:
    - db1
    spec:
      apiVersion: apps/v1beta1
      kind: Deployment
      metadata:
        name: app1
      spec:
        replicas: 1
        bundle:
          metadata:
            labels:
              app: app1
          spec:
            containers:
            - name: app1
              image: quay.io/some/app1

Outputs

Some resource types can have Outputs:

• Values placed into Status field of the object
• New objects likes Secrets and/or ConfigMaps
• Service Catalog objects

Resources can reference outputs of other resources within the same bundle. See what is supported.

Dependencies

Resources may depend on each other explicitly via DependsOn object references. Resources are created in the reverse dependency order.

States

READY is the state of a Resource when it can be considered created. E.g. if it is a DB then it means it was provisioned and set up as requested. State is often part of Status but it depends on kind of resource.

Event-driven and stateless

Smith does not block while waiting for a resource to reach the READY state. Instead, when walking the dependency graph, if a resource is not in the READY state (still being created) it skips processing of that resource. Resources that don't have their dependencies READY are not processed. Resources that can be created concurrently are created concurrently. Full bundle re-processing is triggered by events about the watched resources. Smith is watching all supported resource types and reacts to events to determine which bundle should be re-processed. This scales better than watching individual resources and much better than polling individual resources. Smith controller is built according to recommendations and following the same behaviour, semantics and code "style" as native Kubernetes controllers as closely as possible.

Features

• Supported object kinds: Deployment, Service, ConfigMap, Secret, PodPreset, Ingress;
• Service Catalog support: objects with kind ServiceInstance and ServiceBinding. See an example and recording of the presentation to Service Catalog SIG;
• Dynamic Custom Resources support via special annotations;
• References between objects in the graph to pull parts of objects/fields from dependencies;
• Smith will delete objects which were removed from a Bundle when Bundle reconciliation is performed (e.g. on a Bundle update).

Missing features

See milestone v1.0.

Notes

Presentations

Smith has been presented to:

• SIG Service Catalog - see information, screencast and recording here.
• SIG Apps - see recoding of the meeting.

On App Controller

Mirantis App Controller (discussed here https://github.com/kubernetes/kubernetes/issues/29453) is a very similar workflow engine with a few differences.

• Graph of dependencies is defined explicitly.
• It uses polling and blocks while waiting for the resource to become READY.
• The goal of Smith is to manage instances of CRs. App Controller cannot manage them as of this writing.

On Helm

Helm is a package manager for Kubernetes. Smith operates on a lower level, even though it can be used by a human, that is not the main usecase. Smith is built to be used as a foundation component with human-friendly tooling built on top of it. E.g. Helm could probably use Smith under the covers to manipulate Kubernetes API objects. Another usecase is a PaaS that delegates (some) object manipulations to Smith.

Requirements

• Please run on recent enough Kubernetes version because earlier versions have some bugs that may prevent Smith from working properly. We test on 1.6+;
• Go 1.7+ is required because context package is used and it was added to standard library in this version;
• Working Docker installation - build process uses dockerized Go to isolate from the host system;
• List of project dependencies and their versions can be found in Gopkg.toml and Gopkg.lock files.

Building

• dep is used for package management. Please install it.
• Bazel is used as the build tool. Please install it.
• To install dependencies run

make setup

• To run integration tests with minikube run

make minikube-test

• To run integration tests for Service Catalog support run

make minikube-test-sc

This command assumes Service Catalog and UPS Broker are installed on your minikube. To install them follow the Service Catalog walkthrough.

• To run against minikube run

make minikube-run

• To build the docker image run

make docker

Contributing

Pull requests, issues and comments welcome. For pull requests:

• Add tests for new features and bug fixes
• Follow the existing style
• Separate unrelated changes into multiple pull requests

See the existing issues for things to start contributing.

For bigger changes, make sure you start a discussion first by creating an issue and explaining the intended change.

Atlassian requires contributors to sign a Contributor License Agreement, known as a CLA. This serves as a record stating that the contributor is entitled to contribute the code/documentation/translation to the project and is willing to have it used in distributions and derivative works (or is willing to transfer ownership).

Prior to accepting your contributions we ask that you please follow the appropriate link below to digitally sign the CLA. The Corporate CLA is for those who are contributing as a member of an organization and the individual CLA is for those contributing as an individual.

• CLA for corporate contributors
• CLA for individuals

License

Copyright (c) 2016-2017 Atlassian and others. Apache 2.0 licensed, see LICENSE file.

Documentation

Index

• Constants
• type ByNameStore
• type SmartClient

Constants

const (
	Domain = "smith.atlassian.com"

	// See docs/design/managing-resources.md
	CrFieldPathAnnotation  = Domain + "/CrReadyWhenFieldPath"
	CrFieldValueAnnotation = Domain + "/CrReadyWhenFieldValue"

	BundleNameLabel = Domain + "/BundleName"
)

Types

type ByNameStore

type ByNameStore interface {
	Get(gvk schema.GroupVersionKind, namespace, name string) (obj runtime.Object, exists bool, err error)
}

type SmartClient

type SmartClient interface {
	ForGVK(gvk schema.GroupVersionKind, namespace string) (dynamic.ResourceInterface, error)
}