engine

README

Arcaflow: The Noble Workflow Engine

Arcaflow is a highly-flexible and portable workflow system that helps you to build pipelines of actions via plugins. Plugin steps typically perform one action well, creating or manipulating data that is returned in a machine-readable format. Data is validated according to schemas as it passes through the pipeline in order to clearly diagnose type mismatch problems early. Arcaflow runs on your laptop, a jump host, or in a CI system, requiring only the Arcaflow engine binary, a workflow definition in YAML, and a compatible container runtime.

Complete Arcaflow Documentation

The Arcaflow Engine

The Arcaflow Engine is the core execution component for workflows. It allows you to use actions provided by containerized plugins to build pipelines of work. The Arcaflow engine can be configured to run plugins using Podman, Docker, and Kubernetes.

An ever-growing catalog of official plugins are maintained within the Arcalot organization and are available as versioned containers from Quay.io. You can also build your own containerized plugins using the the Arcaflow SDK, available for Python and Golang. We encourage you to contribute your plugins to the community, and you can start by adding them to the plugins incubator repo via a pull request.

Pre-built engine binaries

Our pre-built engine binaries are available in the releases section for multiple platforms and architectures.

Building from source

This system requires at least Go 1.18 to run and can be built from source:

go build -o arcaflow cmd/arcaflow/main.go

This self-contained engine binary can then be used to run Arcaflow workflows.

Running a simple workflow

A set of example workflows is available to demonstrate workflow features. A basic example workflow.yaml may look like this:

version: v0.2.0  # The compatible workflow schema version
input:  # The input schema for the workflow
  root: RootObject
  objects:
    RootObject:
      id: RootObject
      properties:
        name:
          type:
            type_id: string
steps:  # The individual steps of the workflow
  example:
    plugin:
      deployment_type: image
      src: quay.io/arcalot/arcaflow-plugin-example
    input:
      name: !expr $.input.name
outputs:  # The expected output schema and data for the workflow
  success:
    message: !expr $.steps.example.outputs.success.message

As you can see, a workflow has the root keys of version, input, steps, and outputs. Each of these keys is required in a workflow. Output values and inputs to steps can be specified using the Arcaflow expression language. Input and output references create dependencies between the workflow steps which determine their execution order.

An input YAML file for this basic workflow may look like this:

name: Arca Lot

The Arcaflow engine uses a configuration to define the standard behaviors for deploying plugins within the workflow. The default configuration will use Podman to run the container and will set the log outputs to the info level.

If you have a local Podman setup installed, you can simply run the workflow like this:

arcaflow --input input.yaml

This results in the default behavior of using the built-in configuration and reading the workflow from the workflow.yaml file in the current working directory.

If you don't have a local Podman setup, or if you want to use another deployer or any custom configuration parameters, you can create a config.yaml with your desired parameters. For example:

deployers:
  image: 
    deployer_name: docker
log:
  level: debug
logged_outputs:
  error:
    level: debug

You can load this config by passing the --config flag to Arcaflow.

arcaflow --input input.yaml --config config.yaml

The default workflow file name is workflow.yaml, but you can override this with the --workflow input parameter.

Arcaflow also accepts a --context parameter that defines the base directory for all input files. All relative file paths are from the context directory, and absolute paths are also supported. The default context is the current working directory (.).

A few command examples...

Use the built-in configuration and run the workflow.yaml file from the /my-workflow context directory with no input:

arcaflow --context /my-workflow

Use a custom my-config.yaml configuration file and run the my-workflow.yaml workflow using the my-input.yaml input file from the current directory:

arcaflow --config my-config.yaml --workflow my-workflow.yaml --input my-input.yaml

Use a custom config.yaml configuration file and the default workflow.yaml file from the /my-workflow context directory, and an input.yaml file from the current working directory:

arcaflow --context /my-workflow --config config.yaml --input ${PWD}/input.yaml

Deployers

Image-based deployers are used to deploy plugins to container platforms. Each deployer has configuraiton parameters specific to its platform. These deployers are:

• Podman
• Docker
• Kubernetes

There is also a Python deployer that allows for running Python plugins directly instead of containerized. Note that not all Python plugins may work with the Python deployer, and any plugin dependencies must be present on the target system.

Documentation

Overview

Package engine provides an embeddable engine variant.

Index

• Variables
• func NewDefaultStepRegistry(logger log.Logger, deployerRegistry deployerRegistry.Registry, ...) (step.Registry, error)
• func StepWorkflowPaths(wf *workflow.Workflow) map[string]string
• func SubworkflowCache(wf *workflow.Workflow, rootDir string, converter workflow.YAMLConverter, ...) (loadfile.FileCache, error)
• func SupportedVersion(version string) (string, error)
• type Workflow
• type WorkflowEngine
  • func New(config *config.Config) (WorkflowEngine, error)

Variables

var DefaultDeployerRegistry = registry.New(
	deployer.Any(docker.NewFactory()),
	deployer.Any(kubernetes.NewFactory()),
	deployer.Any(podman.NewFactory()),
	deployer.Any(python.NewFactory()),
)

DefaultDeployerRegistry contains the deployers.

var ErrNoWorkflowFile = fmt.Errorf("no workflow file provided in context")

ErrNoWorkflowFile signals that no workflow file was provided in the context.

Functions

func NewDefaultStepRegistry

func NewDefaultStepRegistry(logger log.Logger, deployerRegistry deployerRegistry.Registry, config *config.Config) (step.Registry, error)

NewDefaultStepRegistry creates a registry with the default step types applied.

func StepWorkflowPaths

func StepWorkflowPaths(wf *workflow.Workflow) map[string]string

StepWorkflowPaths finds all the file paths to workflows referenced in a workflow's steps. The key for each found workflow file is the file path as it is written in this workflow.

func SubworkflowCache

func SubworkflowCache(wf *workflow.Workflow, rootDir string, converter workflow.YAMLConverter, flowCaches []loadfile.FileCache) (loadfile.FileCache, error)

SubworkflowCache creates a file cache of the sub-workflows referenced in this workflow using rootDir as a context.

func SupportedVersion

func SupportedVersion(version string) (string, error)

SupportedVersion confirms whether a given version string is in the set of supported workflow specifications. It returns true when the version is in the set, false otherwise. Earlier schema validation already applies version's regular expression.

Types

type Workflow

type Workflow interface {
	// Run executes the workflow with the passed, YAML-formatted input data.
	Run(
		ctx context.Context,
		input []byte,
	) (
		outputID string,
		outputData any,
		outputIsError bool,
		err error,
	)

	// InputSchema returns the requested input schema for the workflow.
	InputSchema() schema.Scope
	// Outputs returns the list of possible outputs and their schema for the workflow.
	Outputs() map[string]schema.StepOutput
	Namespaces() map[string]map[string]*schema.ObjectSchema
}

Workflow is a runnable, queryable workflow. You can execute it, or query it for schema information.

type WorkflowEngine

type WorkflowEngine interface {
	// RunWorkflow is a simplified shortcut to parse and immediately run a workflow.
	RunWorkflow(
		ctx context.Context,
		input []byte,
		workflowContext loadfile.FileCache,
		workflowFileName string,
	) (outputID string, outputData any, outputError bool, err error)

	// Parse ingests a workflow context as a map of files to their contents and a workflow file name and
	// parses the data into an executable workflow.
	Parse(
		workflowContext loadfile.FileCache,
		workflowFileName string,
	) (
		workflow Workflow,
		err error,
	)
}

WorkflowEngine is responsible for executing workflows and returning their result.

func New

func New(
	config *config.Config,
) (WorkflowEngine, error)

New creates a new workflow engine with the provided configuration. The passed deployerRegistry is responsible for providing deployment plugins.