workflow

Documentation

Overview

Package workflow includes utilities for common cipkg workflow:

• Generator: A Generator implementation for *core.Action which handles dependencies' platforms for cross-compilation.
• LocalPackageManager: A persistent package manager implementation using filesystem for storage and fslock for reference counting.
• Buillder: A standard build workflow builds packages from given generators.

Index

• func Execute(ctx context.Context, cfg *ExecutionConfig, drv *core.Derivation) error
• func MustDecRefRecursiveRuntime(pkg actions.Package)
• func MustIncRefRecursiveRuntime(pkg actions.Package)
• type Builder
  • func NewBuilder(plats generators.Platforms, pm core.PackageManager, ...) *Builder
  • func (b *Builder) Build(ctx context.Context, pe *PackageExecutor, g generators.Generator) (actions.Package, error)
  • func (b *Builder) BuildAll(ctx context.Context, pe *PackageExecutor, gs []generators.Generator) ([]actions.Package, error)
  • func (b *Builder) BuildPackages(ctx context.Context, pe *PackageExecutor, pkgs []actions.Package, ...) error
  • func (b *Builder) GeneratePackages(ctx context.Context, gs []generators.Generator) ([]actions.Package, error)
• type ExecutionConfig
• type Executor
• type Generator
  • func (g *Generator) Generate(ctx context.Context, plats generators.Platforms) (*core.Action, error)
• type LocalPackageManager
  • func NewLocalPackageManager(path string) (*LocalPackageManager, error)
  • func (pm *LocalPackageManager) Get(id string) core.PackageHandler
  • func (pm *LocalPackageManager) Prune(c context.Context, ttl time.Duration, max int)
• type PackageExecutor
  • func NewPackageExecutor(tempDir string, preExpandFn PreExpandHook, preExecFn PreExecuteHook, ...) *PackageExecutor
  • func (p *PackageExecutor) Execute(ctx context.Context, pkg actions.Package) error
  • func (p *PackageExecutor) Expand(ctx context.Context, pkg actions.Package) ([]actions.Package, error)
  • func (p *PackageExecutor) Release() error
• type PostExecuteHook
• type PreExecuteHook
• type PreExpandHook

Functions

func Execute

func Execute(ctx context.Context, cfg *ExecutionConfig, drv *core.Derivation) error

Execute is the default Executor which runs the command presented by the derivation.

func MustDecRefRecursiveRuntime

func MustDecRefRecursiveRuntime(pkg actions.Package)

MustDecRefRecursiveRuntime will DecRef the package with all its runtime dependencies recursively. If an error happened, it may panic with only part of the packages are dereferenced.

func MustIncRefRecursiveRuntime

func MustIncRefRecursiveRuntime(pkg actions.Package)

MustIncRefRecursiveRuntime will IncRef the package with all its runtime dependencies recursively. If an error happened, it may panic with only part of the packages are referenced.

Types

type Builder

type Builder struct {
	// contains filtered or unexported fields
}

func NewBuilder

func NewBuilder(plats generators.Platforms, pm core.PackageManager, ap *actions.ActionProcessor) *Builder

NewBuilder creates a Builder to manage the standard build workflow for actions.Package by converting generators.Generator to actions.Package and execute the *core.Derivation to make the package available locally.

func (*Builder) Build

func (b *Builder) Build(ctx context.Context, pe *PackageExecutor, g generators.Generator) (actions.Package, error)

Build triggers the standard workflow converting generators.Generator to actions.Package which are available in the storage.

func (*Builder) BuildAll

func (b *Builder) BuildAll(ctx context.Context, pe *PackageExecutor, gs []generators.Generator) ([]actions.Package, error)

BuildAll triggers the standard workflow converting []generators.Generator to []actions.Package which are available in the storage.

func (*Builder) BuildPackages

func (b *Builder) BuildPackages(ctx context.Context, pe *PackageExecutor, pkgs []actions.Package, continueOnExecError bool) error

BuildPackages builds the packages and make all packages available in the storage. All packages will be dereferenced after the build. Leave it to the user to decide those of which packages will be used at the runtime. There may be a chance that a package is removed during the short amount of time. But since IncRef will update the last accessed timestamp, cleaning up with any reasonable time window (e.g. 1 hour) is highly unlikely to remove packages just dereferenced and may be IncRef within seconds. And even if it's happened, the caller can retry the process.

func (*Builder) GeneratePackages

func (b *Builder) GeneratePackages(ctx context.Context, gs []generators.Generator) ([]actions.Package, error)

GeneratePackages triggers the standard workflow converting []generators.Generator to []actions.Package without building them.

type ExecutionConfig

type ExecutionConfig struct {
	OutputDir  string
	WorkingDir string

	Stdin  io.Reader
	Stdout io.Writer
	Stderr io.Writer
}

ExecutionConfig includes all configs for Executor.

type Executor

type Executor func(ctx context.Context, cfg *ExecutionConfig, drv *core.Derivation) error

Executor is the function interface for executing the provided derivation. This can be subject to the platform or using sandbox for isolation.

type Generator

type Generator struct {
	Name         string
	Metadata     *core.Action_Metadata
	Args         []string
	Env          environ.Env
	Dependencies []generators.Dependency
}

Generator is a general purpose generator which generates dependencies and computes cross-compile tuples recursively. This Generator implements:

• Add runtime dependencies to action metadata.
• Recursively generating all its dependencies.
• Add environment variables for the list of different types of dependencies.
• Convert generators.Platforms based on the types of the dependency for cross-compiling. e.g. A static libary for host platform should be DepsHostTarget dependency, while a libary for build platform should be DepsBuildHost.

This Generator implementation is preferred to avoid manually handling dependencies.

func (*Generator) Generate

func (g *Generator) Generate(ctx context.Context, plats generators.Platforms) (*core.Action, error)

Generate will recursively generates the *core.Action from its content. The output directories of dependencies will be added to environment variables for build scripts to interact with.

type LocalPackageManager

type LocalPackageManager struct {
	// contains filtered or unexported fields
}

LocalPackageManager is a PackageManager implementation that stores packages locally. It supports recording package references acrossing multiple instances using fslock.

func NewLocalPackageManager

func NewLocalPackageManager(path string) (*LocalPackageManager, error)

func (*LocalPackageManager) Get

func (pm *LocalPackageManager) Get(id string) core.PackageHandler

Get returns the handler for the package.

func (*LocalPackageManager) Prune

func (pm *LocalPackageManager) Prune(c context.Context, ttl time.Duration, max int)

Prune try remove packages if: - Not available, or - Haven't been used for `ttl` time.

type PackageExecutor

type PackageExecutor struct {
	// contains filtered or unexported fields
}

PackageExecutor is the executor for package to be built by running executor. It will ensure all package's dependencies become available in storage when the package is executed.

func NewPackageExecutor

func NewPackageExecutor(tempDir string, preExpandFn PreExpandHook, preExecFn PreExecuteHook, postExecFn PostExecuteHook, execFn Executor) *PackageExecutor

NewPackageExecutor creates a package executor to make packages available. All hooks and execFn are optional. - preExpandFn can be provided to e.g. fetch package from remote cache. - preExecFn can be provided to e.g. setup execution environment. - postExecFn can be provided to e.g. cleanup execution environment. - If execFn is nil, builder.Execute will be used. tempDir will be used as the working directory during execution and can be removed by caller after execution.

func (*PackageExecutor) Execute

func (p *PackageExecutor) Execute(ctx context.Context, pkg actions.Package) error

Execute executes packages' derivations and keeps the reference to the package. PreExecHook and PostExecHook, if not nil, are invoked before and after the execution. If PreExecHook returns error, Execute will be skipped but PostExecHook should be invoked with the error.

func (*PackageExecutor) Expand

func (p *PackageExecutor) Expand(ctx context.Context, pkg actions.Package) ([]actions.Package, error)

Expand expands a package with all its dependencies to a flattened slice of packages for execution. If preExpandFn is not empty, it will be executed to possibly make the package available so its dependencies don't need to be expanded. preExpandFn will be invoked on the package before on it's dependencies.

func (*PackageExecutor) Release

func (p *PackageExecutor) Release() error

Release releases all packages referenced by the PackageExecutor.

type PostExecuteHook

type PostExecuteHook func(ctx context.Context, pkg actions.Package, execErr error) error

type PreExecuteHook

type PreExecuteHook func(ctx context.Context, pkg actions.Package) (context.Context, error)

type PreExpandHook

type PreExpandHook func(ctx context.Context, pkg actions.Package) error