runnerlib

Documentation

Overview

Package runnerlib contains utilities for submitting Go pipelines to a Beam model runner.

Index

• func BuildTempWorkerBinary(ctx context.Context, opts CompileOpts) (string, error)
• func Execute(ctx context.Context, p *pipepb.Pipeline, endpoint string, opt *JobOptions, ...) (*universalPipelineResult, error)
• func IsWorkerCompatibleBinary() (string, bool)
• func Prepare(ctx context.Context, client jobpb.JobServiceClient, p *pipepb.Pipeline, ...) (id, endpoint, stagingToken string, err error)
• func Stage(ctx context.Context, id, endpoint, binary, st string) (retrievalToken string, err error)
• func StageViaLegacyAPI(ctx context.Context, cc *grpc.ClientConn, binary, st string) (retrievalToken string, err error)
• func StageViaPortableAPI(ctx context.Context, cc *grpc.ClientConn, binary, st string) (retErr error)
• func Submit(ctx context.Context, client jobpb.JobServiceClient, id, token string) (string, error)
• func UpdateGoEnvironmentWorker(worker string, p *pipepb.Pipeline) error
• func WaitForCompletion(ctx context.Context, client jobpb.JobServiceClient, jobID string) error
• type CompileOpts
• type JobOptions

Functions

func BuildTempWorkerBinary

func BuildTempWorkerBinary(ctx context.Context, opts CompileOpts) (string, error)

BuildTempWorkerBinary creates a local worker binary in the tmp directory for linux/amd64. Caller responsible for deleting the binary.

func Execute

func Execute(ctx context.Context, p *pipepb.Pipeline, endpoint string, opt *JobOptions, async bool) (*universalPipelineResult, error)

Execute executes a pipeline on the universal runner serving the given endpoint. Convenience function.

func IsWorkerCompatibleBinary

func IsWorkerCompatibleBinary() (string, bool)

IsWorkerCompatibleBinary returns the path to itself and true if running a linux-amd64 binary that can directly be used as a worker binary.

func Prepare

func Prepare(ctx context.Context, client jobpb.JobServiceClient, p *pipepb.Pipeline, opt *JobOptions) (id, endpoint, stagingToken string, err error)

Prepare prepares a job to the given job service. It returns the preparation id artifact staging endpoint, and staging token if successful.

func Stage

func Stage(ctx context.Context, id, endpoint, binary, st string) (retrievalToken string, err error)

Stage stages the worker binary and any additional files to the given artifact staging endpoint. It returns the retrieval token if successful.

func StageViaLegacyAPI

func StageViaLegacyAPI(ctx context.Context, cc *grpc.ClientConn, binary, st string) (retrievalToken string, err error)

StageViaLegacyAPI is a beam internal function for uploading artifacts to the staging service via the legacy API.

It will be unexported at a later time.

func StageViaPortableAPI

func StageViaPortableAPI(ctx context.Context, cc *grpc.ClientConn, binary, st string) (retErr error)

StageViaPortableAPI is a beam internal function for uploading artifacts to the staging service via the portable API.

It will be unexported at a later time.

func Submit

func Submit(ctx context.Context, client jobpb.JobServiceClient, id, token string) (string, error)

Submit submits a job to the given job service. It returns a jobID, if successful.

func UpdateGoEnvironmentWorker

func UpdateGoEnvironmentWorker(worker string, p *pipepb.Pipeline) error

UpdateGoEnvironmentWorker sets the worker artifact payload in the default environment.

func WaitForCompletion

func WaitForCompletion(ctx context.Context, client jobpb.JobServiceClient, jobID string) error

WaitForCompletion monitors the given job until completion. It logs any messages and state changes received.

Types

type CompileOpts

type CompileOpts struct {
	OS, Arch string
}

CompileOpts are additional options for dynamic compiles of the local code for development purposes. Production runs should build the worker binary separately for the target environment. See https://beam.apache.org/documentation/sdks/go-cross-compilation/ for details.

type JobOptions

type JobOptions struct {
	// Name is the job name.
	Name string
	// Experiments are additional experiments.
	Experiments []string

	// Worker is the worker binary override.
	Worker string

	// RetainDocker is an option to pass to the runner indicating the docker containers should be cached.
	RetainDocker bool

	// Indicates a limit on parallelism the runner should impose.
	Parallelism int

	// Loopback indicates this job is running in loopback mode and will reconnect to the local process.
	Loopback bool
}

JobOptions capture the various options for submitting jobs to universal runners.