runnerlib

Documentation

Overview

Package runnerlib contains utilities for submitting Go pipelines to a Beam model runner.

Index

• func BuildTempWorkerBinary(ctx context.Context) (string, error)
• func BuildWorkerBinary(ctx context.Context, filename string) error
• func Execute(ctx context.Context, p *pb.Pipeline, endpoint string, opt *JobOptions, ...) (string, error)
• func Prepare(ctx context.Context, client jobpb.JobServiceClient, p *pb.Pipeline, ...) (id, endpoint string, err error)
• func Stage(ctx context.Context, id, endpoint, binary string, files ...artifact.KeyedFile) (string, error)
• func Submit(ctx context.Context, client jobpb.JobServiceClient, id, token string) (string, error)
• func WaitForCompletion(ctx context.Context, client jobpb.JobServiceClient, jobID string) error
• type JobOptions

Functions

func BuildTempWorkerBinary

func BuildTempWorkerBinary(ctx context.Context) (string, error)

BuildTempWorkerBinary creates a local worker binary in the tmp directory for linux/amd64. Caller responsible for deleting the binary.

func BuildWorkerBinary

func BuildWorkerBinary(ctx context.Context, filename string) error

BuildWorkerBinary creates a local worker binary for linux/amd64. It finds the filename by examining the call stack. We want the user entry (*), for example:

/Users/herohde/go/src/github.com/apache/beam/sdks/go/pkg/beam/runners/beamexec/main.go (skip: 2)

• /Users/herohde/go/src/github.com/apache/beam/sdks/go/examples/wordcount/wordcount.go (skip: 3) /usr/local/go/src/runtime/proc.go (skip: 4) // not always present /usr/local/go/src/runtime/asm_amd64.s (skip: 4 or 5)

func Execute

func Execute(ctx context.Context, p *pb.Pipeline, endpoint string, opt *JobOptions, async bool) (string, error)

Execute executes a pipeline on the universal runner serving the given endpoint. Convenience function.

func Prepare

func Prepare(ctx context.Context, client jobpb.JobServiceClient, p *pb.Pipeline, opt *JobOptions) (id, endpoint string, err error)

Prepare prepares a job to the given job service. It returns the preparation id and artifact staging endpoint, if successful.

func Stage

func Stage(ctx context.Context, id, endpoint, binary string, files ...artifact.KeyedFile) (string, error)

Stage stages the worker binary and any additional files to the given artifact staging endpoint. It returns the commit token if successful.

func Submit

func Submit(ctx context.Context, client jobpb.JobServiceClient, id, token string) (string, error)

Submit submits a job to the given job service. It returns a jobID, if successful.

func WaitForCompletion

func WaitForCompletion(ctx context.Context, client jobpb.JobServiceClient, jobID string) error

WaitForCompletion monitors the given job until completion. It logs any messages and state changes received.

Types

type JobOptions

type JobOptions struct {
	// Name is the job name.
	Name string
	// Experiments are additional experiments.
	Experiments []string

	// Worker is the worker binary override.
	Worker string

	// InternalJavaRunner is the class of the receiving Java runner. To be removed.
	InternalJavaRunner string
}

JobOptions capture the various options for submitting jobs to universal runners.