dataflowlib

Documentation

Overview

Package dataflowlib translates a Beam pipeline model to the Dataflow API job model, for submission to Google Cloud Dataflow.

Index

• func Execute(ctx context.Context, raw *pb.Pipeline, opts *JobOptions, ...) (string, error)
• func Fixup(p *pb.Pipeline) (*pb.Pipeline, error)
• func NewClient(ctx context.Context, endpoint string) (*df.Service, error)
• func PrintJob(ctx context.Context, job *df.Job)
• func StageModel(ctx context.Context, project, modelURL string, model []byte) error
• func StageWorker(ctx context.Context, project, workerURL, worker string) error
• func Submit(ctx context.Context, client *df.Service, project, region string, job *df.Job) (*df.Job, error)
• func Translate(p *pb.Pipeline, opts *JobOptions, workerURL, modelURL string) (*df.Job, error)
• func WaitForCompletion(ctx context.Context, client *df.Service, project, region, jobID string) error
• type JobOptions

Functions

func Execute

func Execute(ctx context.Context, raw *pb.Pipeline, opts *JobOptions, workerURL, modelURL, endpoint string, async bool) (string, error)

Execute submits a pipeline as a Dataflow job.

func Fixup

func Fixup(p *pb.Pipeline) (*pb.Pipeline, error)

Fixup proto pipeline with Dataflow quirks.

func NewClient

func NewClient(ctx context.Context, endpoint string) (*df.Service, error)

NewClient creates a new dataflow client with default application credentials and CloudPlatformScope. The Dataflow endpoint is optionally overridden.

func PrintJob

func PrintJob(ctx context.Context, job *df.Job)

PrintJob logs the Dataflow job.

func StageModel

func StageModel(ctx context.Context, project, modelURL string, model []byte) error

StageModel uploads the pipeline model to GCS as a unique object.

func StageWorker

func StageWorker(ctx context.Context, project, workerURL, worker string) error

StageWorker uploads the worker binary to GCS as a unique object.

func Submit

func Submit(ctx context.Context, client *df.Service, project, region string, job *df.Job) (*df.Job, error)

Submit submits a prepared job to Cloud Dataflow.

func Translate

func Translate(p *pb.Pipeline, opts *JobOptions, workerURL, modelURL string) (*df.Job, error)

Translate translates a pipeline to a Dataflow job.

func WaitForCompletion

func WaitForCompletion(ctx context.Context, client *df.Service, project, region, jobID string) error

WaitForCompletion monitors the given job until completion. It logs any messages and state changes received.

Types

type JobOptions

type JobOptions struct {
	// Name is the job name.
	Name string
	// Experiments are additional experiments.
	Experiments []string
	// Pipeline options
	Options runtime.RawOptions

	Project     string
	Region      string
	Zone        string
	Network     string
	NumWorkers  int64
	MachineType string
	Labels      map[string]string

	TempLocation string

	// Worker is the worker binary override.
	Worker string

	TeardownPolicy string
}

JobOptions capture the various options for submitting jobs to Dataflow.