pipelinex

package
v2.22.0+incompatible Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 3, 2020 License: Apache-2.0 Imports: 6 Imported by: 0

Documentation

Overview

Package pipelinex contains utilities for manipulating Beam proto pipelines. The utilities generally uses shallow copies and do not mutate their inputs.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Bounded

func Bounded(p *pipepb.Pipeline) bool

Bounded returns true iff all PCollections are bounded.

func ContainerImages

func ContainerImages(p *pipepb.Pipeline) []string

ContainerImages returns the set of container images used in the given pipeline.

func Normalize

func Normalize(p *pipepb.Pipeline) (*pipepb.Pipeline, error)

Normalize recomputes derivative information in the pipeline, such as roots and input/output for composite transforms. It also ensures that unique names are so and topologically sorts each subtransform list.

func ShallowCloneFunctionSpec

func ShallowCloneFunctionSpec(p *pb.FunctionSpec) *pb.FunctionSpec

ShallowCloneFunctionSpec makes a shallow copy of the given FunctionSpec.

func ShallowClonePTransform

func ShallowClonePTransform(t *pb.PTransform) *pb.PTransform

ShallowClonePTransform makes a shallow copy of the given PTransform.

func ShallowCloneParDoPayload

func ShallowCloneParDoPayload(p *pb.ParDoPayload) *pb.ParDoPayload

ShallowCloneParDoPayload makes a shallow copy of the given ParDoPayload.

func ShallowCloneSideInput

func ShallowCloneSideInput(p *pb.SideInput) *pb.SideInput

ShallowCloneSideInput makes a shallow copy of the given SideInput.

func TopologicalSort

func TopologicalSort(xforms map[string]*pipepb.PTransform, ids []string) []string

TopologicalSort returns a topologically sorted list of the given ids, generally from the same scope/composite. Assumes acyclic graph.

func TrimCoders

func TrimCoders(coders map[string]*pipepb.Coder, ids ...string) map[string]*pipepb.Coder

TrimCoders returns the transitive closure of the given coders ids.

func Update

func Update(p *pipepb.Pipeline, values *pipepb.Components) (*pipepb.Pipeline, error)

Update merges a pipeline with the given components, which may add, replace or delete its values. It returns the merged pipeline. The input is not modified.

Types

This section is empty.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL