pipelinex

Documentation

Overview

Package pipelinex contains utilities for manipulating Beam proto pipelines. The utilities generally uses shallow copies and do not mutate their inputs.

Index

• func Bounded(p *pipepb.Pipeline) bool
• func ContainerImages(p *pipepb.Pipeline) []string
• func Normalize(p *pipepb.Pipeline) (*pipepb.Pipeline, error)
• func ShallowCloneFunctionSpec(p *pb.FunctionSpec) *pb.FunctionSpec
• func ShallowClonePTransform(t *pb.PTransform) *pb.PTransform
• func ShallowCloneParDoPayload(p *pb.ParDoPayload) *pb.ParDoPayload
• func ShallowCloneSideInput(p *pb.SideInput) *pb.SideInput
• func TopologicalSort(xforms map[string]*pipepb.PTransform, ids []string) []string
• func TrimCoders(coders map[string]*pipepb.Coder, ids ...string) map[string]*pipepb.Coder
• func Update(p *pipepb.Pipeline, values *pipepb.Components) (*pipepb.Pipeline, error)

Functions

func Bounded

func Bounded(p *pipepb.Pipeline) bool

Bounded returns true iff all PCollections are bounded.

func ContainerImages

func ContainerImages(p *pipepb.Pipeline) []string

ContainerImages returns the set of container images used in the given pipeline.

func Normalize

func Normalize(p *pipepb.Pipeline) (*pipepb.Pipeline, error)

Normalize recomputes derivative information in the pipeline, such as roots and input/output for composite transforms. It also ensures that unique names are so and topologically sorts each subtransform list.

func ShallowCloneFunctionSpec

func ShallowCloneFunctionSpec(p *pb.FunctionSpec) *pb.FunctionSpec

ShallowCloneFunctionSpec makes a shallow copy of the given FunctionSpec.

func ShallowClonePTransform

func ShallowClonePTransform(t *pb.PTransform) *pb.PTransform

ShallowClonePTransform makes a shallow copy of the given PTransform.

func ShallowCloneParDoPayload

func ShallowCloneParDoPayload(p *pb.ParDoPayload) *pb.ParDoPayload

ShallowCloneParDoPayload makes a shallow copy of the given ParDoPayload.

func ShallowCloneSideInput

func ShallowCloneSideInput(p *pb.SideInput) *pb.SideInput

ShallowCloneSideInput makes a shallow copy of the given SideInput.

func TopologicalSort

func TopologicalSort(xforms map[string]*pipepb.PTransform, ids []string) []string

TopologicalSort returns a topologically sorted list of the given ids, generally from the same scope/composite. Assumes acyclic graph.

func TrimCoders

func TrimCoders(coders map[string]*pipepb.Coder, ids ...string) map[string]*pipepb.Coder

TrimCoders returns the transitive closure of the given coders ids.

func Update

func Update(p *pipepb.Pipeline, values *pipepb.Components) (*pipepb.Pipeline, error)

Update merges a pipeline with the given components, which may add, replace or delete its values. It returns the merged pipeline. The input is not modified.