primitives

package
v2.58.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 15, 2024 License: Apache-2.0, BSD-3-Clause, MIT Imports: 22 Imported by: 0

Documentation

Overview

Package primitives contains integration tests for primitives in beam.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func BagStateParDo added in v2.42.0 

func BagStateParDo(s beam.Scope)

BagStateParDo tests a DoFn that uses bag state.

func BagStateParDoClear added in v2.42.0 

func BagStateParDoClear(s beam.Scope)

BagStateParDoClear tests a DoFn that uses bag state.

func Checkpoints added in v2.40.0 

func Checkpoints(s beam.Scope)

Checkpoints is a small test pipeline to establish the correctness of the simple test case.

func CoGBK 

func CoGBK(s beam.Scope)

CoGBK tests CoGBK.

func CombiningStateParDo added in v2.42.0 

func CombiningStateParDo(s beam.Scope)

CombiningStateParDo tests a DoFn that uses value state.

func CreateAndSplit added in v2.35.0 

func CreateAndSplit(s beam.Scope, input []stringPair) beam.PCollection

CreateAndSplit makes a KV PCollection from a list of stringPair types

func Drain added in v2.40.0 

func Drain(s beam.Scope)

Drain tests the SDF truncation during drain.

func Flatten 

func Flatten(s beam.Scope)

Flatten tests flatten.

func FlattenDup 

func FlattenDup(s beam.Scope)

FlattenDups tests flatten with the same input multiple times.

func GBKShortRead added in v2.42.0 

func GBKShortRead(s beam.Scope)

GBKShortRead tests GBK with a short read on the iterator.

func MapStateParDo added in v2.42.0 

func MapStateParDo(s beam.Scope)

MapStateParDo tests a DoFn that uses value state.

func MapStateParDoClear added in v2.42.0 

func MapStateParDoClear(s beam.Scope)

MapStateParDoClear tests clearing and removing from a DoFn that uses map state.

func OomParDo added in v2.41.0 

func OomParDo() *beam.Pipeline

OomParDo tests a DoFn that OOMs.

func Panes added in v2.37.0 

func Panes(s beam.Scope)

Panes constructs a teststream and applies a pardo to get the pane timings.

func ParDoKVSideInput 

func ParDoKVSideInput() *beam.Pipeline

ParDoKVSideInput computes the sum of ints using a KV side input.

func ParDoMultiMapSideInput added in v2.35.0 

func ParDoMultiMapSideInput() *beam.Pipeline

ParDoMultiMapSideInput checks that the multimap side input access pattern works correctly, properly producing the correct output with an asymmetric join.

func ParDoMultiOutput 

func ParDoMultiOutput() *beam.Pipeline

ParDoMultiOutput test a DoFn with multiple output.

func ParDoPipelineOptions added in v2.37.0 

func ParDoPipelineOptions() *beam.Pipeline

ParDoPipelineOptions creates a pipeline with flag options to validate that a DoFn can access them as PipelineOptions.

func ParDoSideInput 

func ParDoSideInput() *beam.Pipeline

ParDoSideInput computes the sum of ints using a side input.

func Reshuffle 

func Reshuffle(s beam.Scope)

Reshuffle tests Reshuffle.

func ReshuffleKV 

func ReshuffleKV(s beam.Scope)

ReshuffleKV tests Reshuffle with KV PCollections.

func SetStateParDo added in v2.42.0 

func SetStateParDo(s beam.Scope)

SetStateParDo tests a DoFn that uses set state.

func SetStateParDoClear added in v2.42.0 

func SetStateParDoClear(s beam.Scope)

SetStateParDoClear tests clearing and removing from a DoFn that uses set state.

func TestStreamBoolSequence 

func TestStreamBoolSequence(s beam.Scope)

TestStreamBoolSequence tests the TestStream primitive by inserting boolean elements then advancing the watermark past the point where they were inserted.

func TestStreamByteSliceSequence 

func TestStreamByteSliceSequence(s beam.Scope)

TestStreamByteSliceSequence tests the TestStream primitive by inserting byte slice elements then advancing the watermark to infinity and comparing the output..

func TestStreamFloat64Sequence 

func TestStreamFloat64Sequence(s beam.Scope)

TestStreamFloat64Sequence tests the TestStream primitive by inserting float64 elements then advancing the watermark past the point where they were inserted.

func TestStreamInt16Sequence added in v2.55.0 

func TestStreamInt16Sequence(s beam.Scope)

TestStreamInt16Sequence validates that a non-beam standard coder works with test stream.

func TestStreamInt64Sequence 

func TestStreamInt64Sequence(s beam.Scope)

TestStreamInt64Sequence tests the TestStream primitive by inserting int64 elements then advancing the watermark past the point where they were inserted.

func TestStreamSimple added in v2.57.0 

func TestStreamSimple(s beam.Scope)

TestStreamSimple is a trivial pipeline where teststream sends a single element to a DoFn that checks that it's received the value. Intended for runner validation.

func TestStreamSimple_InfinityDefault added in v2.57.0 

func TestStreamSimple_InfinityDefault(s beam.Scope)

TestStreamSimple_InfinityDefault is the same trivial pipeline that validates that the watermark is automatically advanced to infinity even when the user doesn't set it. Intended for runner validation.

func TestStreamStrings 

func TestStreamStrings(s beam.Scope)

TestStreamSequence tests the TestStream primitive by inserting string elements then advancing the watermark past the point where they were inserted.

func TestStreamTimersEventTime added in v2.57.0 

func TestStreamTimersEventTime(s beam.Scope)

TestStreamTimersEventTime validates event time timers in a test stream "driven" pipeline.

func TestStreamToGBK added in v2.57.0 

func TestStreamToGBK(s beam.Scope)

TestStreamToGBK is a trivial pipeline where teststream sends a single element to a GBK.

func TestStreamTwoBoolSequences 

func TestStreamTwoBoolSequences(s beam.Scope)

TestStreamTwoBoolSequences tests the TestStream primitive by inserting two sets of boolean elements that arrive on-time into the TestStream

func TestStreamTwoFloat64Sequences 

func TestStreamTwoFloat64Sequences(s beam.Scope)

TestStreamTwoFloat64Sequences tests the TestStream primitive by inserting two sets of float64 elements that arrive on-time into the TestStream

func TestStreamTwoInt64Sequences 

func TestStreamTwoInt64Sequences(s beam.Scope)

TestStreamTwoInt64Sequences tests the TestStream primitive by inserting two sets of int64 elements that arrive on-time into the TestStream

func TestStreamTwoUserTypeSequences added in v2.55.0 

func TestStreamTwoUserTypeSequences(s beam.Scope)

TestStreamTwoUserTypeSequences tests the TestStream primitive by inserting two sets of boolean elements that arrive on-time into the TestStream

func TimersEventTimeBounded added in v2.54.0 

func TimersEventTimeBounded(s beam.Scope)

TimersEventTimeBounded validates event time timers in a bounded pipeline.

func TimersEventTimeUnbounded added in v2.54.0 

func TimersEventTimeUnbounded(s beam.Scope)

TimersEventTimeUnbounded validates event time timers in an unbounded pipeline.

func TimersProcessingTimeTestStream_Infinity added in v2.57.0 

func TimersProcessingTimeTestStream_Infinity(s beam.Scope)

TimersProcessingTimeTestStream_Infinity validates processing time timers in a bounded pipeline kicked along by TestStream.

func TimersProcessingTime_Bounded added in v2.57.0 

func TimersProcessingTime_Bounded(s beam.Scope)

TimersProcessingTimeBounded validates processing time timers in a bounded pipeline.

func TimersProcessingTime_Unbounded added in v2.57.0 

func TimersProcessingTime_Unbounded(s beam.Scope)

TimersProcessingTimeBounded validates processing time timers in an unbounded pipeline.

func TriggerAfterAll added in v2.40.0 

func TriggerAfterAll(s beam.Scope)

TriggerAfterAll tests AfterAll trigger. The output pane is fired when all triggers in the subtriggers are ready. In this test, since trigger.AfterCount(int32(5)) won't be ready unless we see 5 elements, trigger.Always() won't fire until we meet that condition. So we fire only once when we see the 5th element.

func TriggerAfterAny added in v2.40.0 

func TriggerAfterAny(s beam.Scope)

TriggerAfterAny tests AfterAny trigger. In this test, trigger.Always() gets ready everytime. So we would expect panes to be fired at every element irrespective of checking for other triggers.

func TriggerAfterEach added in v2.40.0 

func TriggerAfterEach(s beam.Scope)

TriggerAfterEach tests AfterEach trigger. The output pane is fired after each trigger is ready in the order set in subtriggers. In this test, since trigger.AfterCount(int32(3)) is first, first pane is fired after 3 elements, then a pane is fired each for trigger.Always() for element 5.0 and 8.0

func TriggerAfterEndOfWindow added in v2.34.0 

func TriggerAfterEndOfWindow(s beam.Scope)

TriggerAfterEndOfWindow tests the AfterEndOfWindow Trigger. With AfterCount(2) as the early firing trigger and AfterCount(1) as late firing trigger. It fires two times, one with early firing when there are two elements while the third elements waits in. This third element is fired in the late firing.

func TriggerAfterProcessingTime 

func TriggerAfterProcessingTime(s beam.Scope)

TriggerAfterProcessingTime tests the AfterProcessingTime Trigger, it fires output panes once 't' processing time has passed Not yet supported by the flink runner: java.lang.UnsupportedOperationException: Advancing Processing time is not supported by the Flink Runner.

func TriggerAfterSynchronizedProcessingTime added in v2.40.0 

func TriggerAfterSynchronizedProcessingTime(s beam.Scope)

TriggerAfterSynchronizedProcessingTime tests AfterSynchronizedProcessingTime trigger. It fires at the window expiration since the times doesn't synchronize in this test case.

func TriggerAlways 

func TriggerAlways(s beam.Scope)

TriggerAlways tests the Always trigger, it is expected to receive every input value as the output.

func TriggerDefault 

func TriggerDefault(s beam.Scope)

TriggerDefault tests the default trigger which fires the pane after the end of the window

func TriggerElementCount 

func TriggerElementCount(s beam.Scope)

TriggerElementCount tests the ElementCount Trigger, it waits for atleast N elements to be ready to fire an output pane

func TriggerNever added in v2.40.0 

func TriggerNever(s beam.Scope)

TriggerNever tests Never Trigger. It fires at the window expiration.

func TriggerOrFinally added in v2.40.0 

func TriggerOrFinally(s beam.Scope)

TriggerOrFinally tests OrFinally trigger. The main trigger in this test case trigger.Always() is always ready. But the output is produced only when finally trigger is ready. So it is ready at second element in first window and produces two output panes. Similarly, for the second window.

func TriggerRepeat 

func TriggerRepeat(s beam.Scope)

TriggerRepeat tests the repeat trigger. As of now is it is configure to take only one trigger as a subtrigger. In the below test, it is expected to receive three output panes with two elements each.

func ValidateWindowedSideInputs added in v2.35.0 

func ValidateWindowedSideInputs(s beam.Scope)

ValidateWindowedSideInputs checks that side inputs have accurate windowing information when used.

func ValueStateParDo added in v2.42.0 

func ValueStateParDo(s beam.Scope)

ValueStateParDo tests a DoFn that uses value state.

func ValueStateParDoClear added in v2.42.0 

func ValueStateParDoClear(s beam.Scope)

ValueStateParDoClear tests that a DoFn that uses value state can be cleared.

func ValueStateParDoWindowed added in v2.42.0 

func ValueStateParDoWindowed(s beam.Scope)

ValueStateParDoWindowed tests a DoFn that uses windowed value state.

func WindowSums 

func WindowSums(s beam.Scope, sumPerKey func(beam.Scope, beam.PCollection) beam.PCollection)

WindowSums produces a pipeline that generates the numbers of a 3x3 magic square, and configures the pipeline so that PCollection. Sum is a closure to handle summing data over the window, in a few conditions.

func WindowSums_GBK 

func WindowSums_GBK(s beam.Scope)

func WindowSums_Lifted 

func WindowSums_Lifted(s beam.Scope)

Types

type RangeEstimator added in v2.40.0 

type RangeEstimator struct {
	// contains filtered or unexported fields
}

RangeEstimator implements the offsetrange.RangeEndEstimator interface. It provides the estimated end for a restriction.

func (*RangeEstimator) Estimate added in v2.40.0 

func (r *RangeEstimator) Estimate() int64

Estimate returns the estimated end.

func (*RangeEstimator) SetEstimate added in v2.40.0 

func (r *RangeEstimator) SetEstimate(estimate int64)

SetEstimate sets the estimated end.

type TruncateFn added in v2.40.0 

type TruncateFn struct {
	Estimator RangeEstimator
}

TruncateFn is an SDF.

func (*TruncateFn) CreateInitialRestriction added in v2.40.0 

func (fn *TruncateFn) CreateInitialRestriction(_ []byte) offsetrange.Restriction

CreateInitialRestriction creates an initial restriction

func (*TruncateFn) CreateTracker added in v2.40.0 

func (fn *TruncateFn) CreateTracker(rest offsetrange.Restriction) *sdf.LockRTracker

CreateTracker wraps the given restriction into a LockRTracker type.

func (*TruncateFn) ProcessElement added in v2.40.0 

func (fn *TruncateFn) ProcessElement(ctx context.Context, rt *sdf.LockRTracker, _ []byte, emit func(int64)) sdf.ProcessContinuation

ProcessElement continually gets the start position of the restriction and emits the element as it is.

func (*TruncateFn) RestrictionSize added in v2.40.0 

func (fn *TruncateFn) RestrictionSize(_ []byte, rest offsetrange.Restriction) float64

RestrictionSize returns the size of the current restriction

func (*TruncateFn) SplitRestriction added in v2.40.0 

func (fn *TruncateFn) SplitRestriction(_ []byte, rest offsetrange.Restriction) []offsetrange.Restriction

SplitRestriction is similar to the one used in checkpointing.go test.

func (*TruncateFn) TruncateRestriction added in v2.40.0 

func (fn *TruncateFn) TruncateRestriction(ctx context.Context, rt *sdf.LockRTracker, _ []byte) offsetrange.Restriction

TruncateRestriction truncates the restriction during drain.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.