primitives

package
v2.0.0-...-3952df4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 24, 2024 License: Apache-2.0, BSD-3-Clause, MIT Imports: 22 Imported by: 0

Documentation

Overview

Package primitives contains integration tests for primitives in beam.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func BagStateParDo 

func BagStateParDo(s beam.Scope)

BagStateParDo tests a DoFn that uses bag state.

func BagStateParDoClear 

func BagStateParDoClear(s beam.Scope)

BagStateParDoClear tests a DoFn that uses bag state.

func Checkpoints 

func Checkpoints(s beam.Scope)

Checkpoints is a small test pipeline to establish the correctness of the simple test case.

func CoGBK 

func CoGBK(s beam.Scope)

CoGBK tests CoGBK.

func CombiningStateParDo 

func CombiningStateParDo(s beam.Scope)

CombiningStateParDo tests a DoFn that uses value state.

func CreateAndSplit 

func CreateAndSplit(s beam.Scope, input []stringPair) beam.PCollection

CreateAndSplit makes a KV PCollection from a list of stringPair types

func Drain 

func Drain(s beam.Scope)

Drain tests the SDF truncation during drain.

func Flatten 

func Flatten(s beam.Scope)

Flatten tests flatten.

func FlattenDup 

func FlattenDup(s beam.Scope)

FlattenDups tests flatten with the same input multiple times.

func GBKShortRead 

func GBKShortRead(s beam.Scope)

GBKShortRead tests GBK with a short read on the iterator.

func MapStateParDo 

func MapStateParDo(s beam.Scope)

MapStateParDo tests a DoFn that uses value state.

func MapStateParDoClear 

func MapStateParDoClear(s beam.Scope)

MapStateParDoClear tests clearing and removing from a DoFn that uses map state.

func OomParDo 

func OomParDo() *beam.Pipeline

OomParDo tests a DoFn that OOMs.

func Panes 

func Panes(s beam.Scope)

Panes constructs a teststream and applies a pardo to get the pane timings.

func ParDoKVSideInput 

func ParDoKVSideInput() *beam.Pipeline

ParDoKVSideInput computes the sum of ints using a KV side input.

func ParDoMultiMapSideInput 

func ParDoMultiMapSideInput() *beam.Pipeline

ParDoMultiMapSideInput checks that the multimap side input access pattern works correctly, properly producing the correct output with an asymmetric join.

func ParDoMultiOutput 

func ParDoMultiOutput() *beam.Pipeline

ParDoMultiOutput test a DoFn with multiple output.

func ParDoPipelineOptions 

func ParDoPipelineOptions() *beam.Pipeline

ParDoPipelineOptions creates a pipeline with flag options to validate that a DoFn can access them as PipelineOptions.

func ParDoSideInput 

func ParDoSideInput() *beam.Pipeline

ParDoSideInput computes the sum of ints using a side input.

func Reshuffle 

func Reshuffle(s beam.Scope)

Reshuffle tests Reshuffle.

func ReshuffleKV 

func ReshuffleKV(s beam.Scope)

ReshuffleKV tests Reshuffle with KV PCollections.

func SetStateParDo 

func SetStateParDo(s beam.Scope)

SetStateParDo tests a DoFn that uses set state.

func SetStateParDoClear 

func SetStateParDoClear(s beam.Scope)

SetStateParDoClear tests clearing and removing from a DoFn that uses set state.

func TestStreamBoolSequence 

func TestStreamBoolSequence(s beam.Scope)

TestStreamBoolSequence tests the TestStream primitive by inserting boolean elements then advancing the watermark past the point where they were inserted.

func TestStreamByteSliceSequence 

func TestStreamByteSliceSequence(s beam.Scope)

TestStreamByteSliceSequence tests the TestStream primitive by inserting byte slice elements then advancing the watermark to infinity and comparing the output..

func TestStreamFloat64Sequence 

func TestStreamFloat64Sequence(s beam.Scope)

TestStreamFloat64Sequence tests the TestStream primitive by inserting float64 elements then advancing the watermark past the point where they were inserted.

func TestStreamInt16Sequence 

func TestStreamInt16Sequence(s beam.Scope)

TestStreamInt16Sequence validates that a non-beam standard coder works with test stream.

func TestStreamInt64Sequence 

func TestStreamInt64Sequence(s beam.Scope)

TestStreamInt64Sequence tests the TestStream primitive by inserting int64 elements then advancing the watermark past the point where they were inserted.

func TestStreamSimple 

func TestStreamSimple(s beam.Scope)

TestStreamSimple is a trivial pipeline where teststream sends a single element to a DoFn that checks that it's received the value. Intended for runner validation.

func TestStreamSimple_InfinityDefault 

func TestStreamSimple_InfinityDefault(s beam.Scope)

TestStreamSimple_InfinityDefault is the same trivial pipeline that validates that the watermark is automatically advanced to infinity even when the user doesn't set it. Intended for runner validation.

func TestStreamStrings 

func TestStreamStrings(s beam.Scope)

TestStreamSequence tests the TestStream primitive by inserting string elements then advancing the watermark past the point where they were inserted.

func TestStreamTimersEventTime 

func TestStreamTimersEventTime(s beam.Scope)

TestStreamTimersEventTime validates event time timers in a test stream "driven" pipeline.

func TestStreamToGBK 

func TestStreamToGBK(s beam.Scope)

TestStreamToGBK is a trivial pipeline where teststream sends a single element to a GBK.

func TestStreamTwoBoolSequences 

func TestStreamTwoBoolSequences(s beam.Scope)

TestStreamTwoBoolSequences tests the TestStream primitive by inserting two sets of boolean elements that arrive on-time into the TestStream

func TestStreamTwoFloat64Sequences 

func TestStreamTwoFloat64Sequences(s beam.Scope)

TestStreamTwoFloat64Sequences tests the TestStream primitive by inserting two sets of float64 elements that arrive on-time into the TestStream

func TestStreamTwoInt64Sequences 

func TestStreamTwoInt64Sequences(s beam.Scope)

TestStreamTwoInt64Sequences tests the TestStream primitive by inserting two sets of int64 elements that arrive on-time into the TestStream

func TestStreamTwoUserTypeSequences 

func TestStreamTwoUserTypeSequences(s beam.Scope)

TestStreamTwoUserTypeSequences tests the TestStream primitive by inserting two sets of boolean elements that arrive on-time into the TestStream

func TimersEventTimeBounded 

func TimersEventTimeBounded(s beam.Scope)

TimersEventTimeBounded validates event time timers in a bounded pipeline.

func TimersEventTimeUnbounded 

func TimersEventTimeUnbounded(s beam.Scope)

TimersEventTimeUnbounded validates event time timers in an unbounded pipeline.

func TimersProcessingTimeTestStream_Infinity 

func TimersProcessingTimeTestStream_Infinity(s beam.Scope)

TimersProcessingTimeTestStream_Infinity validates processing time timers in a bounded pipeline kicked along by TestStream.

func TimersProcessingTime_Bounded 

func TimersProcessingTime_Bounded(s beam.Scope)

TimersProcessingTimeBounded validates processing time timers in a bounded pipeline.

func TimersProcessingTime_Unbounded 

func TimersProcessingTime_Unbounded(s beam.Scope)

TimersProcessingTimeBounded validates processing time timers in an unbounded pipeline.

func TriggerAfterAll 

func TriggerAfterAll(s beam.Scope)

TriggerAfterAll tests AfterAll trigger. The output pane is fired when all triggers in the subtriggers are ready. In this test, since trigger.AfterCount(int32(5)) won't be ready unless we see 5 elements, trigger.Always() won't fire until we meet that condition. So we fire only once when we see the 5th element.

func TriggerAfterAny 

func TriggerAfterAny(s beam.Scope)

TriggerAfterAny tests AfterAny trigger. In this test, trigger.Always() gets ready everytime. So we would expect panes to be fired at every element irrespective of checking for other triggers.

func TriggerAfterEach 

func TriggerAfterEach(s beam.Scope)

TriggerAfterEach tests AfterEach trigger. The output pane is fired after each trigger is ready in the order set in subtriggers. In this test, since trigger.AfterCount(int32(3)) is first, first pane is fired after 3 elements, then a pane is fired each for trigger.Always() for element 5.0 and 8.0

func TriggerAfterEndOfWindow 

func TriggerAfterEndOfWindow(s beam.Scope)

TriggerAfterEndOfWindow tests the AfterEndOfWindow Trigger. With AfterCount(2) as the early firing trigger and AfterCount(1) as late firing trigger. It fires two times, one with early firing when there are two elements while the third elements waits in. This third element is fired in the late firing.

func TriggerAfterProcessingTime 

func TriggerAfterProcessingTime(s beam.Scope)

TriggerAfterProcessingTime tests the AfterProcessingTime Trigger, it fires output panes once 't' processing time has passed Not yet supported by the flink runner: java.lang.UnsupportedOperationException: Advancing Processing time is not supported by the Flink Runner.

func TriggerAfterSynchronizedProcessingTime 

func TriggerAfterSynchronizedProcessingTime(s beam.Scope)

TriggerAfterSynchronizedProcessingTime tests AfterSynchronizedProcessingTime trigger. It fires at the window expiration since the times doesn't synchronize in this test case.

func TriggerAlways 

func TriggerAlways(s beam.Scope)

TriggerAlways tests the Always trigger, it is expected to receive every input value as the output.

func TriggerDefault 

func TriggerDefault(s beam.Scope)

TriggerDefault tests the default trigger which fires the pane after the end of the window

func TriggerElementCount 

func TriggerElementCount(s beam.Scope)

TriggerElementCount tests the ElementCount Trigger, it waits for atleast N elements to be ready to fire an output pane

func TriggerNever 

func TriggerNever(s beam.Scope)

TriggerNever tests Never Trigger. It fires at the window expiration.

func TriggerOrFinally 

func TriggerOrFinally(s beam.Scope)

TriggerOrFinally tests OrFinally trigger. The main trigger in this test case trigger.Always() is always ready. But the output is produced only when finally trigger is ready. So it is ready at second element in first window and produces two output panes. Similarly, for the second window.

func TriggerRepeat 

func TriggerRepeat(s beam.Scope)

TriggerRepeat tests the repeat trigger. As of now is it is configure to take only one trigger as a subtrigger. In the below test, it is expected to receive three output panes with two elements each.

func ValidateWindowedSideInputs 

func ValidateWindowedSideInputs(s beam.Scope)

ValidateWindowedSideInputs checks that side inputs have accurate windowing information when used.

func ValueStateParDo 

func ValueStateParDo(s beam.Scope)

ValueStateParDo tests a DoFn that uses value state.

func ValueStateParDoClear 

func ValueStateParDoClear(s beam.Scope)

ValueStateParDoClear tests that a DoFn that uses value state can be cleared.

func ValueStateParDoWindowed 

func ValueStateParDoWindowed(s beam.Scope)

ValueStateParDoWindowed tests a DoFn that uses windowed value state.

func WindowSums 

func WindowSums(s beam.Scope, sumPerKey func(beam.Scope, beam.PCollection) beam.PCollection)

WindowSums produces a pipeline that generates the numbers of a 3x3 magic square, and configures the pipeline so that PCollection. Sum is a closure to handle summing data over the window, in a few conditions.

func WindowSums_GBK 

func WindowSums_GBK(s beam.Scope)

func WindowSums_Lifted 

func WindowSums_Lifted(s beam.Scope)

Types

type RangeEstimator 

type RangeEstimator struct {
	// contains filtered or unexported fields
}

RangeEstimator implements the offsetrange.RangeEndEstimator interface. It provides the estimated end for a restriction.

func (*RangeEstimator) Estimate 

func (r *RangeEstimator) Estimate() int64

Estimate returns the estimated end.

func (*RangeEstimator) SetEstimate 

func (r *RangeEstimator) SetEstimate(estimate int64)

SetEstimate sets the estimated end.

type TruncateFn 

type TruncateFn struct {
	Estimator RangeEstimator
}

TruncateFn is an SDF.

func (*TruncateFn) CreateInitialRestriction 

func (fn *TruncateFn) CreateInitialRestriction(_ []byte) offsetrange.Restriction

CreateInitialRestriction creates an initial restriction

func (*TruncateFn) CreateTracker 

func (fn *TruncateFn) CreateTracker(rest offsetrange.Restriction) *sdf.LockRTracker

CreateTracker wraps the given restriction into a LockRTracker type.

func (*TruncateFn) ProcessElement 

func (fn *TruncateFn) ProcessElement(ctx context.Context, rt *sdf.LockRTracker, _ []byte, emit func(int64)) sdf.ProcessContinuation

ProcessElement continually gets the start position of the restriction and emits the element as it is.

func (*TruncateFn) RestrictionSize 

func (fn *TruncateFn) RestrictionSize(_ []byte, rest offsetrange.Restriction) float64

RestrictionSize returns the size of the current restriction

func (*TruncateFn) SplitRestriction 

func (fn *TruncateFn) SplitRestriction(_ []byte, rest offsetrange.Restriction) []offsetrange.Restriction

SplitRestriction is similar to the one used in checkpointing.go test.

func (*TruncateFn) TruncateRestriction 

func (fn *TruncateFn) TruncateRestriction(ctx context.Context, rt *sdf.LockRTracker, _ []byte) offsetrange.Restriction

TruncateRestriction truncates the restriction during drain.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.