reduce

package
v1.3.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 3, 2024 License: Apache-2.0 Imports: 22 Imported by: 0

Documentation

Overview

Package reduce reads messages from isb and is responsible for the first part of reduce subsystem. It is responsible for feeding in the data to the second part of the reduce subsystem called processAndForward and processes the read data from the ISB, writes to the outbound channel called PBQ so the message can be asynchronously processed by `processAndForward`, and then closes the partition if possible based on watermark progression. To write to the outbound channel (PBQ), it has to partition the message first. Hence, also contains partitioning logic. Partitioner identifies a set of elements with a common key and time, and buckets them in to a common window. A partition is uniquely identified using a tuple {window, key}. Type of window does not matter. Partitioner is responsible for managing the persistence and processing of each partition. It uses PBQ for durable persistence of elements that belong to a partition and orchestrates the processing of elements using processAndForward function. Partitioner tracks active partitions, closes the partitions based on watermark progression and co-ordinates the materialization and forwarding the results to the next vertex in the pipeline.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type DataForward

type DataForward struct {
	// contains filtered or unexported fields
}

DataForward is responsible for reading and forwarding the message from ISB to PBQ.

func NewDataForward

func NewDataForward(ctx context.Context,
	vertexInstance *dfv1.VertexInstance,
	fromBuffer isb.BufferReader,
	toBuffers map[string][]isb.BufferWriter,
	pbqManager *pbq.Manager,
	storeManager wal.Manager,
	whereToDecider forwarder.ToWhichStepDecider,
	fw fetch.Fetcher,
	watermarkPublishers map[string]publish.Publisher,
	windowingStrategy window.TimedWindower,
	idleManager wmb.IdleManager,
	of *pnf.ProcessAndForward,
	opts ...Option) (*DataForward, error)

NewDataForward creates a new DataForward

func (*DataForward) ReplayPersistedMessages added in v0.8.1

func (df *DataForward) ReplayPersistedMessages(ctx context.Context) error

ReplayPersistedMessages replays persisted messages, because during boot up, it has to replay the WAL from the store, before it can start reading from ISB. ReplayPersistedMessages will return only after the replay has been completed.

func (*DataForward) ShutDown added in v0.8.1

func (df *DataForward) ShutDown(ctx context.Context)

ShutDown shutdowns the read-loop.

func (*DataForward) Start

func (df *DataForward) Start()

Start starts reading messages from ISG

type Option

type Option func(*Options) error

func WithAllowedLateness added in v0.8.0

func WithAllowedLateness(t time.Duration) Option

WithAllowedLateness sets allowedLateness

func WithReadBatchSize

func WithReadBatchSize(f int64) Option

WithReadBatchSize sets the read batch size

type Options

type Options struct {
	// contains filtered or unexported fields
}

Options for forwarding the message

func DefaultOptions

func DefaultOptions() *Options

Directories

Path Synopsis
pbq
partition
Package partition is a tuple containing (start, end) time and an optional slot.
Package partition is a tuple containing (start, end) time and an optional slot.
wal
wal/aligned/fs
Package fs implements write-ahead-log on FileSystem.
Package fs implements write-ahead-log on FileSystem.
wal/unaligned/fs
Package fs implements write-ahead-log on FileSystem.
Package fs implements write-ahead-log on FileSystem.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL