window

Documentation

Overview

Package window implements windowing constructs. In the world of data processing on an unbounded stream, Windowing is a concept of grouping data using temporal boundaries. We use event-time to discover temporal boundaries on an unbounded, infinite stream and Watermark to ensure the datasets within the boundaries are complete. A reduce function can be applied on this group of data.

Windows are of different types, quite popular ones are Fixed windows and Sliding windows. Sessions are managed via little less popular windowing strategy called Session windows. Windowing is implemented as a two stage process,

• Assign windows - assign the event to a window
• Merge windows - group all the events that below to the same window

The two stage approach is required because assignment of windows could happen as elements are streaming in, but merging could happen before the data materialization happens. This is important esp. when we handle session windows where a new event can change the end time of the window.

For simplicity, we will be truncating the windows' boundaries to the nearest time unit (say, 1 minute windows will be truncated to 0th second). Truncating window time to the nearest boundary will help us do mapping with constant time without affecting the correctness, except for the very first materialization of result (e.g., we started at 9:00.11 and the result will be materialized at 9:01.00 and not at 9:01:11).

Windows may be either aligned (e.g., Fixed, Sliding), i.e. applied across all the data for the window of time in question, or unaligned, (e.g., Session) i.e. applied across only specific subsets of the data (e.g. per key) for the given window of time.

Index

• type AlignedKeyedWindower
• type Windower

Types

type AlignedKeyedWindower

type AlignedKeyedWindower interface {
	// StartTime returns the start time of the window
	StartTime() time.Time
	// EndTime returns the end time of the window
	EndTime() time.Time
	// AddKey adds a key to the window
	AddKey(string)
	// Partitions returns an array of partition ids
	Partitions() []partition.ID
	// Keys returns an array of keys
	Keys() []string
}

AlignedKeyedWindower represents a bounded window (i.e., it will have a definite start and end time), and the keys that also fall into the same window.

type Windower

type Windower interface {
	// AssignWindow assigns the event to the window based on give window configuration.
	AssignWindow(eventTime time.Time) []AlignedKeyedWindower
	// InsertIfNotPresent inserts window to the list of active windows if not present
	// if present it will return the window
	InsertIfNotPresent(aw AlignedKeyedWindower) (AlignedKeyedWindower, bool)
	// RemoveWindows returns list of window(s) that can be closed
	RemoveWindows(time time.Time) []AlignedKeyedWindower
}

Windower manages AlignedKeyedWindower Will be implemented by each of the windowing strategies.