reduce

Documentation

Overview

Package reduce provides a simple interface for transforming a stream of inputs. Currently, other than utility functions/interfaces, the only transformation available is Sort.

Index

• func KeyValueSorter() (disksort.Interface, error)
• type ChannelInput
  • func (c ChannelInput) Next() (interface{}, error)
• type ChannelInputValue
• type ChannelSplitInput
  • func (c ChannelSplitInput) NextSplit() (Input, error)
• type ChannelSplitInputValue
• type CombinedReducers
  • func (c *CombinedReducers) End(ctx context.Context) error
  • func (c *CombinedReducers) Reduce(ctx context.Context, rio IO) error
  • func (c *CombinedReducers) Start(ctx context.Context) error
• type Func
  • func (r Func) End(_ context.Context) error
  • func (r Func) Reduce(ctx context.Context, io IO) error
  • func (r Func) Start(_ context.Context) error
• type IO
• type IOStruct
• type InFunc
  • func (f InFunc) Next(ctx context.Context) (interface{}, error)
• type Input
• type OutFunc
  • func (o OutFunc) Emit(ctx context.Context, i interface{}) error
• type Output
• type Reducer
• type SplitInput
  • func Sort(ctx context.Context, splits SplitInput, r Reducer) (SplitInput, error)
  • func SplitSortedKeyValues(sorter disksort.Interface) (SplitInput, error)

Functions

func KeyValueSorter

func KeyValueSorter() (disksort.Interface, error)

KeyValueSorter returns a disksort for arbitrary *ipb.SortedKeyValues.

Types

type ChannelInput

type ChannelInput <-chan ChannelInputValue

ChannelInput implements the Input interface, yielding each channel value when Next is called.

func (ChannelInput) Next

func (c ChannelInput) Next() (interface{}, error)

Next implements the Input interface.

type ChannelInputValue

type ChannelInputValue struct {
	Value interface{}
	Err   error
}

ChannelInputValue represents the return value of Input#Next.

type ChannelSplitInput

type ChannelSplitInput <-chan ChannelSplitInputValue

ChannelSplitInput implements the SplitInput interface over a channel of ReduceInputs to return on each call to NextSplit.

func (ChannelSplitInput) NextSplit

func (c ChannelSplitInput) NextSplit() (Input, error)

NextSplit implements the SplitInput interface.

type ChannelSplitInputValue

type ChannelSplitInputValue struct {
	Input Input
	Err   error
}

ChannelSplitInputValue represents the return values of SplitInput#NextSplit.

type CombinedReducers

type CombinedReducers struct {
	Reducers []Reducer
}

CombinedReducers allows multiple Reducers to consume the same ReducerInput and output to the same ReducerOutput.

func (*CombinedReducers) End

func (c *CombinedReducers) End(ctx context.Context) error

End implements part of the Reducer interface.

func (*CombinedReducers) Reduce

func (c *CombinedReducers) Reduce(ctx context.Context, rio IO) error

Reduce implements part of the Reducer interface.

func (*CombinedReducers) Start

func (c *CombinedReducers) Start(ctx context.Context) error

Start implements part of the Reducer interface.

type Func

type Func func(context.Context, IO) error

A Func implements the Reducer interface with no-op Start/End methods.

func (Func) End

func (r Func) End(_ context.Context) error

End implements part of the Reducer interface.

func (Func) Reduce

func (r Func) Reduce(ctx context.Context, io IO) error

Reduce implements part of the Reducer interface.

func (Func) Start

func (r Func) Start(_ context.Context) error

Start implements part of the Reducer interface.

type IO

type IO interface {
	Input
	Output
}

IO composes Input and Output.

type IOStruct

type IOStruct struct {
	Input
	Output
}

IOStruct reifies the IO interface.

type InFunc

type InFunc func(context.Context) (interface{}, error)

An InFunc implements the Input interface.

func (InFunc) Next

func (f InFunc) Next(ctx context.Context) (interface{}, error)

Next implements the Input interface.

type Input

type Input interface {
	Next() (interface{}, error)
}

A Input yields input to a Reducer.

type OutFunc

type OutFunc func(context.Context, interface{}) error

An OutFunc implements the Output interface.

func (OutFunc) Emit

func (o OutFunc) Emit(ctx context.Context, i interface{}) error

Emit implements the Output interface.

type Output

type Output interface {
	Emit(context.Context, interface{}) error
}

A Output allows a Reducer to emit values.

type Reducer

type Reducer interface {
	Start(context.Context) error
	Reduce(context.Context, IO) error
	End(context.Context) error
}

A Reducer transforms one stream of values into another stream of values. Reduce may be called multiple times (once per input shard). Start and End will be called once before any call to Reduce and once after every call to Reduce, respectively.

type SplitInput

type SplitInput interface {
	NextSplit() (Input, error)
}

SplitInput represents an input that is sharded.

func Sort

func Sort(ctx context.Context, splits SplitInput, r Reducer) (SplitInput, error)

Sort applies r to each separate input in splits. r should be a Reducer that accepts the same input type that splits contains and MUST output *ipb.SortedKeyValues. The resulting SplitInput will be the set of outputs from r, split on groups sharing the same SortedKeyValue.Key and sorted within a group by SortedKeyValue.SortKey (see SplitSortedKeyValues). The Reducer's Start method will be called once before any call to Reduce and the Reducer's End method will be called once after the final Reduce call is completed.

Sort will return on the first error it encounters. This may cause some of the input to not be read and Start/Reduce/End may not be called depending on when the error occurs.

func SplitSortedKeyValues

func SplitSortedKeyValues(sorter disksort.Interface) (SplitInput, error)

SplitSortedKeyValues constructs a SplitInput that returns a Input for each set of *ipb.SortedKeyValues with the same key.