mapreduce

Documentation

Overview

Package mapreduce of the Tideland Go Library implements the Map/Reduce algorithm for the processing and aggregation mass data.

A type implementing the MapReducer interface has to be implemented and passed to the MapReduce() function. The type is responsible for the input, the mapping, the reducing and the consuming while the package provides the runtime environment for it.

Index

• func MapReduce(mr MapReducer) error
• type KeyValue
• type KeyValueChan
  • func (c KeyValueChan) Close()
• type MapReducer

Functions

func MapReduce

func MapReduce(mr MapReducer) error

MapReduce applies a map and a reduce function to keys and values in parallel.

Types

type KeyValue

type KeyValue interface {
	// Key returns the key for the mapping.
	Key() string

	// Value returns the payload value for processing.
	Value() interface{}
}

KeyValue is a pair of a string key and any data as value.

type KeyValueChan

type KeyValueChan chan KeyValue

KeyValueChan is a channel for the transfer of key/value pairs.

func (KeyValueChan) Close

func (c KeyValueChan) Close()

Close the channel for key/value pairs.

type MapReducer

type MapReducer interface {
	// Input has to return the input channel for the
	// date to process.
	Input() KeyValueChan

	// Map maps a key/value pair to another one and emits it.
	Map(in KeyValue, emit KeyValueChan)

	// Reduce reduces the values delivered via the input
	// channel to the emit channel.
	Reduce(in, emit KeyValueChan)

	// Consume allows the MapReducer to consume the
	// processed data.
	Consume(in KeyValueChan) error
}

MapReducer has to be implemented to control the map/reducing.