cnp

Documentation

Overview

Package cnp includes concurrence and parallel tools

mapreduce core source from go-zero

Index

• Variables
• func MapReduce(gen GenerateFunc, opts ...Option) *mapReduce
• func NotifyDone(writer Writer)
• func Parallel(fns ...func() error) error
• func ParallelVoid(fns ...func())
• type GenerateFunc
• type MapperFunc
• type Option
  • func WithContext(ctx context.Context) Option
  • func WithWorkers(workers int) Option
• type ReducerFunc
• type Writer

Variables

var (
	// ErrCancelWithNil is an error that mapreduce was cancelled with nil.
	ErrCancelWithNil = errors.New("mapreduce cancelled with nil")
	// ErrReduceNoOutput is an error that reduce did not output a value.
	ErrReduceNoOutput = errors.New("reduce not writing value")
)

Functions

func MapReduce

func MapReduce(gen GenerateFunc, opts ...Option) *mapReduce

MapReduce runs mapper and reducer, is the entry method of mapreduce

func NotifyDone

func NotifyDone(writer Writer)

NotifyDone notify the process,reduce has done. if you reduce function do not output value ,you must call this method

func Parallel

func Parallel(fns ...func() error) error

Parallel runs fns parallel, cancelled on any error

func ParallelVoid

func ParallelVoid(fns ...func())

ParallelVoid runs void functions parallel

Types

type GenerateFunc

type GenerateFunc func(source chan<- any)

GenerateFunc is used to let callers send elements into source.

type MapperFunc

type MapperFunc func(item any, writer Writer, cancel func(error))

MapperFunc is used to do element processing and write the output to writer, use cancel func to cancel the processing.

type Option

type Option func(opts *mapReduceOptions)

Option defines the method to customize the mapreduce.

func WithContext

func WithContext(ctx context.Context) Option

WithContext customizes a mapreduce processing accepts a given ctx.

func WithWorkers

func WithWorkers(workers int) Option

WithWorkers customizes a mapreduce processing with given workers.

type ReducerFunc

type ReducerFunc func(pipe <-chan any, writer Writer, cancel func(error))

ReducerFunc is used to reduce all the mapping output and write to writer, use cancel func to cancel the processing.

type Writer

type Writer interface {
	Write(v any)
}

Writer interface wraps Write method.