mapreduce

Documentation

Index

• Constants
• type KeyValue
• type Logger
  • func NewLogger(prefix string, debug bool) *Logger
  • func (l *Logger) Debug(format string, v ...interface{})
  • func (l *Logger) Error(format string, v ...interface{})
  • func (l *Logger) Fatal(format string, v ...interface{})
  • func (l *Logger) Info(format string, v ...interface{})
• type MapReduce
• type Master
  • func NewMaster(host string, httpPort int, rpcPort int, workDir string, debug bool) (*Master, error)
  • func (m *Master) Start() error
  • func (m *Master) Stop() error
  • func (m *Master) Submit(job *service.MrJob, stream service.Master_SubmitServer) error
• type Wrk
  • func NewWorker(id, host string, rpcPort, httpPort int, workDir string, debug bool) (*Wrk, error)
  • func (w *Wrk) Map(job *service.MrJob, server service.MapReduce_MapServer) error
  • func (w *Wrk) Reduce(job *service.MrJob, server service.MapReduce_ReduceServer) error
  • func (w *Wrk) Start() error
  • func (w *Wrk) Stop() error

Constants

const (
	// MasterServiceName name of the master service in Consul.
	MasterServiceName = "master"

	// WorkerServiceName name of the worker service in Consul.
	WorkerServiceName = "worker"

	// GrpcConsulTag tag set for the worker to notify the clients that grpc API should be used.
	GrpcConsulTag = "grpc"

	// Separator separates key and value in intermediate files.
	Separator = ":::"

	// RetriesLimit number of retries of.
	RetriesLimit = 3

	// MapPhase name of the map phase.
	MapPhase = "map"

	// ReducePhase name of the reduce phase.
	ReducePhase = "reduce"
)

Types

type KeyValue

type KeyValue struct {
	Key   string
	Value string
}

KeyValue tuple with key and value.

type Logger

type Logger struct {
	// contains filtered or unexported fields
}

Logger hold bacic info about logger configuration.

func NewLogger

func NewLogger(prefix string, debug bool) *Logger

NewLogger returns new Logger instance.

func (*Logger) Debug

func (l *Logger) Debug(format string, v ...interface{})

Debug logs the formatted message with level DEBUG - only if debug flag is set to true.

func (*Logger) Error

func (l *Logger) Error(format string, v ...interface{})

Error logs the formatted message with level ERROR.

func (*Logger) Fatal

func (l *Logger) Fatal(format string, v ...interface{})

Fatal logs the formatted message with level FATAL.

func (*Logger) Info

func (l *Logger) Info(format string, v ...interface{})

Info logs the formatted message with level INFO.

type MapReduce

type MapReduce interface {
	Map(string, string) []KeyValue
	Reduce(string, []string) string
}

MapReduce it's interface for any job in gomr framework. Every gomr job should implement this interface.

type Master

type Master struct {
	// contains filtered or unexported fields
}

Master holds information about the gomr Master service.

func NewMaster

func NewMaster(host string, httpPort int, rpcPort int, workDir string, debug bool) (*Master, error)

NewMaster returns new instance of gomr Master service.

func (*Master) Start

func (m *Master) Start() error

Start starts the gomr master service - creates the health check server, register master service in consul and tries to connect to available warokers. Func waits until it gets signal{Kill, Interrupt} or Stop func is called.

func (*Master) Stop

func (m *Master) Stop() error

Stop stops the master service.

func (*Master) Submit

func (m *Master) Submit(job *service.MrJob, stream service.Master_SubmitServer) error

Submit exposes the submit for the gRPC gomr client. It submits the job, prepares inputs for map phase, coordinates the map and reduce phase and merge the outputs from the reduce phase. It blocks until the job is done.

type Wrk

type Wrk struct {
	// contains filtered or unexported fields
}

Wrk holds information about the gomr Worker service.

func NewWorker

func NewWorker(id, host string, rpcPort, httpPort int, workDir string, debug bool) (*Wrk, error)

NewWorker returns new instance of gomr worker.

func (*Wrk) Map

func (w *Wrk) Map(job *service.MrJob, server service.MapReduce_MapServer) error

Map starts map phase and returns the status of the tasks as long as its ongoing.

func (*Wrk) Reduce

func (w *Wrk) Reduce(job *service.MrJob, server service.MapReduce_ReduceServer) error

Reduce starts reduce phase and returns the status of the tasks as long as its ongoing.

func (*Wrk) Start

func (w *Wrk) Start() error

Start starts the gomr worker - creates the health check server and register workers service in consul. Func waits until it gets signal{Kill, Interrupt} or Stop func is called.

func (*Wrk) Stop

func (w *Wrk) Stop() error

Stop stops the worker.