mapreduce

Documentation

Overview

common_rpc.go defined all the parameters used in RPC between master and workers

Index

• Constants
• func RemoveContents(dir string) error
• func RunMaster(task *Task, hostname string)
• func RunSequential(task *Task)
• func RunWorker(task *Task, hostname string, masterHostname string, nOps int)
• type KeyValue
• type MapFunc
• type Master
  • func (master *Master) Register(args *RegisterArgs, reply *RegisterReply) error
• type Operation
• type ReduceFunc
• type RegisterArgs
• type RegisterReply
• type RemoteWorker
• type RunArgs
• type ShuffleFunc
• type Task
• type Worker
  • func (worker *Worker) Done(_ *struct{}, _ *struct{}) error
  • func (worker *Worker) RunMap(args *RunArgs, _ *struct{}) error
  • func (worker *Worker) RunReduce(args *RunArgs, _ *struct{}) error

Constants

const (
	REDUCE_PATH = "reduce/"
	RESULT_PATH = "result/"

	OPEN_FILE_MAX_RETRY = 3
)

const (
	IDLE_WORKER_BUFFER     = 100
	RETRY_OPERATION_BUFFER = 100
)

const (
	WORKER_IDLE    workerStatus = "idle"
	WORKER_RUNNING workerStatus = "running"
)

Functions

func RemoveContents

func RemoveContents(dir string) error

func RunMaster

func RunMaster(task *Task, hostname string)

RunMaster will start a master node on the map reduce operations. In the distributed model, a Master should serve multiple workers and distribute the operations to be executed in order to complete the task.

• task: the Task object that contains the mapreduce operation.
• hostname: the tcp/ip address on which it will listen for connections.

func RunSequential

func RunSequential(task *Task)

RunSequential will ensure that map and reduce function runs in a single-core linearly. The Task is passed from the calling package and should contains the definitions for all the required functions and parameters. Notice that this implementation will store data locally. In the distributed version of mapreduce it's common to store the data in the same worker that computed it and just pass a reference to reduce jobs so they can go grab it.

func RunWorker

func RunWorker(task *Task, hostname string, masterHostname string, nOps int)

RunWorker will run a instance of a worker. It'll initialize and then try to register with master. Induced failures: -> nOps = number of operations to run before failure (0 = no failure)

Types

type KeyValue

type KeyValue struct {
	Key   string
	Value string
}

KeyValue is the type used to hold elements of maps and reduces results.

type MapFunc

type MapFunc func([]byte) []KeyValue

type Master

type Master struct {
	// contains filtered or unexported fields
}

func (*Master) Register

func (master *Master) Register(args *RegisterArgs, reply *RegisterReply) error

RPC - Register Procedure that will be called by workers to register within this master.

type Operation

type Operation struct {
	// contains filtered or unexported fields
}

type ReduceFunc

type ReduceFunc func([]KeyValue) []KeyValue

type RegisterArgs

type RegisterArgs struct {
	WorkerHostname string
}

type RegisterReply

type RegisterReply struct {
	WorkerId   int
	ReduceJobs int
}

type RemoteWorker

type RemoteWorker struct {
	// contains filtered or unexported fields
}

type RunArgs

type RunArgs struct {
	Id       int
	FilePath string
}

type ShuffleFunc

type ShuffleFunc func(*Task, string) int

type Task

type Task struct {
	// MapReduce functions
	Map     MapFunc
	Shuffle ShuffleFunc
	Reduce  ReduceFunc

	// Jobs
	NumReduceJobs int
	NumMapFiles   int

	// Channels for data
	InputChan  chan []byte
	OutputChan chan []KeyValue

	// Channels for filepaths
	InputFilePathChan  chan string
	OutputFilePathChan chan string
}

Task is the exposed struct of the Framework that the calling code should initialize with the specific implementation of the operation.

type Worker

type Worker struct {
	// contains filtered or unexported fields
}

func (*Worker) Done

func (worker *Worker) Done(_ *struct{}, _ *struct{}) error

RPC - Done Will be called by Master when the task is done.

func (*Worker) RunMap

func (worker *Worker) RunMap(args *RunArgs, _ *struct{}) error

RPC - RunMap Run the map operation defined in the task and return when it's done.

func (*Worker) RunReduce

func (worker *Worker) RunReduce(args *RunArgs, _ *struct{}) error

RPC - RunMap Run the reduce operation defined in the task and return when it's done.