Documentation ¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type CallReply ¶
type CallReply struct {
// contains filtered or unexported fields
}
schedule() starts and waits for all tasks in the given phase (mapPhase or reducePhase). the mapFiles argument holds the names of the files that are the inputs to the map phase, one per map task. nReduce is the number of reduce tasks. the registerChan argument yields a stream of registered workers; each item is the worker's RPC address, suitable for passing to call(). registerChan will yield all existing registered workers (if any) and new ones as they register.
type DoTaskArgs ¶
type DoTaskArgs struct { JobName string File string // only for map, the input file Phase jobPhase // are we in mapPhase or reducePhase? TaskNumber int // this task's index in the current phase // NumOtherPhase is the total number of tasks in other phase; mappers // need this to compute the number of output bins, and reducers needs // this to know how many input files to collect. NumOtherPhase int }
DoTaskArgs holds the arguments that are passed to a worker when a job is scheduled on it.
type KeyValue ¶
KeyValue is a type used to hold the key/value pairs passed to the map and reduce functions.
type Master ¶
Master holds all the state that the master needs to keep track of.
func Distributed ¶
Distributed schedules map and reduce tasks on workers that register with the master over RPC.
func Sequential ¶
func Sequential(jobName string, files []string, nreduce int, mapF func(string, string) []KeyValue, reduceF func(string, []string) string, ) (mr *Master)
Sequential runs map and reduce tasks sequentially, waiting for each task to complete before running the next.
func (*Master) CleanupFiles ¶
func (mr *Master) CleanupFiles()
CleanupFiles removes all intermediate files produced by running mapreduce.
func (*Master) Register ¶
func (mr *Master) Register(args *RegisterArgs, _ *struct{}) error
Register is an RPC method that is called by workers after they have started up to report that they are ready to receive tasks.
type Parallelism ¶
type Parallelism struct {
// contains filtered or unexported fields
}
track whether workers executed in parallel.
type RegisterArgs ¶
type RegisterArgs struct {
Worker string // the worker's UNIX-domain socket name, i.e. its RPC address
}
RegisterArgs is the argument passed when a worker registers with the master.
type ShutdownReply ¶
type ShutdownReply struct {
Ntasks int
}
ShutdownReply is the response to a WorkerShutdown. It holds the number of tasks this worker has processed since it was started.
type Worker ¶
type Worker struct { sync.Mutex Map func(string, string) []KeyValue Reduce func(string, []string) string // contains filtered or unexported fields }
Worker holds the state for a server waiting for DoTask or Shutdown RPCs
func (*Worker) DoTask ¶
func (wk *Worker) DoTask(arg *DoTaskArgs, _ *struct{}) error
DoTask is called by the master when a new task is being scheduled on this worker.
func (*Worker) Shutdown ¶
func (wk *Worker) Shutdown(_ *struct{}, res *ShutdownReply) error
Shutdown is called by the master when all work has been completed. We should respond with the number of tasks we have processed.