mr

Documentation

Index

• func TaskCompleted(task *Task)
• func Worker(mapf func(string, string) []KeyValue, reducef func(string, []string) string)
• type ByKey
  • func (a ByKey) Len() int
  • func (a ByKey) Less(i, j int) bool
  • func (a ByKey) Swap(i, j int)
• type ExampleArgs
• type ExampleReply
• type KeyValue
• type Master
  • func MakeMaster(files []string, nReduce int) *Master
  • func (m *Master) AssignTask(args *ExampleArgs, reply *Task) error
  • func (m *Master) Done() bool
  • func (m *Master) TaskCompleted(task *Task, reply *ExampleReply) error
• type MasterTask
• type MasterTaskStatus
• type State
• type Task

Functions

func TaskCompleted

func TaskCompleted(task *Task)

Part2-6：worker任务完成后通知master

func Worker

func Worker(mapf func(string, string) []KeyValue, reducef func(string, []string) string)

Part2-3：启动worker main/mrworker.go calls this function.

Types

type ByKey

type ByKey []KeyValue

for sorting by key.

func (ByKey) Len

func (a ByKey) Len() int

for sorting by key.

func (ByKey) Less

func (a ByKey) Less(i, j int) bool

func (ByKey) Swap

func (a ByKey) Swap(i, j int)

type ExampleArgs

type ExampleArgs struct {
	X int
}

type ExampleReply

type ExampleReply struct {
	Y int
}

type KeyValue

type KeyValue struct {
	Key   string
	Value string
}

Map functions return a slice of KeyValue.

type Master

type Master struct {
	TaskQueue     chan *Task          // 等待执行的task
	TaskMeta      map[int]*MasterTask // 当前所有task的信息
	MasterPhase   State               // Master的阶段
	NReduce       int
	InputFiles    []string
	Intermediates [][]string // Map任务产生的R个中间文件的信息
}

Part1-3：此外Master存储Map任务产生的R个中间文件的信息

func MakeMaster

func MakeMaster(files []string, nReduce int) *Master

Part 2-1：启动master create a Master. main/mrmaster.go calls this function. nReduce is the number of reduce tasks to use.

func (*Master) AssignTask

func (m *Master) AssignTask(args *ExampleArgs, reply *Task) error

Part2-2：master监听worker RPC调用，分配任务 master等待worker调用

func (*Master) Done

func (m *Master) Done() bool

Part2-9：master确认所有ReduceTask都已经完成，转入Exit阶段，终止所有master和worker goroutine main/mrmaster.go calls Exit() periodically to find out if the entire job has finished.

func (*Master) TaskCompleted

func (m *Master) TaskCompleted(task *Task, reply *ExampleReply) error

Part2-7：master收到完成后的Task

type MasterTask

type MasterTask struct {
	TaskStatus    MasterTaskStatus
	StartTime     time.Time
	TaskReference *Task
}

Part1-2： Master存储这些Task的信息。与论文不同的是，这里我并没有保留worked的ID，因此master不会主动向worker发送心跳检测

type MasterTaskStatus

type MasterTaskStatus int

Part1-1：每个(Map或者Reduce)Task有分为idle, in-progress, completed 三种状态

const (
	Idle MasterTaskStatus = iota
	InProgress
	Completed
)

type State

type State int

Part1-5：此外我将task和master的状态合并成一个State。task和master的状态应该一致。如果在Reduce阶段收到了迟来MapTask结果，应该直接丢弃。

const (
	Map State = iota
	Reduce
	Exit
	Wait
)

type Task

type Task struct {
	Input         string
	TaskState     State
	NReducer      int
	TaskNumber    int
	Intermediates []string
	Output        string
}

Part1-4：Map和Reduce的Task应该负责不同的事情，但是在实现代码的过程中发现同一个Task结构完全可以兼顾两个阶段的任务