mapreduce

Documentation

Index

• Constants
• func Map(job Job, jobPath string, m materializeInfo, host string, shard, modulos uint64)
• func Materialize(in_repo, branch, commit, out_repo, jobDir string, shard, modulos uint64) error
• func PrepJob(job Job, jobPath string, m materializeInfo) error
• func Reduce(job Job, jobPath string, m materializeInfo, host string, shard, modulos uint64)
• func WaitJob(in_repo, commit, job string)
• type Job

Constants

const (
	ProtoPfs = iota
	ProtoS3  = iota
)

Functions

func Map

func Map(job Job, jobPath string, m materializeInfo, host string, shard, modulos uint64)

func Materialize

func Materialize(in_repo, branch, commit, out_repo, jobDir string, shard, modulos uint64) error

Materialize parses the jobs found in `in_repo`/`commit`/`jobDir` runs them with `in_repo/commit` as input, outputs the results to `out_repo`/`branch` and commits them as `out_repo`/`commit`

func PrepJob

func PrepJob(job Job, jobPath string, m materializeInfo) error

func Reduce

func Reduce(job Job, jobPath string, m materializeInfo, host string, shard, modulos uint64)

func WaitJob

func WaitJob(in_repo, commit, job string)

Types

type Job

type Job struct {
	Type     string   `json:"type"`
	Input    string   `json:"input"`
	Image    string   `json:"image"`
	Command  []string `json:"command"`
	Limit    int      `json:"limit"`
	Parallel int      `json:"parallel"`
}