mapreduce

Documentation

Index

• func Map(job Job, jobName string, m materializeInfo, shard, modulos uint64)
• func Materialize(in_repo, branch, commit, outRepo, jobDir string, shard, modulos uint64) error
• func PrepJob(job Job, jobName string, m materializeInfo) error
• func Reduce(job Job, jobName string, m materializeInfo, shard, modulos uint64)
• func WaitJob(outRepo, branch, commit, job string) error
• type Job
• type Proto

Functions

func Map

func Map(job Job, jobName string, m materializeInfo, shard, modulos uint64)

func Materialize

func Materialize(in_repo, branch, commit, outRepo, jobDir string, shard, modulos uint64) error

Materialize parses the jobs found in `in_repo`/`commit`/`jobDir` runs them with `in_repo/commit` as input, outputs the results to `outRepo`/`branch` and commits them as `outRepo`/`commit`

func PrepJob

func PrepJob(job Job, jobName string, m materializeInfo) error

func Reduce

func Reduce(job Job, jobName string, m materializeInfo, shard, modulos uint64)

func WaitJob

func WaitJob(outRepo, branch, commit, job string) error

Types

type Job

type Job struct {
	Type      string   `json:"type"`
	Input     string   `json:"input"`
	Image     string   `json:"image"`
	Cmd       []string `json:"command"`
	Limit     int      `json:"limit"`
	Parallel  int      `json:"parallel"`
	TimeOut   int      `json:"timeout"`
	CpuShares int      `json:"cpu-shares"`
	Memory    int      `json:"memory"`
}

type Proto

type Proto int

const (
	ProtoPfs Proto = iota
	ProtoS3  Proto = iota
)