task

Documentation

Overview

Package task provides the tracking of state for a single task pushed by the external task queue.

The Task type ... TODO(dev) Improve comments and header before merging to dev.

Index

• Constants
• type Factory
• type Task
  • func NewTask(filename string, src etl.TestSource, prsr etl.Parser, closer io.Closer) *Task
  • func (tt *Task) Close()
  • func (tt *Task) ProcessAllTests(failfast bool) (int, error)
  • func (tt *Task) SetMaxFileSize(max int64)

Constants

const DefaultMaxFileSize = 200 * 1024 * 1024

DefaultMaxFileSize is the default value for the file size limit for calls to NextTest. Larger than this risks an OOM if there are multiple large files at on multiple tasks. This can be overridden with SetMaxFileSize()

Types

type Factory

type Factory interface {
	Get(context.Context, etl.DataPath) (*Task, etl.ProcessingError)
}

Factory provides Get() which always returns a new, complete Task. TODO for the defs that stay in factory package, remove ...Factory.

type Task

type Task struct {
	// TestSource and Parser are both embedded, so their interfaces are delegated
	// to the component structs.
	etl.TestSource // Source from which to read tests.
	etl.Parser     // Parser to parse the tests.
	// contains filtered or unexported fields
}

Task contains the state required to process a single task tar file. TODO(dev) Add unit tests for meta data.

func NewTask

func NewTask(filename string, src etl.TestSource, prsr etl.Parser, closer io.Closer) *Task

NewTask constructs a task, injecting the source and the parser.

func (*Task) Close

func (tt *Task) Close()

Close closes the source and sink.

func (*Task) ProcessAllTests

func (tt *Task) ProcessAllTests(failfast bool) (int, error)

ProcessAllTests loops through all the tests in a tar file, calls the injected parser to parse them, and inserts them into bigquery. Returns the number of files processed. TODO pass in the datatype label.

func (*Task) SetMaxFileSize

func (tt *Task) SetMaxFileSize(max int64)

SetMaxFileSize overrides the default maxFileSize.