worker

Documentation

Index

• func DoGKETask(tsk *task.Task, path etl.DataPath) etl.ProcessingError
• func GetSource(client stiface.Client, uri string) (etl.TestSource, etl.DataPath, int, error)
• func ProcessGKETask(ctx context.Context, path etl.DataPath, tf task.Factory) etl.ProcessingError
• func ProcessTask(fn string) (int, error)
• func ProcessTaskWithClient(client stiface.Client, fn string) (int, error)
• func ProcessTestSource(src etl.TestSource, path etl.DataPath) (int, error)
• type StandardTaskFactory
  • func (tf *StandardTaskFactory) Get(ctx context.Context, dp etl.DataPath) (*task.Task, etl.ProcessingError)

Functions

func DoGKETask

func DoGKETask(tsk *task.Task, path etl.DataPath) etl.ProcessingError

DoGKETask creates task, processes all tests and handle metrics

func GetSource

func GetSource(client stiface.Client, uri string) (etl.TestSource, etl.DataPath, int, error)

GetSource gets the TestSource for the filename. fn is a gs:// GCS uri.

func ProcessGKETask

func ProcessGKETask(ctx context.Context, path etl.DataPath, tf task.Factory) etl.ProcessingError

ProcessGKETask interprets a filename to create a Task, Parser, and Inserter, and processes the file content. Used default BQ Sink, and GCS Source. Returns an http status code and an error if the task did not complete successfully.

func ProcessTask

func ProcessTask(fn string) (int, error)

ProcessTask interprets a filename to create a Task, Parser, and Inserter, and processes the file content. Storage client is implicitly obtained from GetStorageClient. Returns an http status code and an error if the task did not complete successfully. DEPRECATED - should migrate to ProcessGKETask.

func ProcessTaskWithClient

func ProcessTaskWithClient(client stiface.Client, fn string) (int, error)

ProcessTaskWithClient handles processing with an injected client.

func ProcessTestSource

func ProcessTestSource(src etl.TestSource, path etl.DataPath) (int, error)

ProcessTestSource handles processing of all TestSource contents.

Types

type StandardTaskFactory

type StandardTaskFactory struct {
	Sink      factory.SinkFactory
	Source    factory.SourceFactory
	Annotator factory.AnnotatorFactory
}

StandardTaskFactory implements task.Factory

func (*StandardTaskFactory) Get

func (tf *StandardTaskFactory) Get(ctx context.Context, dp etl.DataPath) (*task.Task, etl.ProcessingError)

Get implements task.Factory.Get