pipeline

Documentation

Overview

Package pipeline provides abstraction of pipeline and stages. It is basic tool to convert TAR/TAR GZ file into TFRecord file.

Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved.

Index

• type DefaultPipeline
  • func NewPipeline() *DefaultPipeline
  • func (p *DefaultPipeline) Do() error
  • func (p *DefaultPipeline) FilterEmptySamples() *DefaultPipeline
  • func (p *DefaultPipeline) FilterEmptyTFExamples() *DefaultPipeline
  • func (p *DefaultPipeline) FromTar(input io.Reader) *DefaultPipeline
  • func (p *DefaultPipeline) FromTarGz(input io.Reader) *DefaultPipeline
  • func (p *DefaultPipeline) SampleToTFExample(m ...core.TypesMap) *DefaultPipeline
  • func (p *DefaultPipeline) ToTFRecord(w io.Writer, numWorkers ...int) *DefaultPipeline
  • func (p *DefaultPipeline) TransformSamples(tfs ...transform.SampleTransformation) *DefaultPipeline
  • func (p *DefaultPipeline) TransformTFExamples(tfs ...transform.TFExampleTransformation) *DefaultPipeline
  • func (p *DefaultPipeline) WithSample2TFExampleStage(stage Sample2TFExampleStage) *DefaultPipeline
  • func (p *DefaultPipeline) WithSamplesStage(stage SamplesStage) *DefaultPipeline
  • func (p *DefaultPipeline) WithTFExamplesStage(stage TFExamplesStage) *DefaultPipeline
  • func (p *DefaultPipeline) WithTFRecordStage(stage TFRecordStage) *DefaultPipeline
  • func (p *DefaultPipeline) WithTarStage(stage TarStage) *DefaultPipeline
• type Sample2TFExampleStage
• type SamplesStage
• type TFExamplesStage
• type TFRecordStage
• type TarStage

Types

type DefaultPipeline

type DefaultPipeline struct {
	// contains filtered or unexported fields
}

DefaultPipeline represents TAR file to TFRecord file conversion with an intermediate transformations on core.Sample and core.TFExample

func NewPipeline

func NewPipeline() *DefaultPipeline

func (*DefaultPipeline) Do

func (p *DefaultPipeline) Do() error

Do executes pipeline based on specified stages.

func (*DefaultPipeline) FilterEmptySamples

func (p *DefaultPipeline) FilterEmptySamples() *DefaultPipeline

FilterEmptySamples adds filtering of empty core.Samples as pipeline's SamplesStage.

func (*DefaultPipeline) FilterEmptyTFExamples

func (p *DefaultPipeline) FilterEmptyTFExamples() *DefaultPipeline

FilterEmptyTFExamples adds filtering of empty core.TFExamples as pipeline's TFExamplesStage.

func (*DefaultPipeline) FromTar

func (p *DefaultPipeline) FromTar(input io.Reader) *DefaultPipeline

FromTar adds reading core.Samples from input as input was a TAR file.

func (*DefaultPipeline) FromTarGz

func (p *DefaultPipeline) FromTarGz(input io.Reader) *DefaultPipeline

FromTarGz adds reading core.Samples from input as input was a TAR GZ file.

func (*DefaultPipeline) SampleToTFExample

func (p *DefaultPipeline) SampleToTFExample(m ...core.TypesMap) *DefaultPipeline

Converts Samples to TFExamples. TypesMap defines what are actual sample types. For each (key, mappedType) pair from TypesMap, TFExample will have feature[key] = value, where value is sample[key] converted into type mappedType. If m is not provided, each entry from value will be converted to BytesList If m provided, but sample has key which is not present in TypesMap, value will be converted to BytesList

func (*DefaultPipeline) ToTFRecord

func (p *DefaultPipeline) ToTFRecord(w io.Writer, numWorkers ...int) *DefaultPipeline

Writers TFExamples to specified writer in TFRecord format If numWorkers provided, all pipeline transformations will be done asynchronously. It assumes that all underlying Readers are async-safe. All default Readers, Transformations, Selections are async-safe.

func (*DefaultPipeline) TransformSamples

func (p *DefaultPipeline) TransformSamples(tfs ...transform.SampleTransformation) *DefaultPipeline

TransformTFExamples adds transforming core.Samples according to tfs as pipeline's SamplesStage. Transformations will be executed in order of appearance in tfs

func (*DefaultPipeline) TransformTFExamples

func (p *DefaultPipeline) TransformTFExamples(tfs ...transform.TFExampleTransformation) *DefaultPipeline

TransformTFExamples adds transforming core.TFExamples according to tfs as pipeline's TFExamplesStage. Transformations will be executed in order of appearance in tfs

func (*DefaultPipeline) WithSample2TFExampleStage

func (p *DefaultPipeline) WithSample2TFExampleStage(stage Sample2TFExampleStage) *DefaultPipeline

WithSample2TFExampleStage defines Sample2TFExampleStage of a pipeline. Overrides previous value.

func (*DefaultPipeline) WithSamplesStage

func (p *DefaultPipeline) WithSamplesStage(stage SamplesStage) *DefaultPipeline

WithSamplesStage defines SamplesStage of a pipeline. If SamplesStage has been already set, the resulting SamplesStage will chain together transformations in order of setting.

func (*DefaultPipeline) WithTFExamplesStage

func (p *DefaultPipeline) WithTFExamplesStage(stage TFExamplesStage) *DefaultPipeline

WithTFExamplesStage defines TFExamplesStage of a pipeline. If TFExamplesStage has been already set, the resulting TFExamplesStage will chain together transformations in order of setting.

func (*DefaultPipeline) WithTFRecordStage

func (p *DefaultPipeline) WithTFRecordStage(stage TFRecordStage) *DefaultPipeline

WithTFRecordStage defines TFRecordStage of a pipeline. Overrides previous value.

func (*DefaultPipeline) WithTarStage

func (p *DefaultPipeline) WithTarStage(stage TarStage) *DefaultPipeline

WithTarStage defines TarStage of a pipeline. Overrides previous value.

type Sample2TFExampleStage

type Sample2TFExampleStage func(core.SampleReader) core.TFExampleReader

Samples2ExampleStage transforms core.Sample to core.TFExample: consumes core.SampleReader and produces core.TFExampleReader

type SamplesStage

type SamplesStage func(core.SampleReader) core.SampleReader

SamplesStage makes transformation on core.Sample: consumes core.SampleReader and produces core.SampleReader

type TFExamplesStage

type TFExamplesStage func(core.TFExampleReader) core.TFExampleReader

SamplesStage makes transformation on core.TFExample: consumes core.TFExampleReader and produces core.TFExampleReader

type TFRecordStage

type TFRecordStage func(core.TFExampleReader) error

TFRecordStage consumes core.TFExampleReader

type TarStage

type TarStage func() (core.SampleReader, error)

TarStage produces core.SampleReader