transform

Documentation

Overview

Package transform provides implementation of tfdata.Transformation

Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved.

Package tfdata provides interfaces to interact with TFRecord files and TFExamples.

Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved.

Index

• func NewSampleTransformer(reader core.SampleReader, ts ...SampleTransformation) core.SampleReader
• func NewTFExampleTransformer(reader core.TFExampleReader, ts ...TFExampleTransformation) core.TFExampleReader
• func SamplesToTFExample(reader core.SampleReader, types ...core.TypesMap) core.TFExampleReader
• type ExampleSelectionsTransformation
  • func ExampleSelections(s ...selection.Example) *ExampleSelectionsTransformation
  • func (s *ExampleSelectionsTransformation) TransformTFExample(ex *core.TFExample) *core.TFExample
• type ID
  • func (t ID) TransformSample(s *core.Sample) *core.Sample
  • func (t ID) TransformTFExample(ex *core.TFExample) *core.TFExample
• type Rename
  • func RenameTransformation(dest string, src []string) *Rename
  • func (c *Rename) TransformSample(sample *core.Sample) *core.Sample
  • func (c *Rename) TransformTFExample(ex *core.TFExample) *core.TFExample
• type SampleFuncTransformation
  • func SampleF(f func(*core.Sample) *core.Sample) *SampleFuncTransformation
  • func (t *SampleFuncTransformation) TransformSample(sample *core.Sample) *core.Sample
• type SampleSelectionsTransformation
  • func SampleSelections(s ...selection.Sample) *SampleSelectionsTransformation
  • func (s *SampleSelectionsTransformation) TransformSample(sample *core.Sample) *core.Sample
• type SampleToTFExamplesTypesTransformer
  • func (t *SampleToTFExamplesTypesTransformer) Read() (*core.TFExample, error)
• type SampleTransformation
• type SamplesToTFExamplesTransformer
  • func (t *SamplesToTFExamplesTransformer) Read() (*core.TFExample, error)
• type SamplesTransformer
  • func (t *SamplesTransformer) Read() (*core.Sample, error)
• type TFExampleFuncTransformation
  • func ExampleF(f func(*core.TFExample) *core.TFExample) *TFExampleFuncTransformation
  • func (t *TFExampleFuncTransformation) TransformTFExample(ex *core.TFExample) *core.TFExample
• type TFExampleTransformation
• type TFExampleTransformer
  • func (t *TFExampleTransformer) Read() (*core.TFExample, error)

Functions

func NewSampleTransformer

func NewSampleTransformer(reader core.SampleReader, ts ...SampleTransformation) core.SampleReader

NewSampleTransformer consumes TFExampleReader, applies transformations in order of occurrence, produces SampleReader.

func NewTFExampleTransformer

func NewTFExampleTransformer(reader core.TFExampleReader, ts ...TFExampleTransformation) core.TFExampleReader

NewTFExampleTransformer consumes TFExampleReader, applies transformations in order of occurrence, produces TFExampleReader.

func SamplesToTFExample

func SamplesToTFExample(reader core.SampleReader, types ...core.TypesMap) core.TFExampleReader

SamplesToTFExample consumes SampleReader, applies default Sample to TFExample conversion, produces TFExampleReader. Default Sample to TFExample conversion is put into TFExample each of Sample entries as BytesList

Types

type ExampleSelectionsTransformation

type ExampleSelectionsTransformation struct {
	// contains filtered or unexported fields
}

Example transformation based on selections. Result is union of selections

func ExampleSelections

func ExampleSelections(s ...selection.Example) *ExampleSelectionsTransformation

Return Transformation based on specified Selections. Resulting TFExamples are unions of selections.

func (*ExampleSelectionsTransformation) TransformTFExample

func (s *ExampleSelectionsTransformation) TransformTFExample(ex *core.TFExample) *core.TFExample

type ID

type ID struct{}

ID transformation, does nothing

func (ID) TransformSample

func (t ID) TransformSample(s *core.Sample) *core.Sample

func (ID) TransformTFExample

func (t ID) TransformTFExample(ex *core.TFExample) *core.TFExample

type Rename

type Rename struct {
	// contains filtered or unexported fields
}

func RenameTransformation

func RenameTransformation(dest string, src []string) *Rename

func (*Rename) TransformSample

func (c *Rename) TransformSample(sample *core.Sample) *core.Sample

func (*Rename) TransformTFExample

func (c *Rename) TransformTFExample(ex *core.TFExample) *core.TFExample

type SampleFuncTransformation

type SampleFuncTransformation struct {
	// contains filtered or unexported fields
}

Transformation based on function

func SampleF

func SampleF(f func(*core.Sample) *core.Sample) *SampleFuncTransformation

func (*SampleFuncTransformation) TransformSample

func (t *SampleFuncTransformation) TransformSample(sample *core.Sample) *core.Sample

type SampleSelectionsTransformation

type SampleSelectionsTransformation struct {
	// contains filtered or unexported fields
}

Sample transformation based on selections. Result is union of selections

func SampleSelections

func SampleSelections(s ...selection.Sample) *SampleSelectionsTransformation

Return Transformation based on specified Selections. Resulting Samples are unions of selections.

func (*SampleSelectionsTransformation) TransformSample

func (s *SampleSelectionsTransformation) TransformSample(sample *core.Sample) *core.Sample

type SampleToTFExamplesTypesTransformer

type SampleToTFExamplesTypesTransformer struct {
	// contains filtered or unexported fields
}

func (*SampleToTFExamplesTypesTransformer) Read

func (t *SampleToTFExamplesTypesTransformer) Read() (*core.TFExample, error)

type SampleTransformation

type SampleTransformation interface {
	TransformSample(s *core.Sample) *core.Sample
}

type SamplesToTFExamplesTransformer

type SamplesToTFExamplesTransformer struct {
	// contains filtered or unexported fields
}

Default SamplesToTFExamples transformer: put into TFExample each of Sample entries as BytesList

func (*SamplesToTFExamplesTransformer) Read

func (t *SamplesToTFExamplesTransformer) Read() (*core.TFExample, error)

type SamplesTransformer

type SamplesTransformer struct {
	// contains filtered or unexported fields
}

Transforms SamplesReader based on given transformations

func (*SamplesTransformer) Read

func (t *SamplesTransformer) Read() (*core.Sample, error)

type TFExampleFuncTransformation

type TFExampleFuncTransformation struct {
	// contains filtered or unexported fields
}

Transformation based on function

func ExampleF

func ExampleF(f func(*core.TFExample) *core.TFExample) *TFExampleFuncTransformation

func (*TFExampleFuncTransformation) TransformTFExample

func (t *TFExampleFuncTransformation) TransformTFExample(ex *core.TFExample) *core.TFExample

type TFExampleTransformation

type TFExampleTransformation interface {
	TransformTFExample(ex *core.TFExample) *core.TFExample
}

type TFExampleTransformer

type TFExampleTransformer struct {
	// contains filtered or unexported fields
}

TFExampleTransformer - it does everything locally, doesn't even preload anything from internal reader. However, it's easy to imagine more advanced Transformer (Reader) which has the same Read() interface, but underneath prefetches TFExamples, distributes them amongst external workers and after transformations gathers TFExamples and make them available to Read()

func (*TFExampleTransformer) Read

func (t *TFExampleTransformer) Read() (*core.TFExample, error)