splitter

Documentation

Index

• type IngestionDataSplitter
  • func NewIngestionDataSplitter(ctx context.Context, maxItemsPerSplit int, secondaryGCSPath string, ...) (*IngestionDataSplitter, error)
  • func (s IngestionDataSplitter) SplitAndPublishFormattedData(ctx context.Context, f format.Format, filename string) error

Types

type IngestionDataSplitter

type IngestionDataSplitter struct {
	// contains filtered or unexported fields
}

IngestionDataSplitter provides a struct to split large input data files into smaller splits and write them to a secondary GCS path.

func NewIngestionDataSplitter

func NewIngestionDataSplitter(ctx context.Context, maxItemsPerSplit int, secondaryGCSPath string, gcsClient gcs.GCSClient) (*IngestionDataSplitter, error)

NewIngestionDataSplitter returns a new instance of the IngestionDataSplitter.

func (IngestionDataSplitter) SplitAndPublishFormattedData

func (s IngestionDataSplitter) SplitAndPublishFormattedData(ctx context.Context, f format.Format, filename string) error

SplitAndPublishFormattedData splits the data in f into smaller chunks and writes the individual chunks as a separate file in GCS.