filesystem

Documentation

Index

• Constants
• type Source
  • func (s *Source) ChunkUnit(ctx context.Context, unit sources.SourceUnit, reporter sources.ChunkReporter) error
  • func (s *Source) Chunks(ctx context.Context, chunksChan chan *sources.Chunk) error
  • func (s *Source) Enumerate(ctx context.Context, reporter sources.UnitReporter) error
  • func (s *Source) Init(aCtx context.Context, name string, jobId, sourceId int64, verify bool, ...) error
  • func (s *Source) JobID() int64
  • func (s *Source) SourceID() int64
  • func (s *Source) Type() sourcespb.SourceType
  • func (s *Source) WithFilter(filter *common.Filter)

Constants

const (
	// These buffer sizes are mainly driven by our largest credential size, which is GCP @ ~2.25KB.
	// Having a peek size larger than that ensures that we have complete credential coverage in our chunks.
	BufferSize = 10 * 1024 // 10KB
	PeekSize   = 3 * 1024  // 3KB
)

Types

type Source

type Source struct {
	sources.Progress
	sources.CommonSourceUnitUnmarshaller
	// contains filtered or unexported fields
}

func (*Source) ChunkUnit

func (s *Source) ChunkUnit(ctx context.Context, unit sources.SourceUnit, reporter sources.ChunkReporter) error

ChunkUnit implements SourceUnitChunker interface.

func (*Source) Chunks

func (s *Source) Chunks(ctx context.Context, chunksChan chan *sources.Chunk) error

Chunks emits chunks of bytes over a channel.

func (*Source) Enumerate

func (s *Source) Enumerate(ctx context.Context, reporter sources.UnitReporter) error

Enumerate implements SourceUnitEnumerator interface. This implementation simply passes the configured paths as the source unit, whether it be a single filepath or a directory.

func (*Source) Init

func (s *Source) Init(aCtx context.Context, name string, jobId, sourceId int64, verify bool, connection *anypb.Any, _ int) error

Init returns an initialized Filesystem source.

func (*Source) JobID

func (s *Source) JobID() int64

func (*Source) SourceID

func (s *Source) SourceID() int64

func (*Source) Type

func (s *Source) Type() sourcespb.SourceType

Type returns the type of source. It is used for matching source types in configuration and job input.

func (*Source) WithFilter

func (s *Source) WithFilter(filter *common.Filter)