sources

package
v0.6.20 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 26, 2019 License: Apache-2.0, BSD-3-Clause, MIT Imports: 20 Imported by: 1

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func NewAzBlobSourcePipelineFactory added in v0.6.12 

func NewAzBlobSourcePipelineFactory(params *AzureBlobParams) <-chan FactoryResult

NewAzBlobSourcePipelineFactory TODO

func NewFileSystemSourcePipelineFactory added in v0.6.12 

func NewFileSystemSourcePipelineFactory(params *FileSystemSourceParams) <-chan FactoryResult

NewFileSystemSourcePipelineFactory TODO

func NewHTTPSourcePipelineFactory added in v0.6.12 

func NewHTTPSourcePipelineFactory(params HTTPSourceParams) <-chan FactoryResult

NewHTTPSourcePipelineFactory TODO

func NewPerfSourcePipelineFactory added in v0.6.12 

func NewPerfSourcePipelineFactory(params PerfSourceParams) <-chan FactoryResult

NewPerfSourcePipelineFactory TODO

func NewS3SourcePipelineFactory added in v0.6.12 

func NewS3SourcePipelineFactory(params *S3Params) <-chan FactoryResult

NewS3SourcePipelineFactory returns TODO

Types

type AzureBlobParams 

type AzureBlobParams struct {
	SourceParams
	Container   string
	BlobNames   []string
	AccountName string
	AccountKey  string
	SasExp      int
	BaseBlobURL string
}

AzureBlobParams parameters for the creation of Azure Blob source pipeline

type AzureBlobSource added in v0.6.12 

type AzureBlobSource struct {
	HTTPSource
	// contains filtered or unexported fields
}

AzureBlobSource constructs parts channel and implements data readers for Azure Blobs exposed via HTTP

type FactoryResult added in v0.6.12 

type FactoryResult struct {
	Source pipeline.SourcePipeline
	Err    error
}

FactoryResult TODO

type FileInfo 

type FileInfo struct {
	FileStats   os.FileInfo
	SourceURI   string
	TargetAlias string
	NumOfBlocks int
}

FileInfo Contains the metadata associated with a file to be transferred

type FileSystemSource added in v0.6.12 

type FileSystemSource struct {
	// contains filtered or unexported fields
}

FileSystemSource Contructs blocks queue and implements data readers

func (*FileSystemSource) ConstructBlockInfoQueue added in v0.6.12 

func (f *FileSystemSource) ConstructBlockInfoQueue(blockSize uint64) (partitionsQ chan pipeline.PartsPartition, partsQ chan pipeline.Part, numOfBlocks int, size uint64)

ConstructBlockInfoQueue implements ConstructBlockInfoQueue from the pipeline.SourcePipeline Interface. this implementation uses partitions to group parts into a set that can be read sequentially. This is to avoid Window's memory pressure when calling SetFilePointer numerous times on the same handle

func (*FileSystemSource) ExecuteReader added in v0.6.12 

func (f *FileSystemSource) ExecuteReader(partitionsQ chan pipeline.PartsPartition, partsQ chan pipeline.Part, readPartsQ chan pipeline.Part, id int, wg *sync.WaitGroup)

ExecuteReader implements ExecuteReader from the pipeline.SourcePipeline Interface. For each file the reader will maintain a open handle from which data will be read. This implementation uses partitions (group of parts that can be read sequentially).

func (*FileSystemSource) GetSourcesInfo added in v0.6.12 

func (f *FileSystemSource) GetSourcesInfo() []pipeline.SourceInfo

GetSourcesInfo implements GetSourcesInfo from the pipeline.SourcePipeline Interface. Returns an an array of SourceInfo with the name, alias and size of the files to be transferred.

type FileSystemSourceParams added in v0.6.12 

type FileSystemSourceParams struct {
	SourceParams
	SourcePatterns  []string
	BlockSize       uint64
	TargetAliases   []string
	NumOfPartitions int
}

FileSystemSourceParams parameters used to create a new instance of multi-file source pipeline

type HTTPSource added in v0.6.12 

type HTTPSource struct {
	Sources    []pipeline.SourceInfo
	HTTPClient *http.Client
	// contains filtered or unexported fields
}

HTTPSource constructs parts channel and implements data readers for file exposed via HTTP

func (*HTTPSource) ConstructBlockInfoQueue added in v0.6.12 

func (f *HTTPSource) ConstructBlockInfoQueue(blockSize uint64) (partitionsQ chan pipeline.PartsPartition, partsQ chan pipeline.Part, numOfBlocks int, size uint64)

ConstructBlockInfoQueue implements GetSourcesInfo from the pipeline.SourcePipeline Interface. Constructs the Part's channel arithmetically from the size of the sources.

func (*HTTPSource) ExecuteReader added in v0.6.12 

func (f *HTTPSource) ExecuteReader(partitionsQ chan pipeline.PartsPartition, partsQ chan pipeline.Part, readPartsQ chan pipeline.Part, id int, wg *sync.WaitGroup)

ExecuteReader implements ExecuteReader from the pipeline.SourcePipeline Interface. For each part the reader makes a byte range request to the source starting from the part's Offset to BytesToRead - 1 (zero based).

func (*HTTPSource) GetSourcesInfo added in v0.6.12 

func (f *HTTPSource) GetSourcesInfo() []pipeline.SourceInfo

GetSourcesInfo implements GetSourcesInfo from the pipeline.SourcePipeline Interface. Returns an array of pipeline.SourceInfo[] with the files URL, alias and size.

type HTTPSourceParams added in v0.6.12 

type HTTPSourceParams struct {
	SourceParams
	SourceURIs    []string
	TargetAliases []string
}

HTTPSourceParams input parameters for HTTP pipelines

type ObjectListingResult added in v0.6.12 

type ObjectListingResult struct {
	Sources []pipeline.SourceInfo
	Err     error
}

ObjectListingResult TODO

type PerfSourceParams added in v0.6.12 

type PerfSourceParams struct {
	SourceParams
	Definitions []SourceDefinition
	BlockSize   uint64
}

PerfSourceParams TODO

type PerfSourcePipeline added in v0.6.12 

type PerfSourcePipeline struct {
	// contains filtered or unexported fields
}

PerfSourcePipeline TODO

func (*PerfSourcePipeline) ConstructBlockInfoQueue added in v0.6.12 

func (s *PerfSourcePipeline) ConstructBlockInfoQueue(blockSize uint64) (partitionQ chan pipeline.PartsPartition, partsQ chan pipeline.Part, numOfBlocks int, size uint64)

ConstructBlockInfoQueue TODO

func (*PerfSourcePipeline) ExecuteReader added in v0.6.12 

func (s *PerfSourcePipeline) ExecuteReader(partitionQ chan pipeline.PartsPartition, partsQ chan pipeline.Part, readPartsQ chan pipeline.Part, id int, wg *sync.WaitGroup)

ExecuteReader TODO

func (*PerfSourcePipeline) GetSourcesInfo added in v0.6.12 

func (s *PerfSourcePipeline) GetSourcesInfo() []pipeline.SourceInfo

GetSourcesInfo TODO

type S3Params added in v0.5.22 

type S3Params struct {
	SourceParams
	Bucket          string
	Endpoint        string
	Prefixes        []string
	PreSignedExpMin int
	AccessKey       string
	SecretKey       string
}

S3Params parameters used to create a new instance of a S3 source pipeline

type S3Source added in v0.6.12 

type S3Source struct {
	HTTPSource
	// contains filtered or unexported fields
}

S3Source S3 source HTTP based pipeline

type SourceDefinition added in v0.6.12 

type SourceDefinition struct {
	Names           []string
	Size            uint64
	NumberOfSources int
}

SourceDefinition TODO

func ParseSourceDefinitions added in v0.6.12 

func ParseSourceDefinitions(def string) ([]SourceDefinition, error)

ParseSourceDefinitions TODO

type SourceParams added in v0.5.22 

type SourceParams struct {
	ReferenceMode     bool
	CalculateMD5      bool
	UseExactNameMatch bool
	KeepDirStructure  bool
	FilesPerPipeline  int
	Tracker           *internal.TransferTracker
}

SourceParams input base parameters for blob and S3 based pipelines

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.