downloaders

Documentation

Index

• func FTPDownload(logger *logging.Logger, config types.SystemConfig, dbInstance db.Storage, ...) error
• func HTTPDownload(logger *logging.Logger, config types.SystemConfig, dbInstance db.Storage, ...) error
• func S3Download(logger *logging.Logger, config types.SystemConfig, dbInstance db.Storage, ...) error
• type DownloadFunc
  • func GetDownloadFunc(jobSource string) DownloadFunc

Functions

func FTPDownload

func FTPDownload(logger *logging.Logger, config types.SystemConfig, dbInstance db.Storage, jobID string) error

FTPDownload downloads the file from FTP. Job Source should be in format: ftp://login:password@host/path

func HTTPDownload

func HTTPDownload(logger *logging.Logger, config types.SystemConfig, dbInstance db.Storage, jobID string) error

HTTPDownload function downloads sources using http protocol.

func S3Download

func S3Download(logger *logging.Logger, config types.SystemConfig, dbInstance db.Storage, jobID string) error

S3Download downloads the file from S3 bucket. Job Source should be in format: http://AWSKEY:AWSSECRET@BUCKET.s3.amazonaws.com/OBJECT

Types

type DownloadFunc

type DownloadFunc func(logger *logging.Logger, config types.SystemConfig, dbInstance db.Storage, jobID string) error

DownloadFunc is a function type for the multiple possible ways to download the source file

func GetDownloadFunc

func GetDownloadFunc(jobSource string) DownloadFunc

GetDownloadFunc returns the download function based on the job source.