zedUpload

package module
v0.0.0-...-15d82a0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 2, 2023 License: Apache-2.0 Imports: 27 Imported by: 2

README

zedUpload

This directory contains functions for uploading and downloading to various remote file storage services, specifically:

  • AWS S3
  • Azure Blob
  • http/s
  • sftp

Testing

In order to run full tests, you need to have a remote account. To run the tests, set an environment variable for the credentials you need. Those are listed below. If a credential is missing for a test, it simply will not run.

The required environment variables are as follows:

  • AWS: TEST_AWS_BUCKET, TEST_AWS_KEY, TEST_AWS_SECRET, TEST_AWS_REGION
  • Azure: TEST_AZURE_CONTAINER, TEST_AZURE_ACCOUNT_NAME, TEST_AZURE_ACCOUNT_KEY
  • http: none are needed, as tests use the public ptsv2 for post testing, and Cirros Cloud and Ubuntu Images for download testing
  • sftp: TEST_SFTP_DIR, TEST_SFTP_USER, TEST_SFTP_PASS, TEST_SFTP_REGION

Documentation

Index

Constants

View Source 
const (
	SyncOpUnknown               = 0
	SyncOpUpload                = 1
	SyncOpDownload              = 2
	SyncOpDelete                = 3
	SyncOpDownloadWithSignature = 4
	SyncOpList                  = 5
	SyncOpGetObjectMetaData     = 6
	SyncOpGetURI                = 7
	SysOpPutPart                = 8
	SysOpCompleteParts          = 9
	SysOpDownloadByChunks       = 10
	DefaultNumberOfHandlers     = 11

	StatsUpdateTicker = 1 * time.Second // timer for updating client for stats
	FailPostTimeout   = 2 * time.Minute
)

Operation types supported

View Source 
const (
	//SingleMB represents data size of 1MB
	SingleMB int64 = 1024 * 1024
)

Variables

View Source 
var (
	SyncUnsupport = errors.New("Unsupported function")
	SyncerRetry   = errors.New("Retry - syncer full")
)

Functions

This section is empty.

Types

type AuthInput 

type AuthInput struct {
	// type of auth
	AuthType string

	// required, auth for whom
	Uname string

	// optional, password
	Password string

	// optional, keytabs
	Keys []string
}

type AwsTransportMethod 

type AwsTransportMethod struct {
	// contains filtered or unexported fields
}

func (*AwsTransportMethod) Action 

func (ep *AwsTransportMethod) Action(req *DronaRequest) error

Action : execute selected action targeting AWS datastore.

func (*AwsTransportMethod) Close 

func (ep *AwsTransportMethod) Close() error

func (*AwsTransportMethod) GetNetTrace 

func (ep *AwsTransportMethod) GetNetTrace(description string) (
	nettrace.AnyNetTrace, []nettrace.PacketCapture, error)

GetNetTrace returns collected network trace and packet captures.

func (*AwsTransportMethod) NewRequest 

func (ep *AwsTransportMethod) NewRequest(opType SyncOpType, objname, objloc string, sizelimit int64, ackback bool, reply chan *DronaRequest) *DronaRequest

func (*AwsTransportMethod) Open 

func (ep *AwsTransportMethod) Open() error

func (*AwsTransportMethod) WithBindIntf 

func (ep *AwsTransportMethod) WithBindIntf(intf string) error

WithBindIntf : bind to specific interface for this connection

func (*AwsTransportMethod) WithLogging 

func (ep *AwsTransportMethod) WithLogging(onoff bool) error

WithLogging enables or disables logging.

func (*AwsTransportMethod) WithNetTracing 

func (ep *AwsTransportMethod) WithNetTracing(opts ...nettrace.TraceOpt) error

WithNetTracing enables network tracing.

func (*AwsTransportMethod) WithProxy 

func (ep *AwsTransportMethod) WithProxy(proxy *url.URL) error

WithProxy : connect via the provided proxy URL.

func (*AwsTransportMethod) WithSrcIP 

func (ep *AwsTransportMethod) WithSrcIP(localAddr net.IP) error

WithSrcIP : use the specific IP as source address for this connection.

func (*AwsTransportMethod) WithTrustedCerts 

func (ep *AwsTransportMethod) WithTrustedCerts(certs [][]byte) error

WithTrustedCerts : run requests with these certificates added as trusted.

type AzureTransportMethod 

type AzureTransportMethod struct {
	// contains filtered or unexported fields
}

func (*AzureTransportMethod) Action 

func (ep *AzureTransportMethod) Action(req *DronaRequest) error

Action : execute selected action targeting Azure datastore.

func (*AzureTransportMethod) Close 

func (ep *AzureTransportMethod) Close() error

func (*AzureTransportMethod) GetNetTrace 

func (ep *AzureTransportMethod) GetNetTrace(description string) (
	nettrace.AnyNetTrace, []nettrace.PacketCapture, error)

GetNetTrace returns collected network trace and packet captures.

func (*AzureTransportMethod) NewRequest 

func (ep *AzureTransportMethod) NewRequest(opType SyncOpType, objname, objloc string, sizelimit int64, ackback bool, reply chan *DronaRequest) *DronaRequest

func (*AzureTransportMethod) Open 

func (ep *AzureTransportMethod) Open() error

func (*AzureTransportMethod) WithBindIntf 

func (ep *AzureTransportMethod) WithBindIntf(intf string) error

WithBindIntf : bind to specific interface for this connection

func (*AzureTransportMethod) WithLogging 

func (ep *AzureTransportMethod) WithLogging(onoff bool) error

WithLogging enables or disables logging.

func (*AzureTransportMethod) WithNetTracing 

func (ep *AzureTransportMethod) WithNetTracing(opts ...nettrace.TraceOpt) error

WithNetTracing enables network tracing.

func (*AzureTransportMethod) WithProxy 

func (ep *AzureTransportMethod) WithProxy(proxy *url.URL) error

WithProxy : connect via the provided proxy URL.

func (*AzureTransportMethod) WithSrcIP 

func (ep *AzureTransportMethod) WithSrcIP(localAddr net.IP) error

WithSrcIP : use the specific IP as source address for this connection.

func (*AzureTransportMethod) WithTrustedCerts 

func (ep *AzureTransportMethod) WithTrustedCerts(certs [][]byte) error

WithTrustedCerts : run requests with these certificates added as trusted.

type ChunkData 

type ChunkData struct {
	Size  int64  // complete size to upload/download
	Chunk []byte // chunk data being uploaded/downloaded
	EOF   bool   // denotes last chunk of the file
}

ChunkData contains the details of Chunks being downloaded

type DronaCtx 

type DronaCtx struct {
	// contains filtered or unexported fields
}

func NewDronaCtx 

func NewDronaCtx(name string, noHandlers int) (*DronaCtx, error)

NewDronaCtx

func (*DronaCtx) ListenAndServe 

func (ctx *DronaCtx) ListenAndServe()

Keep working till we are told otherwise

func (*DronaCtx) NewSyncerDest 

func (ctx *DronaCtx) NewSyncerDest(tr SyncTransportType, UrlOrRegion, PathOrBkt string, auth *AuthInput) (DronaEndPoint, error)

NewSyncerDest:

  • add another location end point to syncer

type DronaEndPoint 

type DronaEndPoint interface {
	NewRequest(SyncOpType, string, string, int64, bool, chan *DronaRequest) *DronaRequest
	Open() error
	Action(req *DronaRequest) error
	Close() error
	WithSrcIP(localAddr net.IP) error
	WithTrustedCerts(certs [][]byte) error
	WithProxy(proxy *url.URL) error
	WithBindIntf(intf string) error
	WithLogging(onoff bool) error
	WithNetTracing(opts ...nettrace.TraceOpt) error
	// GetNetTrace : if network tracing is enabled (WithNetTracing() was called),
	// this method returns trace of all network operations performed up to this point,
	// possibly also accompanied by per-interface packet captures (only if enabled by
	// tracing options).
	GetNetTrace(description string) (
		nettrace.AnyNetTrace, []nettrace.PacketCapture, error)
	// contains filtered or unexported methods
}

Interface for various transport implementation

type DronaRequest 

type DronaRequest struct {
	sync.RWMutex

	// Image Sha256
	ImageSha256 string

	// used by Multipart upload
	Adata  []byte
	PartID int64
	SasURI string

	// used by azure and s3 while Multipart Upload
	Blocks []string
	// generated SasURI TTL
	Duration time.Duration
	// generated while creating multipart file
	UploadID string
	// generated after uploading the part to the multipart file
	EtagID string
	// contains filtered or unexported fields
}

func ReadMetaFile 

func ReadMetaFile(metaloc string) (error, *DronaRequest)

func (*DronaRequest) Cancel 

func (req *DronaRequest) Cancel() error

Cancel checks if WithCancel was used.

func (*DronaRequest) GetAsize 

func (req *DronaRequest) GetAsize() int64

Return object actual synced down size

func (*DronaRequest) GetChunkDetails 

func (req *DronaRequest) GetChunkDetails() (int64, []byte, bool)

GetChunkDetails Return the chunk details

func (*DronaRequest) GetContentLength 

func (req *DronaRequest) GetContentLength() int64

func (*DronaRequest) GetContentType 

func (req *DronaRequest) GetContentType() string

GetContentType return the content type if available. If not, return an empty string.

func (*DronaRequest) GetDnStatus 

func (req *DronaRequest) GetDnStatus() error

Return object error status string

func (*DronaRequest) GetDoneParts 

func (req *DronaRequest) GetDoneParts() types.DownloadedParts

GetDoneParts returns already downloaded parts indexes

func (*DronaRequest) GetImageList 

func (req *DronaRequest) GetImageList() []string

func (*DronaRequest) GetLocalName 

func (req *DronaRequest) GetLocalName() string

Return object local name

func (*DronaRequest) GetOsize 

func (req *DronaRequest) GetOsize() int64

Return object actual synced down size

func (*DronaRequest) GetRemoteFileMD5 

func (req *DronaRequest) GetRemoteFileMD5() string

func (*DronaRequest) GetSha256 

func (req *DronaRequest) GetSha256() string

GetSha256 returns an image sha256

func (*DronaRequest) GetStatus 

func (req *DronaRequest) GetStatus() string

Return the status

func (*DronaRequest) GetUpStatus 

func (req *DronaRequest) GetUpStatus() (string, error)

func (*DronaRequest) IsDnUpdate 

func (req *DronaRequest) IsDnUpdate() bool

Return is it update

func (*DronaRequest) IsError 

func (req *DronaRequest) IsError() bool

Return the if the object was downloaded with error

func (*DronaRequest) Post 

func (req *DronaRequest) Post() error

func (*DronaRequest) Progress 

func (req *DronaRequest) Progress() (int64, int64, uint)

Progress return download progress returns asize (bytes), osize (bytes), progress (percent out of 100)

func (*DronaRequest) WithCancel 

func (req *DronaRequest) WithCancel(ctx context.Context) *DronaRequest

WithCancel can be used to setup cancellation Caller should call req.Cancel when done even on success

func (*DronaRequest) WithDoneParts 

func (req *DronaRequest) WithDoneParts(doneParts types.DownloadedParts) *DronaRequest

WithDoneParts can be used to set already downloaded parts indexes

func (*DronaRequest) WithLogger 

func (req *DronaRequest) WithLogger(logger types.Logger) *DronaRequest

WithLogger sets logger

func (*DronaRequest) WriteMetaFile 

func (req *DronaRequest) WriteMetaFile(metaloc string) error

helper function to write the metafile for the transcation that is defined by this request

type GsTransportMethod 

type GsTransportMethod struct {
	// contains filtered or unexported fields
}

GsTransportMethod stores data needed to communicate with Google Cloud Storage

func (*GsTransportMethod) Action 

func (ep *GsTransportMethod) Action(req *DronaRequest) error

Action do the operation with Google Storage datastore

func (*GsTransportMethod) Close 

func (ep *GsTransportMethod) Close() error

Close not implemented

func (*GsTransportMethod) GetNetTrace 

func (ep *GsTransportMethod) GetNetTrace(description string) (
	nettrace.AnyNetTrace, []nettrace.PacketCapture, error)

GetNetTrace returns collected network trace and packet captures.

func (*GsTransportMethod) NewRequest 

func (ep *GsTransportMethod) NewRequest(opType SyncOpType, objname, objloc string, sizelimit int64, ackback bool, reply chan *DronaRequest) *DronaRequest

NewRequest returns DronaRequest for provided options

func (*GsTransportMethod) Open 

func (ep *GsTransportMethod) Open() error

Open not implemented

func (*GsTransportMethod) WithBindIntf 

func (ep *GsTransportMethod) WithBindIntf(intf string) error

WithBindIntf : bind to specific interface for this connection

func (*GsTransportMethod) WithLogging 

func (ep *GsTransportMethod) WithLogging(onoff bool) error

WithLogging enables or disables logging.

func (*GsTransportMethod) WithNetTracing 

func (ep *GsTransportMethod) WithNetTracing(opts ...nettrace.TraceOpt) error

WithNetTracing enables network tracing.

func (*GsTransportMethod) WithProxy 

func (ep *GsTransportMethod) WithProxy(proxy *url.URL) error

WithProxy : connect via the provided proxy URL.

func (*GsTransportMethod) WithSrcIP 

func (ep *GsTransportMethod) WithSrcIP(localAddr net.IP) error

WithSrcIP : use the specific IP as source address for this connection.

func (*GsTransportMethod) WithTrustedCerts 

func (ep *GsTransportMethod) WithTrustedCerts(certs [][]byte) error

WithTrustedCerts : run requests with these certificates added as trusted.

type HttpTransportMethod 

type HttpTransportMethod struct {
	// contains filtered or unexported fields
}

func (*HttpTransportMethod) Action 

func (ep *HttpTransportMethod) Action(req *DronaRequest) error

Action : execute selected action targeting HTTP datastore.

func (*HttpTransportMethod) Close 

func (ep *HttpTransportMethod) Close() error

func (*HttpTransportMethod) GetNetTrace 

func (ep *HttpTransportMethod) GetNetTrace(description string) (
	nettrace.AnyNetTrace, []nettrace.PacketCapture, error)

GetNetTrace returns collected network trace and packet captures.

func (*HttpTransportMethod) NewRequest 

func (ep *HttpTransportMethod) NewRequest(opType SyncOpType, objname, objloc string,
	sizelimit int64, ackback bool, reply chan *DronaRequest) *DronaRequest

func (*HttpTransportMethod) Open 

func (ep *HttpTransportMethod) Open() error

func (*HttpTransportMethod) WithBindIntf 

func (ep *HttpTransportMethod) WithBindIntf(intf string) error

WithBindIntf : bind to specific interface for this connection

func (*HttpTransportMethod) WithLogging 

func (ep *HttpTransportMethod) WithLogging(onoff bool) error

WithLogging enables or disables logging.

func (*HttpTransportMethod) WithNetTracing 

func (ep *HttpTransportMethod) WithNetTracing(opts ...nettrace.TraceOpt) error

WithNetTracing enables network tracing.

func (*HttpTransportMethod) WithProxy 

func (ep *HttpTransportMethod) WithProxy(proxy *url.URL) error

WithProxy : connect via the provided proxy URL.

func (*HttpTransportMethod) WithSrcIP 

func (ep *HttpTransportMethod) WithSrcIP(localAddr net.IP) error

WithSrcIP : use the specific IP as source address for this connection.

func (*HttpTransportMethod) WithTrustedCerts 

func (ep *HttpTransportMethod) WithTrustedCerts(certs [][]byte) error

WithTrustedCerts : run requests with these certificates added as trusted.

type OCITransportMethod 

type OCITransportMethod struct {
	// contains filtered or unexported fields
}

OCITransportMethod transport method to send images from OCI distribution registries

func (*OCITransportMethod) Action 

func (ep *OCITransportMethod) Action(req *DronaRequest) error

Action perform an action using this method, one of Download/Upload/Delete/List/GetObjectMetaData

func (*OCITransportMethod) Close 

func (ep *OCITransportMethod) Close() error

Close unsupported

func (*OCITransportMethod) GetNetTrace 

func (ep *OCITransportMethod) GetNetTrace(description string) (
	nettrace.AnyNetTrace, []nettrace.PacketCapture, error)

GetNetTrace returns collected network trace and packet captures.

func (*OCITransportMethod) NewRequest 

func (ep *OCITransportMethod) NewRequest(opType SyncOpType, objname, objloc string,
	sizelimit int64, ackback bool, reply chan *DronaRequest) *DronaRequest

NewRequest create a new DronaRequest with this OCITransportMethod as the endpoint

func (*OCITransportMethod) Open 

func (ep *OCITransportMethod) Open() error

Open unsupported

func (*OCITransportMethod) WithBindIntf 

func (ep *OCITransportMethod) WithBindIntf(intf string) error

WithBindIntf : bind to specific interface for this connection

func (*OCITransportMethod) WithLogging 

func (ep *OCITransportMethod) WithLogging(onoff bool) error

WithLogging enables or disables logging.

func (*OCITransportMethod) WithNetTracing 

func (ep *OCITransportMethod) WithNetTracing(opts ...nettrace.TraceOpt) error

WithNetTracing enables network tracing.

func (*OCITransportMethod) WithProxy 

func (ep *OCITransportMethod) WithProxy(proxy *url.URL) error

WithProxy : connect via the provided proxy URL.

func (*OCITransportMethod) WithSrcIP 

func (ep *OCITransportMethod) WithSrcIP(localAddr net.IP) error

WithSrcIP : use the specific IP as source address for this connection.

func (*OCITransportMethod) WithTrustedCerts 

func (ep *OCITransportMethod) WithTrustedCerts(certs [][]byte) error

WithTrustedCerts : run requests with these certificates added as trusted.

type SftpTransportMethod 

type SftpTransportMethod struct {
	// contains filtered or unexported fields
}

func (*SftpTransportMethod) Action 

func (ep *SftpTransportMethod) Action(req *DronaRequest) error

Action : execute selected action targeting SFTP datastore.

func (*SftpTransportMethod) Close 

func (ep *SftpTransportMethod) Close() error

func (*SftpTransportMethod) GetNetTrace 

func (ep *SftpTransportMethod) GetNetTrace(description string) (
	nettrace.AnyNetTrace, []nettrace.PacketCapture, error)

GetNetTrace is not supported.

func (*SftpTransportMethod) NewRequest 

func (ep *SftpTransportMethod) NewRequest(opType SyncOpType, objname, objloc string, sizelimit int64, ackback bool, reply chan *DronaRequest) *DronaRequest

func (*SftpTransportMethod) Open 

func (ep *SftpTransportMethod) Open() error

func (*SftpTransportMethod) WithBindIntf 

func (ep *SftpTransportMethod) WithBindIntf(intf string) error

WithBindIntf is not supported.

func (*SftpTransportMethod) WithLogging 

func (ep *SftpTransportMethod) WithLogging(onoff bool) error

WithLogging enables or disables logging.

func (*SftpTransportMethod) WithNetTracing 

func (ep *SftpTransportMethod) WithNetTracing(opts ...nettrace.TraceOpt) error

WithNetTracing is not supported.

func (*SftpTransportMethod) WithProxy 

func (ep *SftpTransportMethod) WithProxy(proxy *url.URL) error

WithProxy is not supported.

func (*SftpTransportMethod) WithSrcIP 

func (ep *SftpTransportMethod) WithSrcIP(localAddr net.IP) error

WithSrcIP is not supported.

func (*SftpTransportMethod) WithTrustedCerts 

func (ep *SftpTransportMethod) WithTrustedCerts(certs [][]byte) error

WithTrustedCerts is not supported.

type SyncMetaFile 

type SyncMetaFile struct {
	Operation SyncOpType

	// Object that needs to be downloaded
	Name string

	// Filled by Syncer, actual size
	Asize int64

	// Status of Download, we convert here to string because this
	// field is going to be json marshalled
	Status string

	EndTime   string
	StartTime string
}

type SyncOpType 

type SyncOpType int

Sync Operation type

type SyncTransportType 

type SyncTransportType string

Sync Transport Type 

const (
	SyncAwsTr         SyncTransportType = "s3"
	SyncAzureTr       SyncTransportType = "azure"
	SyncGSTr          SyncTransportType = "google"
	SyncHttpTr        SyncTransportType = "http"
	SyncSftpTr        SyncTransportType = "sftp"
	SyncOCIRegistryTr SyncTransportType = "oci"
)

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.