transfer

Documentation

Overview

Package transfer simplifies interaction with the Object Storage service by abstracting away the method used to upload objects. Depending on the configuration parameters, UploadManager may choose to do a single put_object request, or break up the upload into multiple parts and utilize multi-part uploads.

An advantage of using multi-part uploads is the ability to retry individual failed parts, as well as being able to upload parts in parallel to reduce upload time.

To use this package, you must be authorized in an IAM policy. If you're not authorized, talk to an administrator.

Index

• type FileUploader
• type MultiPartUploadPart
• type MultipartUploadResponse
• type SinglepartUploadResponse
• type StreamUploader
• type UploadCallBack
• type UploadFileRequest
• type UploadManager
  • func NewUploadManager() *UploadManager
  • func (uploadManager *UploadManager) ResumeUploadFile(ctx context.Context, uploadID string) (response UploadResponse, err error)
  • func (uploadManager *UploadManager) UploadFile(ctx context.Context, request UploadFileRequest) (response UploadResponse, err error)
  • func (uploadManager *UploadManager) UploadStream(ctx context.Context, request UploadStreamRequest) (response UploadResponse, err error)
• type UploadRequest
  • func (request UploadRequest) RetryPolicy() *common.RetryPolicy
• type UploadResponse
  • func (resp UploadResponse) IsResumable() bool
• type UploadResponseType
• type UploadStreamRequest

Types

type FileUploader

type FileUploader interface {
	// split file into multiple parts and uploads them to blob storage, then merge
	UploadFileMultiparts(ctx context.Context, request UploadFileRequest) (response UploadResponse, err error)

	// uploads a file to blob storage via PutObject API
	UploadFilePutObject(ctx context.Context, request UploadFileRequest) (response UploadResponse, err error)

	// resume a file upload, use it when UploadFile failed
	ResumeUploadFile(ctx context.Context, uploadID string) (response UploadResponse, err error)
}

FileUploader is an interface to upload a file

type MultiPartUploadPart

type MultiPartUploadPart struct {
	PartNum    int
	TotalParts int
	Size       int64
	Offset     int64
	Hash       *string
	OpcMD5     *string
	Etag       *string
	Err        error
}

MultiPartUploadPart holds the details of Part that is uploaded

type MultipartUploadResponse

type MultipartUploadResponse struct {
	objectstorage.CommitMultipartUploadResponse

	// The upload ID for a multipart upload.
	UploadID *string
	// contains filtered or unexported fields
}

MultipartUploadResponse is the response from commitMultipart API operation.

type SinglepartUploadResponse

type SinglepartUploadResponse struct {
	objectstorage.PutObjectResponse
}

SinglepartUploadResponse is the response from putObject API operation.

type StreamUploader

type StreamUploader interface {
	// uploads a stream to blob storage
	UploadStream(ctx context.Context, request UploadStreamRequest) (response UploadResponse, err error)
}

StreamUploader is an interface for upload a stream

type UploadCallBack

type UploadCallBack func(multiPartUploadPart MultiPartUploadPart)

UploadCallBack API that gets invoked after a Part is successuly uploaded

type UploadFileRequest

type UploadFileRequest struct {
	UploadRequest

	// The path of the file to be uploaded (includs file name)
	FilePath string
}

UploadFileRequest defines the input parameters for UploadFile method

type UploadManager

type UploadManager struct {
	FileUploader   FileUploader
	StreamUploader StreamUploader
}

UploadManager is the interface that groups the upload methods

func NewUploadManager

func NewUploadManager() *UploadManager

NewUploadManager return a pointer to UploadManager

func (*UploadManager) ResumeUploadFile

func (uploadManager *UploadManager) ResumeUploadFile(ctx context.Context, uploadID string) (response UploadResponse, err error)

ResumeUploadFile resumes a multipart file upload.

func (*UploadManager) UploadFile

func (uploadManager *UploadManager) UploadFile(ctx context.Context, request UploadFileRequest) (response UploadResponse, err error)

UploadFile uploads an object to Object Storage. Depending on the options provided and the size of the object, the object may be uploaded in multiple parts or just an signle object.

func (*UploadManager) UploadStream

func (uploadManager *UploadManager) UploadStream(ctx context.Context, request UploadStreamRequest) (response UploadResponse, err error)

UploadStream uploads streaming data to Object Storage. If the stream is non-empty, this will always perform a multipart upload, splitting parts based on the part size (10 MiB if none specified). If the stream is empty, this will upload a single empty object to Object Storage. Stream uploads are not currently resumable.

type UploadRequest

type UploadRequest struct {
	// The top-level namespace used for the request.
	NamespaceName *string `mandatory:"true"`

	// The name of the bucket. Avoid entering confidential information. Example: my-new-bucket1
	BucketName *string `mandatory:"true"`

	// The name of the object. Avoid entering confidential information. Example: test/object1.log
	ObjectName *string `mandatory:"true"`

	// [Optional] Override the default part size of 128 MiB, value is in bytes.
	// The max part size is 50GiB
	PartSize *int64 `mandatory:"false"`

	// [Optional] Whether or not this UploadManager supports performing mulitpart uploads. Defaults to True.
	AllowMultipartUploads *bool `mandatory:"false"`

	// [Optional] Whether or not this UploadManager supports uploading individual parts of a multipart upload in parallel.
	// This setting has no effect on uploads that are performed using a single put_object call. Defaults to True.
	AllowParrallelUploads *bool `mandatory:"false"`

	// The number of go routines for uploading individual parts of a multipart upload.
	// This setting is only used if allow_parallel_uploads is set to True. Defaults to 5.
	// The upper bounds of the number is 10,000.
	NumberOfGoroutines *int `mandatory:"false"`

	// A configured object storage client to use for interacting with the Object Storage service.
	// Default timeout is 60s which includes the time for reading the body.
	// Default timeout doesn't work for big file size and big part size(once upload each part longer than 60s), need to manually update timeout to support big file upload.
	ObjectStorageClient *objectstorage.ObjectStorageClient `mandatory:"false"`

	// [Optional] The entity tag of the object to match.
	IfMatch *string `mandatory:"false"`

	// [Optional] The entity tag of the object to avoid matching. The only valid value is ‘*’,
	// which indicates that the request should fail if the object already exists.
	IfNoneMatch *string `mandatory:"false"`

	// [Optional] The base-64 encoded MD5 hash of the body. This parameter is only used if the object is uploaded in a single part.
	ContentMD5 *string `mandatory:"false"`

	// [Optional] The content type of the object to upload.
	ContentType *string `mandatory:"false"`

	// [Optional] The content language of the object to upload.
	ContentLanguage *string `mandatory:"false"`

	// [Optional] The content encoding of the object to upload.
	ContentEncoding *string `mandatory:"false"`

	// [Optional] Arbitrary string keys and values for the user-defined metadata for the object.
	// Keys must be in "opc-meta-*" format.
	Metadata map[string]string `mandatory:"false"`

	// [Optional] The client request ID for tracing.
	OpcClientRequestID *string `mandatory:"false"`

	// Metadata about the request. This information will not be transmitted to the service, but
	// represents information that the SDK will consume to drive retry behavior.
	RequestMetadata common.RequestMetadata

	// [Optional] The storage tier of the object to upload. If not specified, the storage tier is
	// defaulted to 'Standard'
	StorageTier objectstorage.PutObjectStorageTierEnum `mandatory:"false"`

	// [Optional] Callback API that can be invoked during multiPartUploads
	CallBack UploadCallBack `mandatory:"false"`

	// [Optional] Whether or not this UploadManager supports performing multipart uploads md5 checksum verification. Defaults to False.
	EnableMultipartChecksumVerification *bool `mandatory:"false"`
}

UploadRequest defines the input parameters for UploadFile method

func (UploadRequest) RetryPolicy

func (request UploadRequest) RetryPolicy() *common.RetryPolicy

RetryPolicy implements the OCIRetryableRequest interface. This retrieves the specified retry policy.

type UploadResponse

type UploadResponse struct {

	// Polymorphic response type indicates the response type
	Type UploadResponseType

	// response for putObject API response (single part upload), will be nil if the operation is multiPart upload
	*SinglepartUploadResponse

	// response for commitMultipart API response (multipart upload), will be nil if the operation is singlePart upload
	*MultipartUploadResponse
}

UploadResponse is the response from commitMultipart or the putObject API operation.

func (UploadResponse) IsResumable

func (resp UploadResponse) IsResumable() bool

IsResumable is a function to check is previous failed upload resumable or not

type UploadResponseType

type UploadResponseType string

UploadResponseType with underlying type: string

const (
	MultipartUpload  UploadResponseType = "MULTIPARTUPLOAD"
	SinglepartUpload UploadResponseType = "SINGLEPARTUPLOAD"
)

Set of constants representing the allowable values for VolumeAttachmentLifecycleState

type UploadStreamRequest

type UploadStreamRequest struct {
	UploadRequest

	// The reader of input stream
	StreamReader io.Reader
}

UploadStreamRequest defines the input parameters for UploadFile method