transfer

package
v65.76.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 15, 2024 License: Apache-2.0, UPL-1.0 Imports: 17 Imported by: 3

Documentation

Overview

Package transfer simplifies interaction with the Object Storage service by abstracting away the method used to upload objects. Depending on the configuration parameters, UploadManager may choose to do a single put_object request, or break up the upload into multiple parts and utilize multi-part uploads.

An advantage of using multi-part uploads is the ability to retry individual failed parts, as well as being able to upload parts in parallel to reduce upload time.

To use this package, you must be authorized in an IAM policy. If you're not authorized, talk to an administrator.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type FileUploader 

type FileUploader interface {
	// split file into multiple parts and uploads them to blob storage, then merge
	UploadFileMultiparts(ctx context.Context, request UploadFileRequest) (response UploadResponse, err error)

	// uploads a file to blob storage via PutObject API
	UploadFilePutObject(ctx context.Context, request UploadFileRequest) (response UploadResponse, err error)

	// resume a file upload, use it when UploadFile failed
	ResumeUploadFile(ctx context.Context, uploadID string) (response UploadResponse, err error)
}

FileUploader is an interface to upload a file

type MultiPartUploadPart 

type MultiPartUploadPart struct {
	PartNum    int
	TotalParts int
	Size       int64
	Offset     int64
	Hash       *string
	OpcMD5     *string
	Etag       *string
	Err        error
}

MultiPartUploadPart holds the details of Part that is uploaded

type MultipartUploadResponse 

type MultipartUploadResponse struct {
	objectstorage.CommitMultipartUploadResponse

	// The upload ID for a multipart upload.
	UploadID *string
	// contains filtered or unexported fields
}

MultipartUploadResponse is the response from commitMultipart API operation.

type SinglepartUploadResponse 

type SinglepartUploadResponse struct {
	objectstorage.PutObjectResponse
}

SinglepartUploadResponse is the response from putObject API operation.

type StreamUploader 

type StreamUploader interface {
	// uploads a stream to blob storage
	UploadStream(ctx context.Context, request UploadStreamRequest) (response UploadResponse, err error)
}

StreamUploader is an interface for upload a stream

type UploadCallBack 

type UploadCallBack func(multiPartUploadPart MultiPartUploadPart)

UploadCallBack API that gets invoked after a Part is successuly uploaded

type UploadFileRequest 

type UploadFileRequest struct {
	UploadRequest

	// The path of the file to be uploaded (includs file name)
	FilePath string
}

UploadFileRequest defines the input parameters for UploadFile method

type UploadManager 

type UploadManager struct {
	FileUploader   FileUploader
	StreamUploader StreamUploader
}

UploadManager is the interface that groups the upload methods

func NewUploadManager 

func NewUploadManager() *UploadManager

NewUploadManager return a pointer to UploadManager

func (*UploadManager) ResumeUploadFile 

func (uploadManager *UploadManager) ResumeUploadFile(ctx context.Context, uploadID string) (response UploadResponse, err error)

ResumeUploadFile resumes a multipart file upload.

func (*UploadManager) UploadFile 

func (uploadManager *UploadManager) UploadFile(ctx context.Context, request UploadFileRequest) (response UploadResponse, err error)

UploadFile uploads an object to Object Storage. Depending on the options provided and the size of the object, the object may be uploaded in multiple parts or just an signle object.

func (*UploadManager) UploadStream 

func (uploadManager *UploadManager) UploadStream(ctx context.Context, request UploadStreamRequest) (response UploadResponse, err error)

UploadStream uploads streaming data to Object Storage. If the stream is non-empty, this will always perform a multipart upload, splitting parts based on the part size (10 MiB if none specified). If the stream is empty, this will upload a single empty object to Object Storage. Stream uploads are not currently resumable.

type UploadRequest 

type UploadRequest struct {
	// The top-level namespace used for the request.
	NamespaceName *string `mandatory:"true"`

	// The name of the bucket. Avoid entering confidential information. Example: my-new-bucket1
	BucketName *string `mandatory:"true"`

	// The name of the object. Avoid entering confidential information. Example: test/object1.log
	ObjectName *string `mandatory:"true"`

	// [Optional] Override the default part size of 128 MiB, value is in bytes.
	// The max part size is 50GiB
	PartSize *int64 `mandatory:"false"`

	// [Optional] Whether or not this UploadManager supports performing mulitpart uploads. Defaults to True.
	AllowMultipartUploads *bool `mandatory:"false"`

	// [Optional] Whether or not this UploadManager supports uploading individual parts of a multipart upload in parallel.
	// This setting has no effect on uploads that are performed using a single put_object call. Defaults to True.
	AllowParrallelUploads *bool `mandatory:"false"`

	// The number of go routines for uploading individual parts of a multipart upload.
	// This setting is only used if allow_parallel_uploads is set to True. Defaults to 5.
	// The upper bounds of the number is 10,000.
	NumberOfGoroutines *int `mandatory:"false"`

	// A configured object storage client to use for interacting with the Object Storage service.
	// Default timeout is 60s which includes the time for reading the body.
	// Default timeout doesn't work for big file size and big part size(once upload each part longer than 60s), need to manually update timeout to support big file upload.
	ObjectStorageClient *objectstorage.ObjectStorageClient `mandatory:"false"`

	// [Optional] The entity tag of the object to match.
	IfMatch *string `mandatory:"false"`

	// [Optional] The entity tag of the object to avoid matching. The only valid value is ‘*’,
	// which indicates that the request should fail if the object already exists.
	IfNoneMatch *string `mandatory:"false"`

	// [Optional] The base-64 encoded MD5 hash of the body. This parameter is only used if the object is uploaded in a single part.
	ContentMD5 *string `mandatory:"false"`

	// [Optional] The content type of the object to upload.
	ContentType *string `mandatory:"false"`

	// [Optional] The content language of the object to upload.
	ContentLanguage *string `mandatory:"false"`

	// [Optional] The content encoding of the object to upload.
	ContentEncoding *string `mandatory:"false"`

	// [Optional] Arbitrary string keys and values for the user-defined metadata for the object.
	// Keys must be in "opc-meta-*" format.
	Metadata map[string]string `mandatory:"false"`

	// [Optional] The client request ID for tracing.
	OpcClientRequestID *string `mandatory:"false"`

	// Metadata about the request. This information will not be transmitted to the service, but
	// represents information that the SDK will consume to drive retry behavior.
	RequestMetadata common.RequestMetadata

	// [Optional] The storage tier of the object to upload. If not specified, the storage tier is
	// defaulted to 'Standard'
	StorageTier objectstorage.PutObjectStorageTierEnum `mandatory:"false"`

	// [Optional] Callback API that can be invoked during multiPartUploads
	CallBack UploadCallBack `mandatory:"false"`

	// [Optional] Whether or not this UploadManager supports performing multipart uploads md5 checksum verification. Defaults to False.
	EnableMultipartChecksumVerification *bool `mandatory:"false"`

	// The optional header that specifies "AES256" as the encryption algorithm. For more information, see
	// Using Your Own Keys for Server-Side Encryption (https://docs.cloud.oracle.com/Content/Object/Tasks/usingyourencryptionkeys.htm).
	OpcSseCustomerAlgorithm *string `mandatory:"false"`

	// The optional header that specifies the base64-encoded 256-bit encryption key to use to encrypt or
	// decrypt the data. For more information, see
	// Using Your Own Keys for Server-Side Encryption (https://docs.cloud.oracle.com/Content/Object/Tasks/usingyourencryptionkeys.htm).
	OpcSseCustomerKey *string `mandatory:"false"`

	// The optional header that specifies the base64-encoded SHA256 hash of the encryption key. This
	// value is used to check the integrity of the encryption key. For more information, see
	// Using Your Own Keys for Server-Side Encryption (https://docs.cloud.oracle.com/Content/Object/Tasks/usingyourencryptionkeys.htm).
	OpcSseCustomerKeySha256 *string `mandatory:"false"`

	// The OCID (https://docs.cloud.oracle.com/Content/General/Concepts/identifiers.htm) of a master encryption key used to call the Key
	// Management service to generate a data encryption key or to encrypt or decrypt a data encryption key.
	OpcSseKmsKeyId *string `mandatory:"false"`
}

UploadRequest defines the input parameters for UploadFile method

func (UploadRequest) RetryPolicy 

func (request UploadRequest) RetryPolicy() *common.RetryPolicy

RetryPolicy implements the OCIRetryableRequest interface. This retrieves the specified retry policy.

type UploadResponse 

type UploadResponse struct {

	// Polymorphic response type indicates the response type
	Type UploadResponseType

	// response for putObject API response (single part upload), will be nil if the operation is multiPart upload
	*SinglepartUploadResponse

	// response for commitMultipart API response (multipart upload), will be nil if the operation is singlePart upload
	*MultipartUploadResponse
}

UploadResponse is the response from commitMultipart or the putObject API operation.

func (UploadResponse) IsResumable 

func (resp UploadResponse) IsResumable() bool

IsResumable is a function to check is previous failed upload resumable or not

type UploadResponseType 

type UploadResponseType string

UploadResponseType with underlying type: string 

const (
	MultipartUpload  UploadResponseType = "MULTIPARTUPLOAD"
	SinglepartUpload UploadResponseType = "SINGLEPARTUPLOAD"
)

Set of constants representing the allowable values for VolumeAttachmentLifecycleState

type UploadStreamRequest 

type UploadStreamRequest struct {
	UploadRequest

	// The reader of input stream
	StreamReader io.Reader
}

UploadStreamRequest defines the input parameters for UploadFile method

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.