s3manager

package
v1.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 7, 2024 License: Apache-2.0 Imports: 20 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var DefaultUploadConcurrency = 5 // Default number of goroutines
View Source 
var DefaultUploadOptions = &UploadOptions{
	PartSize:          DefaultUploadPartSize,
	Parallel:          DefaultUploadConcurrency,
	Jobs:              3,
	LeavePartsOnError: false,
	S3:                nil,
}

DefaultUploadOptions The default set of options used when opts is nil in Upload().

View Source 
var DefaultUploadPartSize int64 = 1024 * 1024 * 5 // Default part size, 5MB
View Source 
var MaxUploadPartSize int64 = 1024 * 1024 * 1024 * 5 // Max part size, 5GB
View Source 
var MaxUploadParts = 10000 // max number of parts in a multipart upload
View Source 
var MinUploadPartSize int64 = 1024 * 100 // Min part size, 100KB

Functions

This section is empty.

Types

type CloudURL added in v1.2.1 

type CloudURL struct {
	// contains filtered or unexported fields
}

CloudURL describes ks3 url

type CompleteUploadPart added in v1.3.0 

type CompleteUploadPart struct {
	Part      *s3.CompletedPart
	TrunkSize int64
}

type CompleteUploadParts added in v1.3.0 

type CompleteUploadParts []*CompleteUploadPart

CompleteUploadParts completedParts is a wrapper to make parts sortable by their part number, since KS3 required this list to be sent in sorted order.

func (CompleteUploadParts) Len added in v1.3.0 

func (a CompleteUploadParts) Len() int

func (CompleteUploadParts) Less added in v1.3.0 

func (a CompleteUploadParts) Less(i, j int) bool

func (CompleteUploadParts) Swap added in v1.3.0 

func (a CompleteUploadParts) Swap(i, j int)

type FileCounter added in v1.2.2 

type FileCounter struct {
	TotalNum   int64
	SuccessNum int64
	FailNum    int64
}

type FileURL added in v1.2.1 

type FileURL struct {
	// contains filtered or unexported fields
}

FileURL describes file url

func (*FileURL) Init added in v1.2.1 

func (fu *FileURL) Init(urlStr string) error

Init simulate inheritance, and polymorphism

func (*FileURL) IsCloudURL added in v1.2.1 

func (fu *FileURL) IsCloudURL() bool

IsCloudURL simulate inheritance, and polymorphism

func (*FileURL) IsFileURL added in v1.2.1 

func (fu *FileURL) IsFileURL() bool

IsFileURL simulate inheritance, and polymorphism

func (*FileURL) ToString added in v1.2.1 

func (fu *FileURL) ToString() string

ToString simulate inheritance, and polymorphism

type MultiUploadFailure 

type MultiUploadFailure interface {
	awserr.Error

	// UploadID Returns the upload id for the S3 multipart upload that failed.
	UploadID() string
}

type StorageURLer added in v1.2.1 

type StorageURLer interface {
	IsCloudURL() bool
	IsFileURL() bool
	ToString() string
}

StorageURLer is the interface for all url

func StorageURLFromString added in v1.2.1 

func StorageURLFromString(urlStr string) (StorageURLer, error)

StorageURLFromString analysis input url type and build a storage url from the url

type UploadDirInput added in v1.3.0 

type UploadDirInput struct {
	// The path to the folder to be uploaded.
	RootDir string
	// The name of the bucket.
	Bucket string
	// Prefix of the object.
	Prefix string
	// The ACL of the object.
	ACL string
	// The StorageClass of the object.
	StorageClass string
}

type UploadInput 

type UploadInput struct {
	// The canned ACL to apply to the object.
	ACL *string `location:"header" locationName:"x-amz-acl" type:"string"`

	Size int64

	Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"`

	// Specifies caching behavior along the request/reply chain.
	CacheControl *string `location:"header" locationName:"Cache-Control" type:"string"`

	// Specifies presentational information for the object.
	ContentDisposition *string `location:"header" locationName:"Content-Disposition" type:"string"`

	// Specifies what content encodings have been applied to the object and thus
	// what decoding mechanisms must be applied to obtain the media-type referenced
	// by the Content-Type header field.
	ContentEncoding *string `location:"header" locationName:"Content-Encoding" type:"string"`

	// The language the content is in.
	ContentLanguage *string `location:"header" locationName:"Content-Language" type:"string"`

	// A standard MIME type describing the format of the object data.
	ContentType *string `location:"header" locationName:"Content-Type" type:"string"`

	// The date and time at which the object is no longer cacheable.
	Expires *time.Time `location:"header" locationName:"Expires" type:"timestamp" timestampFormat:"rfc822"`

	// Gives the grantee READ, READ_ACP, and WRITE_ACP permissions on the object.
	GrantFullControl *string `location:"header" locationName:"x-amz-grant-full-control" type:"string"`

	// Allows grantee to read the object data and its metadata.
	GrantRead *string `location:"header" locationName:"x-amz-grant-read" type:"string"`

	// Allows grantee to read the object ACL.
	GrantReadACP *string `location:"header" locationName:"x-amz-grant-read-acp" type:"string"`

	// Allows grantee to write the ACL for the applicable object.
	GrantWriteACP *string `location:"header" locationName:"x-amz-grant-write-acp" type:"string"`

	Key *string `location:"uri" locationName:"Key" type:"string" required:"true"`

	// A map of metadata to store with the object in S3.
	Metadata map[string]*string `location:"headers" locationName:"x-amz-meta-" type:"map"`

	// Confirms that the requester knows that she or he will be charged for the
	// request. Bucket owners need not specify this parameter in their requests.
	// Documentation on downloading objects from requester pays buckets can be found
	// at http://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html
	RequestPayer *string `location:"header" locationName:"x-amz-request-payer" type:"string"`

	// Specifies the algorithm to use to when encrypting the object (e.g., AES256,
	// aws:kms).
	SSECustomerAlgorithm *string `location:"header" locationName:"x-amz-server-side-encryption-customer-algorithm" type:"string"`

	// Specifies the customer-provided encryption key for Amazon S3 to use in encrypting
	// data. This value is used to store the object, and then it is discarded; Amazon
	// does not store the encryption key. The key must be appropriate for use with
	// the algorithm specified in the x-amz-server-side​-encryption​-customer-algorithm
	// header.
	SSECustomerKey *string `location:"header" locationName:"x-amz-server-side-encryption-customer-key" type:"string"`

	// Specifies the 128-bit MD5 digest of the encryption key according to RFC 1321.
	// Amazon S3 uses this header for a message integrity check to ensure the encryption
	// key was transmitted without error.
	SSECustomerKeyMD5 *string `location:"header" locationName:"x-amz-server-side-encryption-customer-key-MD5" type:"string"`

	// Specifies the AWS KMS key ID to use for object encryption. All GET and PUT
	// requests for an object protected by AWS KMS will fail if not made via SSL
	// or using SigV4. Documentation on configuring any of the officially supported
	// AWS SDKs and CLI can be found at http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingAWSSDK.html#specify-signature-version
	SSEKMSKeyID *string `location:"header" locationName:"x-amz-server-side-encryption-aws-kms-key-id" type:"string"`

	// The Server-side encryption algorithm used when storing this object in S3
	// (e.g., AES256, aws:kms).
	ServerSideEncryption *string `location:"header" locationName:"x-amz-server-side-encryption" type:"string"`

	// The type of storage to use for the object. Defaults to 'STANDARD'.
	StorageClass *string `location:"header" locationName:"x-amz-storage-class" type:"string"`

	// If the bucket is configured as a website, redirects requests for this object
	// to another object in the same bucket or to an external URL. Amazon S3 stores
	// the value of this header in the object metadata.
	WebsiteRedirectLocation *string `location:"header" locationName:"x-amz-website-redirect-location" type:"string"`

	// The readable body payload to send to S3.
	Body io.Reader
}

UploadInput contains all input for upload requests to Amazon S3.

type UploadOptions 

type UploadOptions struct {
	// The buffer size (in bytes) to use when buffering data into chunks and
	// sending them as parts to KS3. The minimum allowed part size is 5MB, and
	// if this value is set to zero, the DefaultPartSize value will be used.
	PartSize int64

	//Number of concurrent tasks for internal operation of a single file
	Parallel int
	//Number of concurrent tasks in multi-file operation
	Jobs int
	// Setting this value to true will cause the SDK to avoid calling
	// AbortMultipartUpload on a failure, leaving all successfully uploaded
	// parts on S3 for manual recovery.
	//
	// Note that storing parts of an incomplete multipart upload counts towards
	// space usage on S3 and will add additional costs if not cleaned up.
	LeavePartsOnError bool

	//Set whether to upload hidden files
	UploadHidden bool
	//Set whether to upload existing files
	SkipAlreadyFile bool
	// The client to use when uploading to S3. Leave this as nil to use the
	// default S3 client.
	S3 *s3.S3
}

UploadOptions keeps tracks of extra options to pass to an Upload() call.

type UploadOutput 

type UploadOutput struct {
	// The URL where the object was uploaded to.
	Location string

	// The ID for a multipart upload to S3. In the case of an error the error
	// can be cast to the MultiUploadFailure interface to extract the upload ID.
	UploadID string

	ETag string
}

UploadOutput represents a response from the Upload() call.

type Uploader 

type Uploader struct {
	// contains filtered or unexported fields
}

The Uploader structure that calls Upload(). It is safe to call Upload() on this structure for multiple objects and across concurrent goroutines.

func NewUploader 

func NewUploader(opts *UploadOptions) *Uploader

NewUploader creates a new Uploader object to upload data to S3. Pass in an optional opts structure to customize the uploader behavior.

func (*Uploader) Upload 

func (u *Uploader) Upload(input *UploadInput) (*UploadOutput, error)

Upload uploads an object to S3, intelligently buffering large files into smaller chunks and sending them in parallel across multiple goroutines. You can configure the buffer size and concurrency through the opts parameter.

If opts is set to nil, DefaultUploadOptions will be used.

It is safe to call this method for multiple objects and across concurrent goroutines.

func (*Uploader) UploadDir added in v1.2.1 

func (u *Uploader) UploadDir(input *UploadDirInput) error

func (*Uploader) UploadDirWithContext added in v1.2.9 

func (u *Uploader) UploadDirWithContext(ctx aws.Context, input *UploadDirInput) error

func (*Uploader) UploadWithContext added in v1.2.9 

func (u *Uploader) UploadWithContext(ctx aws.Context, input *UploadInput) (*UploadOutput, error)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.