s3manager

package
v1.0.13 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 3, 2021 License: Apache-2.0 Imports: 23 Imported by: 0

Documentation

Overview

Package s3manager provides utilities to upload and download objects from S3 concurrently. Helpful for when working with large objects.

Index

Examples

Constants

View Source 
const (
	// DefaultBatchSize is the batch size we initialize when constructing a batch delete client.
	// This value is used when calling DeleteObjects. This represents how many objects to delete
	// per DeleteObjects call.
	DefaultBatchSize = 100
)
View Source 
const (
	// ErrDeleteBatchFailCode represents an error code which will be returned
	// only when DeleteObjects.Errors has an error that does not contain a code.
	ErrDeleteBatchFailCode = "DeleteBatchError"
)

Variables

View Source 
var DefaultDownloadConcurrency = 5

The default number of goroutines to spin up when using Download().

View Source 
var DefaultDownloadOptions = &DownloadOptions{
	PartSize:    DefaultDownloadPartSize,
	Concurrency: DefaultDownloadConcurrency,
}

The default set of options used when opts is nil in Download().

View Source 
var DefaultDownloadPartSize int64 = 1024 * 1024 * 5

The default range of bytes to get at a time when using Download().

View Source 
var DefaultUploadConcurrency = 5

The default number of goroutines to spin up when using Upload().

View Source 
var DefaultUploadOptions = &UploadOptions{
	PartSize:          DefaultUploadPartSize,
	Concurrency:       DefaultUploadConcurrency,
	LeavePartsOnError: false,
	S3:                nil,
}

The default set of options used when opts is nil in Upload().

View Source 
var DefaultUploadPartSize = MinUploadPartSize

The default part size to buffer chunks of a payload into.

View Source 
var MaxUploadParts = 10000

The maximum allowed number of parts in a multi-part upload on Amazon S3.

View Source 
var MinUploadPartSize int64 = 1024 * 1024 * 5

The minimum allowed part size when uploading a part to Amazon S3.

Functions

func GetBucketRegion added in v1.0.13 

func GetBucketRegion(ctx aws.Context, c client.ConfigProvider, bucket, regionHint string, opts ...request.Option) (string, error)

GetBucketRegion will attempt to get the region for a bucket using the regionHint to determine which AWS partition to perform the query on.

The request will not be signed, and will not use your AWS credentials.

A "NotFound" error code will be returned if the bucket does not exist in the AWS partition the regionHint belongs to. If the regionHint parameter is an empty string GetBucketRegion will fallback to the ConfigProvider's region config. If the regionHint is empty, and the ConfigProvider does not have a region value, an error will be returned..

For example to get the region of a bucket which exists in "eu-central-1" you could provide a region hint of "us-west-2". 

sess := session.Must(session.NewSession())

bucket := "my-bucket"
region, err := s3manager.GetBucketRegion(ctx, sess, bucket, "us-west-2")
if err != nil {
    if aerr, ok := err.(awserr.Error); ok && aerr.Code() == "NotFound" {
         fmt.Fprintf(os.Stderr, "unable to find bucket %s's region not found\n", bucket)
    }
    return err
}
fmt.Printf("Bucket %s is in %s region\n", bucket, region)

By default the request will be made to the Amazon S3 endpoint using the Path style addressing. 

s3.us-west-2.amazonaws.com/bucketname

This is not compatible with Amazon S3's FIPS endpoints. To override this behavior to use Virtual Host style addressing, provide a functional option that will set the Request's Config.S3ForcePathStyle to aws.Bool(false). 

region, err := s3manager.GetBucketRegion(ctx, sess, "bucketname", "us-west-2", func(r *request.Request) {
    r.S3ForcePathStyle = aws.Bool(false)
})

To configure the GetBucketRegion to make a request via the Amazon S3 FIPS endpoints directly when a FIPS region name is not available, (e.g. fips-us-gov-west-1) set the Config.Endpoint on the Session, or client the utility is called with. The hint region will be ignored if an endpoint URL is configured on the session or client. 

sess, err := session.NewSession(&aws.Config{
    Endpoint: aws.String("https://s3-fips.us-west-2.amazonaws.com"),
})

region, err := s3manager.GetBucketRegion(context.Background(), sess, "bucketname", "")

func GetBucketRegionWithClient added in v1.0.13 

func GetBucketRegionWithClient(ctx aws.Context, svc s3iface.S3API, bucket string, opts ...request.Option) (string, error)

GetBucketRegionWithClient is the same as GetBucketRegion with the exception that it takes a S3 service client instead of a Session. The regionHint is derived from the region the S3 service client was created in.

By default the request will be made to the Amazon S3 endpoint using the Path style addressing. 

s3.us-west-2.amazonaws.com/bucketname

This is not compatible with Amazon S3's FIPS endpoints. To override this behavior to use Virtual Host style addressing, provide a functional option that will set the Request's Config.S3ForcePathStyle to aws.Bool(false). 

region, err := s3manager.GetBucketRegionWithClient(ctx, client, "bucketname", func(r *request.Request) {
    r.S3ForcePathStyle = aws.Bool(false)
})

To configure the GetBucketRegion to make a request via the Amazon S3 FIPS endpoints directly when a FIPS region name is not available, (e.g. fips-us-gov-west-1) set the Config.Endpoint on the Session, or client the utility is called with. The hint region will be ignored if an endpoint URL is configured on the session or client. 

region, err := s3manager.GetBucketRegionWithClient(context.Background(),
s3.New(sess, &aws.Config{
    Endpoint: aws.String("https://s3-fips.us-west-2.amazonaws.com"),
}),
"bucketname")

See GetBucketRegion for more information.

func NewBatchError added in v1.0.13 

func NewBatchError(code, message string, err []Error) awserr.Error

NewBatchError will return a BatchError that satisfies the awserr.Error interface.

Types

type BatchDelete added in v1.0.13 

type BatchDelete struct {
	Client    s3iface.S3API
	BatchSize int
}

BatchDelete will use the s3 package's service client to perform a batch delete.

func NewBatchDelete added in v1.0.13 

func NewBatchDelete(c client.ConfigProvider, options ...func(*BatchDelete)) *BatchDelete

NewBatchDelete will return a new delete client that can delete a batched amount of objects.

Example: 

batcher := s3manager.NewBatchDelete(sess, size)

objects := []BatchDeleteObject{
	{
		Object:	&s3.DeleteObjectInput {
			Key: aws.String("key"),
			Bucket: aws.String("bucket"),
		},
	},
}

if err := batcher.Delete(aws.BackgroundContext(), &s3manager.DeleteObjectsIterator{
	Objects: objects,
}); err != nil {
	return err
}

func NewBatchDeleteWithClient added in v1.0.13 

func NewBatchDeleteWithClient(client s3iface.S3API, options ...func(*BatchDelete)) *BatchDelete

NewBatchDeleteWithClient will return a new delete client that can delete a batched amount of objects.

Example: 

batcher := s3manager.NewBatchDeleteWithClient(client, size)

objects := []BatchDeleteObject{
	{
		Object:	&s3.DeleteObjectInput {
			Key: aws.String("key"),
			Bucket: aws.String("bucket"),
		},
	},
}

if err := batcher.Delete(aws.BackgroundContext(), &s3manager.DeleteObjectsIterator{
	Objects: objects,
}); err != nil {
	return err
}

func (*BatchDelete) Delete added in v1.0.13 

func (d *BatchDelete) Delete(ctx aws.Context, iter BatchDeleteIterator) error

Delete will use the iterator to queue up objects that need to be deleted. Once the batch size is met, this will call the deleteBatch function.

type BatchDeleteIterator added in v1.0.13 

type BatchDeleteIterator interface {
	Next() bool
	Err() error
	DeleteObject() BatchDeleteObject
}

BatchDeleteIterator is an interface that uses the scanner pattern to iterate through what needs to be deleted.

func NewDeleteListIterator added in v1.0.13 

func NewDeleteListIterator(svc s3iface.S3API, input *s3.ListObjectsInput, opts ...func(*DeleteListIterator)) BatchDeleteIterator

NewDeleteListIterator will return a new DeleteListIterator.

type BatchDeleteObject added in v1.0.13 

type BatchDeleteObject struct {
	Object *s3.DeleteObjectInput
	// After will run after each iteration during the batch process. This function will
	// be executed whether or not the request was successful.
	After func() error
}

BatchDeleteObject is a wrapper object for calling the batch delete operation.

type BatchDownloadIterator added in v1.0.13 

type BatchDownloadIterator interface {
	Next() bool
	Err() error
	DownloadObject() BatchDownloadObject
}

BatchDownloadIterator is an interface that uses the scanner pattern to iterate through a series of objects to be downloaded.

type BatchDownloadObject added in v1.0.13 

type BatchDownloadObject struct {
	Object *s3.GetObjectInput
	Writer io.WriterAt
	// After will run after each iteration during the batch process. This function will
	// be executed whether or not the request was successful.
	After func() error
}

BatchDownloadObject contains all necessary information to run a batch operation once.

type BatchError added in v1.0.13 

type BatchError struct {
	Errors Errors
	// contains filtered or unexported fields
}

BatchError will contain the key and bucket of the object that failed to either upload or download.

func (*BatchError) Code added in v1.0.13 

func (err *BatchError) Code() string

Code will return the code associated with the batch error.

func (*BatchError) Error added in v1.0.13 

func (err *BatchError) Error() string

func (*BatchError) Message added in v1.0.13 

func (err *BatchError) Message() string

Message will return the message associated with the batch error.

func (*BatchError) OrigErr added in v1.0.13 

func (err *BatchError) OrigErr() error

OrigErr will return the original error. Which, in this case, will always be nil for batched operations.

type BatchUploadIterator added in v1.0.13 

type BatchUploadIterator interface {
	Next() bool
	Err() error
	UploadObject() BatchUploadObject
}

BatchUploadIterator is an interface that uses the scanner pattern to iterate through what needs to be uploaded.

type BatchUploadObject added in v1.0.13 

type BatchUploadObject struct {
	Object *UploadInput
	// After will run after each iteration during the batch process. This function will
	// be executed whether or not the request was successful.
	After func() error
}

BatchUploadObject contains all necessary information to run a batch operation once.

type BufferedReadSeeker added in v1.0.13 

type BufferedReadSeeker struct {
	// contains filtered or unexported fields
}

BufferedReadSeeker is buffered io.ReadSeeker

func NewBufferedReadSeeker added in v1.0.13 

func NewBufferedReadSeeker(r io.ReadSeeker, b []byte) *BufferedReadSeeker

NewBufferedReadSeeker returns a new BufferedReadSeeker if len(b) == 0 then the buffer will be initialized to 64 KiB.

func (*BufferedReadSeeker) Read added in v1.0.13 

func (b *BufferedReadSeeker) Read(p []byte) (n int, err error)

Read will read up len(p) bytes into p and will return the number of bytes read and any error that occurred. If the len(p) > the buffer size then a single read request will be issued to the underlying io.ReadSeeker for len(p) bytes. A Read request will at most perform a single Read to the underlying io.ReadSeeker, and may return < len(p) if serviced from the buffer.

func (*BufferedReadSeeker) ReadAt added in v1.0.13 

func (b *BufferedReadSeeker) ReadAt(p []byte, off int64) (int, error)

ReadAt will read up to len(p) bytes at the given file offset. This will result in the buffer being cleared.

func (*BufferedReadSeeker) Seek added in v1.0.13 

func (b *BufferedReadSeeker) Seek(offset int64, whence int) (int64, error)

Seek will position then underlying io.ReadSeeker to the given offset and will clear the buffer.

type BufferedReadSeekerWriteTo added in v1.0.13 

type BufferedReadSeekerWriteTo struct {
	*BufferedReadSeeker
}

BufferedReadSeekerWriteTo wraps a BufferedReadSeeker with an io.WriteAt implementation.

func (*BufferedReadSeekerWriteTo) WriteTo added in v1.0.13 

func (b *BufferedReadSeekerWriteTo) WriteTo(writer io.Writer) (int64, error)

WriteTo writes to the given io.Writer from BufferedReadSeeker until there's no more data to write or an error occurs. Returns the number of bytes written and any error encountered during the write.

type BufferedReadSeekerWriteToPool added in v1.0.13 

type BufferedReadSeekerWriteToPool struct {
	// contains filtered or unexported fields
}

BufferedReadSeekerWriteToPool uses a sync.Pool to create and reuse []byte slices for buffering parts in memory

func NewBufferedReadSeekerWriteToPool added in v1.0.13 

func NewBufferedReadSeekerWriteToPool(size int) *BufferedReadSeekerWriteToPool

NewBufferedReadSeekerWriteToPool will return a new BufferedReadSeekerWriteToPool that will create a pool of reusable buffers . If size is less then < 64 KiB then the buffer will default to 64 KiB. Reason: io.Copy from writers or readers that don't support io.WriteTo or io.ReadFrom respectively will default to copying 32 KiB.

func (*BufferedReadSeekerWriteToPool) GetWriteTo added in v1.0.13 

func (p *BufferedReadSeekerWriteToPool) GetWriteTo(seeker io.ReadSeeker) (r ReadSeekerWriteTo, cleanup func())

GetWriteTo will wrap the provided io.ReadSeeker with a BufferedReadSeekerWriteTo. The provided cleanup must be called after operations have been completed on the returned io.ReadSeekerWriteTo in order to signal the return of resources to the pool.

type DeleteListIterator added in v1.0.13 

type DeleteListIterator struct {
	Bucket    *string
	Paginator request.Pagination
	// contains filtered or unexported fields
}

DeleteListIterator is an alternative iterator for the BatchDelete client. This will iterate through a list of objects and delete the objects.

Example: 

iter := &s3manager.DeleteListIterator{
	Client: svc,
	Input: &s3.ListObjectsInput{
		Bucket:  aws.String("bucket"),
		MaxKeys: aws.Int64(5),
	},
	Paginator: request.Pagination{
		NewRequest: func() (*request.Request, error) {
			var inCpy *ListObjectsInput
			if input != nil {
				tmp := *input
				inCpy = &tmp
			}
			req, _ := c.ListObjectsRequest(inCpy)
			return req, nil
		},
	},
}

batcher := s3manager.NewBatchDeleteWithClient(svc)
if err := batcher.Delete(aws.BackgroundContext(), iter); err != nil {
	return err
}

func (*DeleteListIterator) DeleteObject added in v1.0.13 

func (iter *DeleteListIterator) DeleteObject() BatchDeleteObject

DeleteObject will return the current object to be deleted.

func (*DeleteListIterator) Err added in v1.0.13 

func (iter *DeleteListIterator) Err() error

Err will return the last known error from Next.

func (*DeleteListIterator) Next added in v1.0.13 

func (iter *DeleteListIterator) Next() bool

Next will use the S3API client to iterate through a list of objects.

type DeleteObjectsIterator added in v1.0.13 

type DeleteObjectsIterator struct {
	Objects []BatchDeleteObject
	// contains filtered or unexported fields
}

DeleteObjectsIterator is an interface that uses the scanner pattern to iterate through a series of objects to be deleted.

func (*DeleteObjectsIterator) DeleteObject added in v1.0.13 

func (iter *DeleteObjectsIterator) DeleteObject() BatchDeleteObject

DeleteObject will return the BatchDeleteObject at the current batched index.

func (*DeleteObjectsIterator) Err added in v1.0.13 

func (iter *DeleteObjectsIterator) Err() error

Err will return an error. Since this is just used to satisfy the BatchDeleteIterator interface this will only return nil.

func (*DeleteObjectsIterator) Next added in v1.0.13 

func (iter *DeleteObjectsIterator) Next() bool

Next will increment the default iterator's index and ensure that there is another object to iterator to.

type DownloadObjectsIterator added in v1.0.13 

type DownloadObjectsIterator struct {
	Objects []BatchDownloadObject
	// contains filtered or unexported fields
}

DownloadObjectsIterator implements the BatchDownloadIterator interface and allows for batched download of objects.

func (*DownloadObjectsIterator) DownloadObject added in v1.0.13 

func (batcher *DownloadObjectsIterator) DownloadObject() BatchDownloadObject

DownloadObject will return the BatchDownloadObject at the current batched index.

func (*DownloadObjectsIterator) Err added in v1.0.13 

func (batcher *DownloadObjectsIterator) Err() error

Err will return an error. Since this is just used to satisfy the BatchDeleteIterator interface this will only return nil.

func (*DownloadObjectsIterator) Next added in v1.0.13 

func (batcher *DownloadObjectsIterator) Next() bool

Next will increment the default iterator's index and ensure that there is another object to iterator to.

type DownloadOptions 

type DownloadOptions struct {
	// The buffer size (in bytes) to use when buffering data into chunks and
	// sending them as parts to S3. The minimum allowed part size is 5MB, and
	// if this value is set to zero, the DefaultPartSize value will be used.
	PartSize int64

	// The number of goroutines to spin up in parallel when sending parts.
	// If this is set to zero, the DefaultConcurrency value will be used.
	Concurrency int

	// An S3 client to use when performing downloads. Leave this as nil to use
	// a default client.
	S3 *s3.S3
}

DownloadOptions keeps tracks of extra options to pass to an Download() call.

type Downloader 

type Downloader struct {
	// contains filtered or unexported fields
}

The Downloader structure that calls Download(). It is safe to call Download() on this structure for multiple objects and across concurrent goroutines.

func NewDownloader 

func NewDownloader(opts *DownloadOptions) *Downloader

NewDownloader creates a new Downloader structure that downloads an object from S3 in concurrent chunks. Pass in an optional DownloadOptions struct to customize the downloader behavior.

func (*Downloader) Download 

func (d *Downloader) Download(w io.WriterAt, input *s3.GetObjectInput) (n int64, err error)

Download downloads an object in S3 and writes the payload into w using concurrent GET requests.

It is safe to call this method for multiple objects and across concurrent goroutines.

type Error added in v1.0.13 

type Error struct {
	OrigErr error
	Bucket  *string
	Key     *string
}

Error will contain the original error, bucket, and key of the operation that failed during batch operations.

func (*Error) Error added in v1.0.13 

func (err *Error) Error() string

type Errors added in v1.0.13 

type Errors []Error

Errors is a typed alias for a slice of errors to satisfy the error interface.

func (Errors) Error added in v1.0.13 

func (errs Errors) Error() string

type MultiUploadFailure 

type MultiUploadFailure interface {
	awserr.Error

	// Returns the upload id for the S3 multipart upload that failed.
	UploadID() string
}

A MultiUploadFailure wraps a failed S3 multipart upload. An error returned will satisfy this interface when a multi part upload failed to upload all chucks to S3. In the case of a failure the UploadID is needed to operate on the chunks, if any, which were uploaded.

Example: 

u := s3manager.NewUploader(opts)
output, err := u.upload(input)
if err != nil {
    if multierr, ok := err.(MultiUploadFailure); ok {
        // Process error and its associated uploadID
        fmt.Println("Error:", multierr.Code(), multierr.Message(), multierr.UploadID())
    } else {
        // Process error generically
        fmt.Println("Error:", err.Error())
    }
}

type PooledBufferedReadFromProvider added in v1.0.13 

type PooledBufferedReadFromProvider struct {
	// contains filtered or unexported fields
}

PooledBufferedReadFromProvider is a WriterReadFromProvider that uses a sync.Pool to manage allocation and reuse of *bufio.Writer structures.

func NewPooledBufferedWriterReadFromProvider added in v1.0.13 

func NewPooledBufferedWriterReadFromProvider(size int) *PooledBufferedReadFromProvider

NewPooledBufferedWriterReadFromProvider returns a new PooledBufferedReadFromProvider Size is used to control the size of the underlying *bufio.Writer created for calls to GetReadFrom.

func (*PooledBufferedReadFromProvider) GetReadFrom added in v1.0.13 

func (p *PooledBufferedReadFromProvider) GetReadFrom(writer io.Writer) (r WriterReadFrom, cleanup func())

GetReadFrom takes an io.Writer and wraps it with a type which satisfies the WriterReadFrom interface/ Additionally a cleanup function is provided which must be called after usage of the WriterReadFrom has been completed in order to allow the reuse of the *bufio.Writer

type ReadSeekerWriteTo added in v1.0.13 

type ReadSeekerWriteTo interface {
	io.ReadSeeker
	io.WriterTo
}

ReadSeekerWriteTo defines an interface implementing io.WriteTo and io.ReadSeeker

type ReadSeekerWriteToProvider added in v1.0.13 

type ReadSeekerWriteToProvider interface {
	GetWriteTo(seeker io.ReadSeeker) (r ReadSeekerWriteTo, cleanup func())
}

ReadSeekerWriteToProvider provides an implementation of io.WriteTo for an io.ReadSeeker

type UploadInput 

type UploadInput struct {

	// The canned ACL to apply to the object. For more information, see Canned ACL
	// (https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#CannedACL).
	//
	// This action is not supported by Amazon S3 on Outposts.
	ACL *string `location:"header" locationName:"x-amz-acl" type:"string" enum:"ObjectCannedACL"`

	// The readable body payload to send to S3.
	Body io.Reader

	// The bucket name to which the PUT action was initiated.
	//
	// When using this action with an access point, you must direct requests to
	// the access point hostname. The access point hostname takes the form AccessPointName-AccountId.s3-accesspoint.Region.amazonaws.com.
	// When using this action with an access point through the AWS SDKs, you provide
	// the access point ARN in place of the bucket name. For more information about
	// access point ARNs, see Using access points (https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-access-points.html)
	// in the Amazon S3 User Guide.
	//
	// When using this action with Amazon S3 on Outposts, you must direct requests
	// to the S3 on Outposts hostname. The S3 on Outposts hostname takes the form
	// AccessPointName-AccountId.outpostID.s3-outposts.Region.amazonaws.com. When
	// using this action using S3 on Outposts through the AWS SDKs, you provide
	// the Outposts bucket ARN in place of the bucket name. For more information
	// about S3 on Outposts ARNs, see Using S3 on Outposts (https://docs.aws.amazon.com/AmazonS3/latest/userguide/S3onOutposts.html)
	// in the Amazon S3 User Guide.
	//
	// Bucket is a required field
	Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"`

	// Specifies whether Amazon S3 should use an S3 Bucket Key for object encryption
	// with server-side encryption using AWS KMS (SSE-KMS). Setting this header
	// to true causes Amazon S3 to use an S3 Bucket Key for object encryption with
	// SSE-KMS.
	//
	// Specifying this header with a PUT action doesn’t affect bucket-level settings
	// for S3 Bucket Key.
	BucketKeyEnabled *bool `location:"header" locationName:"x-amz-server-side-encryption-bucket-key-enabled" type:"boolean"`

	// Can be used to specify caching behavior along the request/reply chain. For
	// more information, see http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9
	// (http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9).
	CacheControl *string `location:"header" locationName:"Cache-Control" type:"string"`

	// Specifies presentational information for the object. For more information,
	// see http://www.w3.org/Protocols/rfc2616/rfc2616-sec19.html#sec19.5.1 (http://www.w3.org/Protocols/rfc2616/rfc2616-sec19.html#sec19.5.1).
	ContentDisposition *string `location:"header" locationName:"Content-Disposition" type:"string"`

	// Specifies what content encodings have been applied to the object and thus
	// what decoding mechanisms must be applied to obtain the media-type referenced
	// by the Content-Type header field. For more information, see http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.11
	// (http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.11).
	ContentEncoding *string `location:"header" locationName:"Content-Encoding" type:"string"`

	// The language the content is in.
	ContentLanguage *string `location:"header" locationName:"Content-Language" type:"string"`

	// The base64-encoded 128-bit MD5 digest of the message (without the headers)
	// according to RFC 1864. This header can be used as a message integrity check
	// to verify that the data is the same data that was originally sent. Although
	// it is optional, we recommend using the Content-MD5 mechanism as an end-to-end
	// integrity check. For more information about REST request authentication,
	// see REST Authentication (https://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html).
	ContentMD5 *string `location:"header" locationName:"Content-MD5" type:"string"`

	// A standard MIME type describing the format of the contents. For more information,
	// see http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.17 (http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.17).
	ContentType *string `location:"header" locationName:"Content-Type" type:"string"`

	// The account ID of the expected bucket owner. If the bucket is owned by a
	// different account, the request will fail with an HTTP 403 (Access Denied)
	// error.
	ExpectedBucketOwner *string `location:"header" locationName:"x-amz-expected-bucket-owner" type:"string"`

	// The date and time at which the object is no longer cacheable. For more information,
	// see http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.21 (http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.21).
	Expires *time.Time `location:"header" locationName:"Expires" type:"timestamp"`

	// Gives the grantee READ, READ_ACP, and WRITE_ACP permissions on the object.
	//
	// This action is not supported by Amazon S3 on Outposts.
	GrantFullControl *string `location:"header" locationName:"x-amz-grant-full-control" type:"string"`

	// Allows grantee to read the object data and its metadata.
	//
	// This action is not supported by Amazon S3 on Outposts.
	GrantRead *string `location:"header" locationName:"x-amz-grant-read" type:"string"`

	// Allows grantee to read the object ACL.
	//
	// This action is not supported by Amazon S3 on Outposts.
	GrantReadACP *string `location:"header" locationName:"x-amz-grant-read-acp" type:"string"`

	// Allows grantee to write the ACL for the applicable object.
	//
	// This action is not supported by Amazon S3 on Outposts.
	GrantWriteACP *string `location:"header" locationName:"x-amz-grant-write-acp" type:"string"`

	// Object key for which the PUT action was initiated.
	//
	// Key is a required field
	Key *string `location:"uri" locationName:"Key" min:"1" type:"string" required:"true"`

	// A map of metadata to store with the object in S3.
	Metadata map[string]*string `location:"headers" locationName:"x-amz-meta-" type:"map"`

	// Specifies whether a legal hold will be applied to this object. For more information
	// about S3 Object Lock, see Object Lock (https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock.html).
	ObjectLockLegalHoldStatus *string `location:"header" locationName:"x-amz-object-lock-legal-hold" type:"string" enum:"ObjectLockLegalHoldStatus"`

	// The Object Lock mode that you want to apply to this object.
	ObjectLockMode *string `location:"header" locationName:"x-amz-object-lock-mode" type:"string" enum:"ObjectLockMode"`

	// The date and time when you want this object's Object Lock to expire. Must
	// be formatted as a timestamp parameter.
	ObjectLockRetainUntilDate *time.Time `location:"header" locationName:"x-amz-object-lock-retain-until-date" type:"timestamp" timestampFormat:"iso8601"`

	// Confirms that the requester knows that they will be charged for the request.
	// Bucket owners need not specify this parameter in their requests. For information
	// about downloading objects from requester pays buckets, see Downloading Objects
	// in Requestor Pays Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html)
	// in the Amazon S3 User Guide.
	RequestPayer *string `location:"header" locationName:"x-amz-request-payer" type:"string" enum:"RequestPayer"`

	// Specifies the algorithm to use to when encrypting the object (for example,
	// AES256).
	SSECustomerAlgorithm *string `location:"header" locationName:"x-amz-server-side-encryption-customer-algorithm" type:"string"`

	// Specifies the customer-provided encryption key for Amazon S3 to use in encrypting
	// data. This value is used to store the object and then it is discarded; Amazon
	// S3 does not store the encryption key. The key must be appropriate for use
	// with the algorithm specified in the x-amz-server-side-encryption-customer-algorithm
	// header.
	SSECustomerKey *string `marshal-as:"blob" location:"header" locationName:"x-amz-server-side-encryption-customer-key" type:"string" sensitive:"true"`

	// Specifies the 128-bit MD5 digest of the encryption key according to RFC 1321.
	// Amazon S3 uses this header for a message integrity check to ensure that the
	// encryption key was transmitted without error.
	SSECustomerKeyMD5 *string `location:"header" locationName:"x-amz-server-side-encryption-customer-key-MD5" type:"string"`

	// Specifies the AWS KMS Encryption Context to use for object encryption. The
	// value of this header is a base64-encoded UTF-8 string holding JSON with the
	// encryption context key-value pairs.
	SSEKMSEncryptionContext *string `location:"header" locationName:"x-amz-server-side-encryption-context" type:"string" sensitive:"true"`

	// If x-amz-server-side-encryption is present and has the value of aws:kms,
	// this header specifies the ID of the AWS Key Management Service (AWS KMS)
	// symmetrical customer managed customer master key (CMK) that was used for
	// the object. If you specify x-amz-server-side-encryption:aws:kms, but do not
	// providex-amz-server-side-encryption-aws-kms-key-id, Amazon S3 uses the AWS
	// managed CMK in AWS to protect the data. If the KMS key does not exist in
	// the same account issuing the command, you must use the full ARN and not just
	// the ID.
	SSEKMSKeyId *string `location:"header" locationName:"x-amz-server-side-encryption-aws-kms-key-id" type:"string" sensitive:"true"`

	// The server-side encryption algorithm used when storing this object in Amazon
	// S3 (for example, AES256, aws:kms).
	ServerSideEncryption *string `location:"header" locationName:"x-amz-server-side-encryption" type:"string" enum:"ServerSideEncryption"`

	// By default, Amazon S3 uses the STANDARD Storage Class to store newly created
	// objects. The STANDARD storage class provides high durability and high availability.
	// Depending on performance needs, you can specify a different Storage Class.
	// Amazon S3 on Outposts only uses the OUTPOSTS Storage Class. For more information,
	// see Storage Classes (https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html)
	// in the Amazon S3 User Guide.
	StorageClass *string `location:"header" locationName:"x-amz-storage-class" type:"string" enum:"StorageClass"`

	// The tag-set for the object. The tag-set must be encoded as URL Query parameters.
	// (For example, "Key1=Value1")
	Tagging *string `location:"header" locationName:"x-amz-tagging" type:"string"`

	// If the bucket is configured as a website, redirects requests for this object
	// to another object in the same bucket or to an external URL. Amazon S3 stores
	// the value of this header in the object metadata. For information about object
	// metadata, see Object Key and Metadata (https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html).
	//
	// In the following example, the request header sets the redirect to an object
	// (anotherPage.html) in the same bucket:
	//
	// x-amz-website-redirect-location: /anotherPage.html
	//
	// In the following example, the request header sets the object redirect to
	// another website:
	//
	// x-amz-website-redirect-location: http://www.example.com/
	//
	// For more information about website hosting in Amazon S3, see Hosting Websites
	// on Amazon S3 (https://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosting.html)
	// and How to Configure Website Page Redirects (https://docs.aws.amazon.com/AmazonS3/latest/dev/how-to-page-redirect.html).
	WebsiteRedirectLocation *string `location:"header" locationName:"x-amz-website-redirect-location" type:"string"`
	// contains filtered or unexported fields
}

UploadInput provides the input parameters for uploading a stream or buffer to an object in an Amazon S3 bucket. This type is similar to the s3 package's PutObjectInput with the exception that the Body member is an io.Reader instead of an io.ReadSeeker.

type UploadObjectsIterator added in v1.0.13 

type UploadObjectsIterator struct {
	Objects []BatchUploadObject
	// contains filtered or unexported fields
}

UploadObjectsIterator implements the BatchUploadIterator interface and allows for batched upload of objects.

func (*UploadObjectsIterator) Err added in v1.0.13 

func (batcher *UploadObjectsIterator) Err() error

Err will return an error. Since this is just used to satisfy the BatchUploadIterator interface this will only return nil.

func (*UploadObjectsIterator) Next added in v1.0.13 

func (batcher *UploadObjectsIterator) Next() bool

Next will increment the default iterator's index and ensure that there is another object to iterator to.

func (*UploadObjectsIterator) UploadObject added in v1.0.13 

func (batcher *UploadObjectsIterator) UploadObject() BatchUploadObject

UploadObject will return the BatchUploadObject at the current batched index.

type UploadOptions 

type UploadOptions struct {
	// The buffer size (in bytes) to use when buffering data into chunks and
	// sending them as parts to S3. The minimum allowed part size is 5MB, and
	// if this value is set to zero, the DefaultPartSize value will be used.
	PartSize int64

	// The number of goroutines to spin up in parallel when sending parts.
	// If this is set to zero, the DefaultConcurrency value will be used.
	Concurrency int

	// Setting this value to true will cause the SDK to avoid calling
	// AbortMultipartUpload on a failure, leaving all successfully uploaded
	// parts on S3 for manual recovery.
	//
	// Note that storing parts of an incomplete multipart upload counts towards
	// space usage on S3 and will add additional costs if not cleaned up.
	LeavePartsOnError bool

	// The client to use when uploading to S3. Leave this as nil to use the
	// default S3 client.
	S3 *s3.S3
}

UploadOptions keeps tracks of extra options to pass to an Upload() call.

type UploadOutput 

type UploadOutput struct {
	// The URL where the object was uploaded to.
	Location string

	// The ID for a multipart upload to S3. In the case of an error the error
	// can be cast to the MultiUploadFailure interface to extract the upload ID.
	UploadID string
}

UploadOutput represents a response from the Upload() call.

type Uploader 

type Uploader struct {
	// contains filtered or unexported fields
}

The Uploader structure that calls Upload(). It is safe to call Upload() on this structure for multiple objects and across concurrent goroutines.

func NewUploader 

func NewUploader(opts *UploadOptions) *Uploader

NewUploader creates a new Uploader object to upload data to S3. Pass in an optional opts structure to customize the uploader behavior.

Example (OverrideReadSeekerProvider)

ExampleNewUploader_overrideReadSeekerProvider gives an example on a custom ReadSeekerWriteToProvider can be provided to Uploader to define how parts will be buffered in memory. 

package main

import (
	"bytes"
	"fmt"

	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/s3/s3manager"
)

func main() {
	sess := session.Must(session.NewSession())

	uploader := s3manager.NewUploader(sess, func(u *s3manager.Uploader) {
		// Define a strategy that will buffer 25 MiB in memory
		u.BufferProvider = s3manager.NewBufferedReadSeekerWriteToPool(25 * 1024 * 1024)
	})

	_, err := uploader.Upload(&s3manager.UploadInput{
		Bucket: aws.String("examplebucket"),
		Key:    aws.String("largeobject"),
		Body:   bytes.NewReader([]byte("large_multi_part_upload")),
	})
	if err != nil {
		fmt.Println(err.Error())
	}
}

Output:
Example (OverrideTransport)

ExampleNewUploader_overrideTransport gives an example on how to override the default HTTP transport. This can be used to tune timeouts such as response headers, or write / read buffer usage when writing or reading respectively from the net/http transport. 

package main

import (
	"bytes"
	"fmt"
	"net/http"
	"time"

	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/s3/s3manager"
)

func main() {
	// Create Transport
	tr := &http.Transport{
		ResponseHeaderTimeout: 1 * time.Second,
		// WriteBufferSize: 1024*1024 // Go 1.13
		// ReadBufferSize: 1024*1024 // Go 1.13
	}

	sess := session.Must(session.NewSession(&aws.Config{
		HTTPClient: &http.Client{Transport: tr},
	}))

	uploader := s3manager.NewUploader(sess)

	_, err := uploader.Upload(&s3manager.UploadInput{
		Bucket: aws.String("examplebucket"),
		Key:    aws.String("largeobject"),
		Body:   bytes.NewReader([]byte("large_multi_part_upload")),
	})
	if err != nil {
		fmt.Println(err.Error())
	}
}

Output:

func (*Uploader) Upload 

func (u *Uploader) Upload(input *UploadInput) (*UploadOutput, error)

Upload uploads an object to S3, intelligently buffering large files into smaller chunks and sending them in parallel across multiple goroutines. You can configure the buffer size and concurrency through the opts parameter.

If opts is set to nil, DefaultUploadOptions will be used.

It is safe to call this method for multiple objects and across concurrent goroutines.

type WriterReadFrom added in v1.0.13 

type WriterReadFrom interface {
	io.Writer
	io.ReaderFrom
}

WriterReadFrom defines an interface implementing io.Writer and io.ReaderFrom

type WriterReadFromProvider added in v1.0.13 

type WriterReadFromProvider interface {
	GetReadFrom(writer io.Writer) (w WriterReadFrom, cleanup func())
}

WriterReadFromProvider provides an implementation of io.ReadFrom for the given io.Writer

Source Files

View all Source files

Directories

Path Synopsis
Package s3manageriface provides an interface for the s3manager package
 Package s3manageriface provides an interface for the s3manager package

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.