s3client

package module
v1.10.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 18, 2021 License: MIT Imports: 16 Imported by: 10

README

dp-s3

Client to interact with AWS S3

Getting started
Setting up AWS credentials

In order to access AWS S3, this library will require your access key id and access secret key. You can either setup a default profile in ~/.aws/credentials file:

[default]
aws_access_key_id=<id>
aws_secret_access_key=<secret>
region=eu-west-1

Or export the values as environmental variables:

export AWS_ACCESS_KEY_ID=<id>
export AWS_SECRET_ACCESS_KEY=<secret>

More information in Amazon documentation

Setting up IAM policy

The functionality implemented by this library requires that the user has some permissions defined by an IAM policy.

  • Health-check functionality performs a HEAD bucket operation, requiring allowed s3:ListBucket for all resources.

  • Get functionality requires allowed s3:GetObject for the objects under the hierarchy you want to allow (e.g. my-bucket/prefix/*).

  • Upload (PUT) functionality requires allowed s3:PutObject for the objects under the hierarchy you want to allow (e.g. my-bucket/prefix/*).

  • Multipart upload functionality requires allowed s3:PutObject, s3:GetObject, s3:AbortMultipartUpload, s3:ListMultipartUploadParts for objects under the hierarchy you want to allow (e.g. my-bucket/prefix/*); and s3:ListBucketMultipartUploads for the bucket (e.g. my-bucket).

Please, see our terraform repository for more information.

S3 Client Usage

The S3 client wraps the AWS SDK s3 client and offers functionality to read objects from S3 and upload objects using multipart upload, which is an AWS SDK functionality to perform uploads in chunks. More information here

The client contains a bucket and region, note that the bucket needs to be created in the region that you provide in order to access it.

There are 2 available constructors:

  • Constructor without AWS session (will create a new session):
s3cli := s3client.NewClient(<region>, <bucket>)
  • Constructor with AWS session (will reuse the provided session):
s3cli := s3client.NewClientWithSession(<bucket>, <awsSession>)

It is recommended to create a single AWS session in your service and reuse it if you need other clients. The client offers a session getter: s3cli.Session()

The S3 client exposes functions to get or upload files using the vanilla aws sdk, or the s3crypto wrapper, which allows you to provide a psk (pre-shared key) for encryption.

Functions that have the suffix WithPSK allow you to provide a psk for encryption. For example:

  • Get an un-encrypted object from S3
file, err := s3cli.Get("my/s3/file")
  • Get an encrypted object from S3, using a psk:
file, err := s3cli.GetWithPSK("my/s3/file", psk)
Uploader Usage

The Uploader is a higher level S3 client that wraps the SDK uploader, from s3manager package, as well as the lower level S3 client. This offers functionality to put objects in S3 in a single func call, hiding the low level details of chunking. More information here

Similarly to the s3 client, you can create an uploader and establish a new session, or reuse an existing one:

  • Constructor without AWS session (will create a new session):
s3Uploader := s3client.NewUploader(<region>, <bucket>)
  • Constructor with AWS session (will reuse the provided session):
s3Uploader := s3client.NewUploaderWithSession(<bucket>, <awsSession>)

Similarly to the s3 client, it is recommended to reuse AWS sessions between clients/uploaderes.

Functions that have the suffix WithPSK allow you to provide a psk for encryption. For example:

  • Upload an un-encrypted object to S3
result, err := s3Uploader.Upload(&s3manager.UploadInput{
		Body:   file.Reader,
		Key:    &filename,
	})
  • Upload an encrypted object to S3, using a psk:
result, err := s3Uploader.UploadWithPSK(&s3manager.UploadInput{
		Body:   file.Reader,
		Key:    &filename,
	}, psk)
URL Usage

S3Url is a structure intended to be used for S3 URL string manipulation in its different formats. To create a new structure you need to provide region, bucketName and object key, and optionally the scheme:

s3Url, err := func NewURL(<region>, <bucket>, <s3ObjcetKey>)
s3Url, err := func NewURLWithScheme(<scheme>, <region>, <bucket>, <s3ObjcetKey>)

If you want to parse a URL into an s3Url object, you can use ParseURL() method, providing the format style:

s3Url, err := ParseURL(<rawURL>, <URLStyle>)

Once you have a valid s3Url object, you can obtain the URL string representation in the required format style by calling String() method:

str, err := s3Url.String(<URLStyle>)
Valid URL format Styles

The following URL styles are supported:

  • PathStyle: https://s3-eu-west-1.amazonaws.com/myBucket/my/s3/object/key
  • GlobalPathStyle: https://s3.amazonaws.com/myBucket/my/s3/object/key
  • VirtualHostedStyle: https://myBucket.s3-eu-west-1.amazonaws.com/my/s3/object/key
  • GlobalVirtualHostedStyle: https://myBucket.s3.amazonaws.com/my/s3/object/key
  • AliasVirtualHostedStyle: 'https://myBucket/my/s3/object/key

More information in S3 official documentation

Health package

The S3 checker function performs a HEAD bucket operation . The health check will succeed only if the bucket can be accessed using the client (i.e. client must be authenticated correctly, bucket must exist and have been created in the same region as the client).

Read the Health Check Specification for details.

After creating an S3 client as described above, call s3 health checker with s3cli.Checker(context.Background()) and this will return a check object:

{
    "name": "string",
    "status": "string",
    "message": "string",
    "status_code": "int",
    "last_checked": "ISO8601 - UTC date time",
    "last_success": "ISO8601 - UTC date time",
    "last_failure": "ISO8601 - UTC date time"
}
Contributing

See CONTRIBUTING for details.

License

Copyright © 2020, Office for National Statistics (https://www.ons.gov.uk)

Released under MIT license, see LICENSE for details.

Documentation

Index

Constants

View Source 
const (
	// PathStyle example: 'https://s3-eu-west-1.amazonaws.com/myBucket/my/s3/object/key'
	PathStyle = iota
	// GlobalPathStyle example: 'https://s3.amazonaws.com/myBucket/my/s3/object/key'
	GlobalPathStyle
	// VirtualHostedStyle example: 'https://myBucket.s3-eu-west-1.amazonaws.com/my/s3/object/key'
	VirtualHostedStyle
	// GlobalVirtualHostedStyle example: 'https://myBucket.s3.amazonaws.com/my/s3/object/key'
	GlobalVirtualHostedStyle
	// AliasVirtualHostedStyle example: 'https://myBucket/my/s3/object/key'
	AliasVirtualHostedStyle
)

Possible S3 URL format styles, as defined in https://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html

View Source 
const MsgHealthy = "S3 is healthy"

MsgHealthy is the message in the Check structure when S3 is healthy

View Source 
const ServiceName = "S3"

ServiceName S3

Variables

This section is empty.

Functions

This section is empty.

Types

type Error added in v1.7.0 

type Error struct {
	// contains filtered or unexported fields
}

Error is the s3client package's error type

func NewError added in v1.7.0 

func NewError(err error, logData map[string]interface{}) *Error

NewError creates a new Error

func (*Error) Error added in v1.7.0 

func (e *Error) Error() string

Error implements the Go standard error interface

func (*Error) LogData added in v1.7.0 

func (e *Error) LogData() map[string]interface{}

LogData implements the DataLogger interface which allows you extract embedded log.Data from an error

func (*Error) Unwrap added in v1.7.0 

func (e *Error) Unwrap() error

Unwrap returns the wrapped error

type S3 

type S3 struct {
	// contains filtered or unexported fields
}

S3 client with SDK client, CryptoClient and BucketName

func InstantiateClient added in v1.2.0 

func InstantiateClient(sdkClient S3SDKClient, cryptoClient S3CryptoClient, bucketName, region string, s *session.Session) *S3

InstantiateClient creates a new instance of S3 struct with the provided clients, bucket and region

func NewClient added in v1.2.0 

func NewClient(region string, bucketName string) (*S3, error)

NewClient creates a new S3 Client configured for the given region and bucket name. Note: This function will create a new session, if you already have a session, please use NewUploaderWithSession instead Any error establishing the AWS session will be returned

func NewClientWithSession added in v1.2.0 

func NewClientWithSession(bucketName string, s *session.Session) *S3

NewClientWithSession creates a new S3 Client configured for the given bucket name, using the provided session and region within it.

func (*S3) BucketName 

func (cli *S3) BucketName() string

BucketName returns the bucket name used by this S3 client

func (*S3) CheckPartUploaded added in v1.2.0 

func (cli *S3) CheckPartUploaded(ctx context.Context, req *UploadPartRequest) (bool, error)

CheckPartUploaded returns true only if the chunk corresponding to the provided chunkNumber has been uploaded. If all the chunks have been uploaded, we complete the upload operation.

func (*S3) Checker 

func (cli *S3) Checker(ctx context.Context, state *health.CheckState) error

Checker validates that the S3 bucket exists, and updates the provided CheckState accordingly. Any error during the state update will be returned

func (*S3) Get 

func (cli *S3) Get(key string) (io.ReadCloser, *int64, error)

Get returns an io.ReadCloser instance for the given path (inside the bucket configured for this client) and the content length (size in bytes). They 'key' parameter refers to the path for the file under the bucket.

func (*S3) GetFromS3URL added in v1.1.0 

func (cli *S3) GetFromS3URL(rawURL string, style URLStyle) (io.ReadCloser, *int64, error)

GetFromS3URL returns an io.ReadCloser instance and the content length (size in bytes) for the given S3 URL, in the format specified by URLStyle. More information about s3 URL styles: https://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html If the URL defines a region (if provided) or bucket different from the one configured in this client, an error will be returned.

func (*S3) GetFromS3URLWithPSK added in v1.4.0 

func (cli *S3) GetFromS3URLWithPSK(rawURL string, style URLStyle, psk []byte) (io.ReadCloser, *int64, error)

GetFromS3URLWithPSK returns an io.ReadCloser instance and the content length (size in bytes) for the given S3 URL, in the format specified by URLStyle, using the provided PSK for encryption. More information about s3 URL styles: https://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html If the URL defines a region (if provided) or bucket different from the one configured in this client, an error will be returned.

func (*S3) GetWithPSK added in v1.3.0 

func (cli *S3) GetWithPSK(key string, psk []byte) (io.ReadCloser, *int64, error)

GetWithPSK returns an io.ReadCloser instance for the given path (inside the bucket configured for this client) and the content length (size in bytes). It uses the provided PSK for encryption. The 'key' parameter refers to the path for the file under the bucket.

func (*S3) Head added in v1.9.0 

func (cli *S3) Head(key string) (*s3.HeadObjectOutput, error)

Head returns a HeadObjectOutput containing an object metadata obtained from ah HTTP HEAD call

func (*S3) PutWithPSK added in v1.3.0 

func (cli *S3) PutWithPSK(key *string, reader *bytes.Reader, psk []byte) error

PutWithPSK uploads the provided contents to the key in the bucket configured for this client, using the provided PSK. The 'key' parameter refers to the path for the file under the bucket.

func (*S3) Session added in v1.2.0 

func (cli *S3) Session() *session.Session

Session returns the Session of this client

func (*S3) UploadPart added in v1.2.0 

func (cli *S3) UploadPart(ctx context.Context, req *UploadPartRequest, payload []byte) error

UploadPart handles the uploading a file to AWS S3, into the bucket configured for this client

func (*S3) UploadPartWithPsk added in v1.2.0 

func (cli *S3) UploadPartWithPsk(ctx context.Context, req *UploadPartRequest, payload []byte, psk []byte) error

UploadPartWithPsk handles the uploading a file to AWS S3, into the bucket configured for this client, using a user-defined psk

func (*S3) ValidateBucket added in v1.1.0 

func (cli *S3) ValidateBucket() error

ValidateBucket checks that the bucket exists and returns an error if it does not exist or there was some other error trying to get this information.

type S3CryptoClient added in v1.1.0 

type S3CryptoClient interface {
	UploadPartWithPSK(in *s3.UploadPartInput, psk []byte) (out *s3.UploadPartOutput, err error)
	GetObjectWithPSK(in *s3.GetObjectInput, psk []byte) (out *s3.GetObjectOutput, err error)
	PutObjectWithPSK(in *s3.PutObjectInput, psk []byte) (out *s3.PutObjectOutput, err error)
}

S3CryptoClient represents the cryptoclient with methods required to upload parts with encryption

type S3CryptoUploader added in v1.2.0 

type S3CryptoUploader interface {
	UploadWithPSK(ctx context.Context, in *s3manager.UploadInput, psk []byte) (out *s3manager.UploadOutput, err error)
}

S3CryptoUploader represents the s3crypto Uploader with methods required to upload parts with encryption

type S3SDKClient added in v1.1.0 

type S3SDKClient interface {
	ListMultipartUploads(in *s3.ListMultipartUploadsInput) (out *s3.ListMultipartUploadsOutput, err error)
	ListParts(in *s3.ListPartsInput) (out *s3.ListPartsOutput, err error)
	CompleteMultipartUpload(in *s3.CompleteMultipartUploadInput) (out *s3.CompleteMultipartUploadOutput, err error)
	CreateMultipartUpload(in *s3.CreateMultipartUploadInput) (out *s3.CreateMultipartUploadOutput, err error)
	UploadPart(in *s3.UploadPartInput) (out *s3.UploadPartOutput, err error)
	HeadBucket(in *s3.HeadBucketInput) (out *s3.HeadBucketOutput, err error)
	HeadObject(in *s3.HeadObjectInput) (out *s3.HeadObjectOutput, err error)
	GetObject(in *s3.GetObjectInput) (out *s3.GetObjectOutput, err error)
}

S3SDKClient represents the sdk client with methods required by dp-s3 client

type S3SDKUploader added in v1.2.0 

type S3SDKUploader interface {
	Upload(in *s3manager.UploadInput, options ...func(*s3manager.Uploader)) (out *s3manager.UploadOutput, err error)
	UploadWithContext(ctx context.Context, in *s3manager.UploadInput, options ...func(*s3manager.Uploader)) (out *s3manager.UploadOutput, err error)
}

S3SDKUploader represents the sdk uploader with methods required by dp-s3 client

type S3Url added in v1.4.0 

type S3Url struct {
	Scheme     string
	Region     string
	BucketName string
	Key        string
}

S3Url represents an S3 URL with bucketName, key and region (optional). This struct is intended to be used for S3 URL string manipulation/translation in its possible format styles.

func NewURL 

func NewURL(region, bucketName, key string) (*S3Url, error)

NewURL instantiates a new S3Url struct with the provided region, bucket name and object key

func NewURLWithScheme added in v1.4.0 

func NewURLWithScheme(scheme, region, bucketName, key string) (*S3Url, error)

NewURLWithScheme instantiates a new S3Url struct with the provided scheme, region, bucket and object key

func ParseAliasVirtualHostedURL added in v1.4.0 

func ParseAliasVirtualHostedURL(avhURL string) (*S3Url, error)

ParseAliasVirtualHostedURL creates an S3Url struct from the provided dns-alias-virtual-hosted-style url string Example: 'https://myBucket/my/s3/object/key'

func ParseGlobalPathStyleURL added in v1.4.0 

func ParseGlobalPathStyleURL(gpURL string) (*S3Url, error)

ParseGlobalPathStyleURL creates an S3Url struct from the provided global-path-style url string Example: 'https://s3.amazonaws.com/myBucket/my/s3/object/key' This method is compatible with PathStyle format (if region is present in the URL, it will be ignored)

func ParseGlobalVirtualHostedURL added in v1.4.0 

func ParseGlobalVirtualHostedURL(gvhURL string) (*S3Url, error)

ParseGlobalVirtualHostedURL creates an S3Url struct from the provided global-virtual-hosted-style url string Example: 'https://myBucket.s3.amazonaws.com/my/s3/object/key'

func ParsePathStyleURL added in v1.4.0 

func ParsePathStyleURL(pathStyleURL string) (*S3Url, error)

ParsePathStyleURL creates an S3Url struct from the provided path-style url string Example: 'https://s3-eu-west-1.amazonaws.com/myBucket/my/s3/object/key'.

func ParseURL added in v1.4.0 

func ParseURL(rawURL string, style URLStyle) (*S3Url, error)

ParseURL creates an S3Url struct from the provided rawULR and format style

func ParseVirtualHostedURL added in v1.4.0 

func ParseVirtualHostedURL(vhURL string) (*S3Url, error)

ParseVirtualHostedURL creates an S3Url struct from the provided virtual-hosted-style url string Example: 'https://myBucket.s3-eu-west-1.amazonaws.com/my/s3/object/key'

func (*S3Url) String added in v1.4.0 

func (s3Url *S3Url) String(style URLStyle) (string, error)

String returns the S3 URL string in the requested format style.

type URLStyle added in v1.4.0 

type URLStyle int

URLStyle is the type to define the URL style iota enumeration corresponding an S3 url (path, virtualHosted, etc)

func (URLStyle) String added in v1.4.0 

func (style URLStyle) String() string

Values of the format styles

type UploadPartRequest added in v1.2.0 

type UploadPartRequest struct {
	UploadKey   string
	Type        string
	ChunkNumber int64
	TotalChunks int
	FileName    string
}

UploadPartRequest represents a part upload request

type Uploader added in v1.2.0 

type Uploader struct {
	*S3
	// contains filtered or unexported fields
}

Uploader extends S3 client in order to perform Upload operations easily using the aws s3manager package. It allows Uploads with or without user-provided PSK for encryption.

func InstantiateUploader added in v1.2.0 

func InstantiateUploader(s3Client *S3, sdkUploader S3SDKUploader, cryptoUploader S3CryptoUploader) *Uploader

InstantiateUploader creates a new instance of Uploader struct with the provided clients, bucket and region

func NewUploader added in v1.2.0 

func NewUploader(region string, bucketName string) (*Uploader, error)

NewUploader creates a new Uploader configured for the given region and bucket name. Note: This function will create a new AWS session, if you already have a valid session, please use NewUploaderWithSession instead If an error occurs while establishing the AWS session, it will be returned

func NewUploaderWithSession added in v1.2.0 

func NewUploaderWithSession(bucketName string, s *session.Session) *Uploader

NewUploaderWithSession creates a new Uploader configured for the given bucket name, using the provided session and regions defined by it.

func (*Uploader) Session added in v1.2.0 

func (u *Uploader) Session() *session.Session

Session returns the Session of this uploader

func (*Uploader) Upload added in v1.2.0 

func (u *Uploader) Upload(input *s3manager.UploadInput, options ...func(*s3manager.Uploader)) (*s3manager.UploadOutput, error)

Upload uploads a file to S3 using the AWS s3Manager, which will automatically split up large objects and upload them concurrently.

func (*Uploader) UploadWithContext added in v1.8.0 

func (u *Uploader) UploadWithContext(ctx context.Context, input *s3manager.UploadInput, options ...func(*s3manager.Uploader)) (*s3manager.UploadOutput, error)

Upload uploads a file to S3 using the AWS s3Manager with context, which will automatically split up large objects and upload them concurrently. The provided context may be used to abort the operation.

func (*Uploader) UploadWithPSK added in v1.2.0 

func (u *Uploader) UploadWithPSK(input *s3manager.UploadInput, psk []byte) (*s3manager.UploadOutput, error)

UploadWithPSK uploads a file to S3 using cryptoclient, which allows you to encrypt the file with a given psk.

func (*Uploader) UploadWithPSKAndContext added in v1.10.0 

func (u *Uploader) UploadWithPSKAndContext(ctx context.Context, input *s3manager.UploadInput, psk []byte, options ...func(*s3manager.Uploader)) (*s3manager.UploadOutput, error)

UploadWithPSKAndContext uploads a file to S3 using cryptoclient, which allows you to encrypt the file with a given psk. The provided context may be used to abort the operation.

func (*Uploader) ValidateInput added in v1.10.0 

func (u *Uploader) ValidateInput(input *s3manager.UploadInput) (log.Data, error)

ValidateInput checks the input and returns an error if there is a bucket override mismatch or s3 key is not provided

Source Files

View all Source files

Directories

Path Synopsis
File copied form s3crypto repository Original repo: https://github.com/ONSdigital/s3crypto
 File copied form s3crypto repository Original repo: https://github.com/ONSdigital/s3crypto

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.