s3io

package module
v2.6.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 27, 2024 License: MIT Imports: 22 Imported by: 1

README

S3IO

Go Reference Go Report Card

An abstraction layer on top of the s3 sdk to do io read/write opperations on s3 objects. The s3io reader and writer stream the objects from and to your s3 instance while being memory efficient.

// Note the "WithBucket..." are options
bucket, err := s3io.OpenBucket(ctx, "my-bucket-name", s3io.WithBucketCredentials(accessKey, secretKey))
if err != nil {
  return err
}

// Note the "WithBucket..." are options specifically for this writer session
writer := bucket.NewWriter(ctx, "path/to/object.txt", s3io.WithWriterRetries(3))
defer writer.Close() // makes sure your upload won't keep hanging

if _, err := io.WriteString(writer, "Hello world!"); err != nil {
  return err
}

if err := writer.Close(); err != nil {
  return err 
}

reader := bucket.NewReader(ctx, "path/to/object.txt")

_, err := io.Copy(os.Stdout, reader)


// Use the bucket as fs.FS
subSys, err := fs.Sub(bucket, "path/to/subdir")
...

Documentation

Overview

Package s3io is an abstraction layer on the s3 sdk.

The s3io package provides a bucket that can interact with all elements within the bucket. And can be configured with the "WithBucket..." options 

bucket, err := s3io.OpenBucket(ctx, "my-bucket-name", s3io.WithBucketCredentials("access-key", "secret-key"))

There is an ObjectReader to preform read opperations on an s3 object. 

rd := bucket.NewReader(ctx, "path/to/object.txt", s3io.WithReaderConcurrency(10))

_, err := io.Copy(os.Stdout, rd)

And there is an ObjectWriter to preform write opperations on an s3 object. Note The writer MUST close to safe the object. 

wr := bucket.NewWriter(ctx, "path/to/object.txt")
defer wr.Close()

_, err := io.WriteString(wr, "Hello world!")
if err != nil {
  return err
}

if err := wr.Close(); err != nil {
  return err
}

The s3io reader and writer stream the objects from and to your s3 instance while being memory efficient.

Index

Examples

Constants

View Source 
const (
	MinChunkSize     int64 = 1024 * 1024 * 5
	DefaultChunkSize int64 = MinChunkSize
)

Variables

This section is empty.

Functions

func NewObjectReader 

func NewObjectReader(ctx context.Context, s3 DownloadAPIClient, input *s3.GetObjectInput, opts ...ObjectReaderOption) io.Reader

NewObjectReader returns a new ObjectReader to do io.Reader opperations on your s3 object

func NewObjectWriter 

func NewObjectWriter(ctx context.Context, s3 UploadAPIClient, input *s3.PutObjectInput, opts ...ObjectWriterOption) io.WriteCloser

NewObjectWriter returns a new ObjectWriter to do io.Writer opperations on your s3 object

func ReadAll added in v2.2.0 

func ReadAll(ctx context.Context, s3 DownloadAPIClient, input *s3.GetObjectInput, opts ...ObjectReaderOption) ([]byte, error)

ReadAll reads all the bytes for a given object

func WriteAllBytes added in v2.2.0 

func WriteAllBytes(ctx context.Context, s3 UploadAPIClient, input *s3.PutObjectInput, p []byte, opts ...ObjectWriterOption) (int, error)

WriteAllBytes writes the given bytes into hte given object

func WriteAllFromBody added in v2.2.0 

func WriteAllFromBody(ctx context.Context, s3 UploadAPIClient, input *s3.PutObjectInput, opts ...ObjectWriterOption) (int64, error)

WriteAllFromBody writes to the given object using the input.Body

Types

type Bucket 

type Bucket struct {
	// contains filtered or unexported fields
}

Bucket is an abstraction to interact with objects in your S3 bucket

func NewRawBucket added in v2.5.0 

func NewRawBucket(
	name string,
	readChunkSize, writeChunkSize int64,
	concurrency int,
	logger *slog.Logger,
	cli BucketApiClient,
) *Bucket

NewRawBucket returns a new bucket instance. For normal operations use OpenBucket instead as this will connect and verify the bucket.

func OpenBucket 

func OpenBucket(ctx context.Context, name string, opts ...BucketOption) (*Bucket, error)

OpenBucket returns a bucket to interact with.

func OpenBucketkwithClient 

func OpenBucketkwithClient(ctx context.Context, cli *s3.Client, name string, opts ...BucketOption) (*Bucket, error)

OpenBucketkwithClient returns a bucket to interact with.

func OpenURL added in v2.4.0 

func OpenURL(ctx context.Context, u string, opts ...BucketOption) (*Bucket, error)

OpenURL opens the bucket with all the connection options in the url. The url is written as: s3://access-key:access-secret@host/bucketname?region=us-east&pathstyle=true

The url assumes the host has a https protocol unless the "insecure" query param is set to "true". To create the bucket if it doesn't exist set the "create" query param to "true". To use the pathstyle url set "pathstyle" to "true".

func (*Bucket) Client 

func (b *Bucket) Client() *s3.Client

Client returns the s3 client the Bucket uses

func (*Bucket) Delete 

func (b *Bucket) Delete(ctx context.Context, keys ...string) error

Delete deletes the given object keys

func (*Bucket) DeleteBucket added in v2.6.0 

func (b *Bucket) DeleteBucket(ctx context.Context) error

DeleteBucket deletes the whole bucket

func (*Bucket) Exists 

func (b *Bucket) Exists(ctx context.Context, key string) (bool, error)

Exists returns a a boolean indicating whether the requested object exists.

func (*Bucket) Glob added in v2.3.0 

func (b *Bucket) Glob(pattern string) ([]string, error)

Glob is an implementation of fs.GlobFS

func (*Bucket) List 

func (b *Bucket) List(ctx context.Context, prefix string) ([]types.Object, error)

List returns a list of objects details.

Use the prefix "/" to list all the objects in the bucket.

func (*Bucket) Name 

func (b *Bucket) Name() string

Name returns the specified bucket's name

func (*Bucket) NewReader 

func (b *Bucket) NewReader(ctx context.Context, key string, opts ...ObjectReaderOption) io.Reader

NewReader returns a new ObjectReader to do io.Reader opperations with your s3 object

func (*Bucket) NewWriter 

func (b *Bucket) NewWriter(ctx context.Context, key string, opts ...ObjectWriterOption) io.WriteCloser

NewWriter returns a new ObjectWriter to do io.Write opparations with your s3 object

func (*Bucket) Open added in v2.3.0 

func (b *Bucket) Open(name string) (fs.File, error)

Open is the fs.FS implenentation for the Bucket.

Open returns an fs.ErrInvalid argument if there was an error in checking if the file exists and an fs.ErrNotExists if the object doesn't exist

Example 
// you're able to use the bucket as fs.FS.
// here's an example how to use it with your html/template's.

ctx := context.Background()

bucket, err := OpenBucket(ctx, "bucket-name",
	WithBucketCredentials("access-key", "access-secret"),
	WithBucketCreateIfNotExists(),
)
if err != nil {
	log.Fatal(err)
}

templateEngine, err := template.ParseFS(bucket, "path/to/templates/*.html.tmpl")
if err != nil {
	log.Fatal(err)
}

err = templateEngine.ExecuteTemplate(os.Stdout, "index.html.tmpl", map[string]any{
	"arg1": "foo",
	"arg2": "bar",
})
if err != nil {
	log.Fatal(err)
}

// Or use it as simple subFS
subSys, err := fs.Sub(bucket, "path/to/subdir")
if err != nil {
	log.Fatal(err)
}

file, err := subSys.Open("somefile")
if err != nil {
	log.Fatal(err)
}

if _, err := io.Copy(os.Stdout, file); err != nil {
	log.Fatal(err)
}

Output:

func (*Bucket) ReadAll added in v2.2.0 

func (b *Bucket) ReadAll(ctx context.Context, key string, opts ...ObjectReaderOption) ([]byte, error)

ReadAll reads all the bytes of the given object

func (*Bucket) ReadFrom added in v2.5.0 

func (b *Bucket) ReadFrom(ctx context.Context, key string, rd io.Reader, opts ...ObjectWriterOption) (int64, error)

ReadFrom reads the bytes from the given reader into the object and closes the reader if it implements io.Closer

func (*Bucket) WriteAll added in v2.2.0 

func (b *Bucket) WriteAll(ctx context.Context, key string, p []byte, opts ...ObjectWriterOption) (int, error)

WriteAll writes all the given bytes into the given object

func (*Bucket) WriteFrom deprecated added in v2.2.0 

func (b *Bucket) WriteFrom(ctx context.Context, key string, from io.Reader, opts ...ObjectWriterOption) (int64, error)

WriteFrom writes all the bytes from the reader into the given object

Deprecated: use the improved and better named ReadFrom instead

func (*Bucket) WriteTo added in v2.5.0 

func (b *Bucket) WriteTo(ctx context.Context, key string, wr io.Writer, opts ...ObjectReaderOption) (int64, error)

WriteTo write all the bytes from the object into the given writer and closes the writer if it implements io.Closer

type BucketApiClient added in v2.5.0 

type BucketApiClient interface {
	DownloadAPIClient
	UploadAPIClient
	s3.HeadObjectAPIClient
	s3.ListObjectsV2APIClient
	DeleteObject(ctx context.Context, input *s3.DeleteObjectInput, optFns ...func(*s3.Options)) (*s3.DeleteObjectOutput, error)
	DeleteObjects(ctx context.Context, input *s3.DeleteObjectsInput, optFns ...func(*s3.Options)) (*s3.DeleteObjectsOutput, error)
	DeleteBucket(ctx context.Context, input *s3.DeleteBucketInput, optFns ...func(*s3.Options)) (*s3.DeleteBucketOutput, error)
}

type BucketOption 

type BucketOption func(*bucketBuilder)

func BucketOptions 

func BucketOptions(opts ...BucketOption) BucketOption

BucketOptions bundles bucket options

func WithBucketCli 

func WithBucketCli(cli *s3.Client) BucketOption

WithCli directly sets the s3 client for the bucket.

func WithBucketCliLoaderOptions 

func WithBucketCliLoaderOptions(opts ...func(*config.LoadOptions) error) BucketOption

WithBucketCliLoaderOptions sets the config.LoaderOptions for the aws config. Only works if the cli is not already provided.

func WithBucketConcurrency 

func WithBucketConcurrency(size int) BucketOption

WithBucketConcurrency sets the default read/write concurrency

func WithBucketCreateIfNotExists 

func WithBucketCreateIfNotExists() BucketOption

WithBucketCreateIfNotExitsts will create the bucket if it doesn't already exist.

func WithBucketCredentials 

func WithBucketCredentials(accessKey, secretKey string) BucketOption

WithBucketCredentials sets the access key and secret key for the cli. Only works if the cli is not already provided.

func WithBucketHost 

func WithBucketHost(url, region string, usePathStyle bool) BucketOption

WithBucketHost sets the endpoint, region and if it uses a pathstyle for the cli. Only works if the cli is not already provided.

func WithBucketLogger 

func WithBucketLogger(logger *slog.Logger) BucketOption

WithBucketLogger sets the default logger for any opperation. Setting the logger provides debug logs.

func WithBucketReadChunkSize 

func WithBucketReadChunkSize(size int64) BucketOption

WithBucketReadChunkSize sets the default read chunksize

func WithBucketRetries 

func WithBucketRetries(i int) BucketOption

WithBucketRetries sets the default amount of retries for any given opperation

func WithBucketS3Options 

func WithBucketS3Options(opts ...func(*s3.Options)) BucketOption

WithBucketS3Options sets the s3 options for t.he s3 client. Only works if the cli is not already provided.

func WithBucketWriteChunkSize 

func WithBucketWriteChunkSize(size int64) BucketOption

WithBucketWriteChunkSize sets the default write chunksize

type BucketURIError added in v2.4.1 

type BucketURIError string

BucketURIError 

var (
	ErrInvalidScheme BucketURIError = "url doesn't start with s3 scheme"
	ErrNoBucketName  BucketURIError = "no bucketname"
	ErrNoCredentials BucketURIError = "missing credentials"
)

func (BucketURIError) Error added in v2.4.1 

func (b BucketURIError) Error() string

type DownloadAPIClient 

type DownloadAPIClient interface {
	GetObject(context.Context, *s3.GetObjectInput, ...func(*s3.Options)) (*s3.GetObjectOutput, error)
}

DownloadAPIClient is an S3 API client that can invoke the GetObject operation.

type ObjectReader 

type ObjectReader struct {
	// contains filtered or unexported fields
}

ObjectReader is an io.Reader implementation for an S3 Object

func (*ObjectReader) Close added in v2.3.0 

func (r *ObjectReader) Close() error

close is the io.Close implementation for the ObjectReader

func (*ObjectReader) Read 

func (r *ObjectReader) Read(p []byte) (int, error)

Read is the io.Reader implementation for the ObjectReader.

It returns an fs.ErrNotExists if the object doesn't exist in the given bucket. And returns an io.EOF when all bytes are read.

Example 
ctx := context.Background()

bucket, err := OpenBucket(ctx, "bucket-name",
	WithBucketCredentials("access-key", "access-secret"),
	WithBucketCreateIfNotExists(),
)
if err != nil {
	log.Fatal(err)
}

rd := bucket.NewReader(ctx, "path/to/object.txt")
if err != nil {
	log.Fatal(err)
}

if _, err := io.Copy(os.Stdout, rd); err != nil {
	log.Fatal(err)
}

Output:

func (*ObjectReader) Stat added in v2.3.0 

func (r *ObjectReader) Stat() (fs.FileInfo, error)

Stat is the fs.File implementaion for the ObjectReader.

type ObjectReaderOption 

type ObjectReaderOption func(*ObjectReader)

ObjectReaderOption is an option for the given read operation.

func ObjectReaderOptions 

func ObjectReaderOptions(opts ...ObjectReaderOption) ObjectReaderOption

ObjectReaderOptions is a collection of ObjectReaderOption's.

func WithReaderChunkSize 

func WithReaderChunkSize(size int64) ObjectReaderOption

WithReaderChunkSize sets the chunksize for this reader

func WithReaderConcurrency 

func WithReaderConcurrency(i int) ObjectReaderOption

WithReaderConcurrency set the concurency amount for this reader

func WithReaderLogger 

func WithReaderLogger(logger *slog.Logger) ObjectReaderOption

WithReaderLogger sets the logger for this reader

func WithReaderRetries 

func WithReaderRetries(i int) ObjectReaderOption

WithReaderRetries sets the retry count for this reader

func WithReaderS3Options 

func WithReaderS3Options(opts ...func(*s3.Options)) ObjectReaderOption

WithReaderS3Options adds s3 options to the reader opperations

type ObjectWriter 

type ObjectWriter struct {
	// contains filtered or unexported fields
}

ObjectWriter is an io.WriteCloser implementation for an s3 Object

func (*ObjectWriter) Close 

func (w *ObjectWriter) Close() error

Close completes the write opperation.

If the byte size is less than writer's chunk size then a simply PutObject opperation is preformed. Otherwise a multipart upload complete opperation is preformed. The error returned is the error from this store opperation.

If an error occured while uploading parts this error might also be a upload part error joined with a AbortMultipartUpload error.

func (*ObjectWriter) Write 

func (w *ObjectWriter) Write(p []byte) (int, error)

Write is the io.Writer implementation of the ObjectWriter

The object is stored when the Close method is called.

Example 
ctx := context.Background()

bucket, err := OpenBucket(ctx, "bucket-name",
	WithBucketCredentials("access-key", "access-secret"),
	WithBucketCreateIfNotExists(),
)
if err != nil {
	log.Fatal(err)
}

wr := bucket.NewWriter(ctx, "path/to/object.txt")
if err != nil {
	log.Fatal(err)
}

if _, err := io.WriteString(wr, "hello world"); err != nil {
	log.Fatal(err)
}

// Note: you must close to finilize the upload
if err := wr.Close(); err != nil {
	log.Fatal(err)
}

Output:

type ObjectWriterOption 

type ObjectWriterOption func(*ObjectWriter)

ObjectWriterOption is an option for the given write operation

func ObjectWriterOptions 

func ObjectWriterOptions(opts ...ObjectWriterOption) ObjectWriterOption

ObjectWriterOptions is a collection of ObjectWriterOption's

func WithWriteRetries 

func WithWriteRetries(i int) ObjectWriterOption

WithWriterRetries sets the retry count for this writer

func WithWriterACL added in v2.2.1 

func WithWriterACL(acl types.ObjectCannedACL) ObjectWriterOption

Set ACL after giving the input

func WithWriterChunkSize 

func WithWriterChunkSize(size int64) ObjectWriterOption

WithWriterChunkSize sets the chunksize for this writer. If set below the minimal chunk size of 5Mb then it will be set to the minimal chunksize.

func WithWriterClientOptions 

func WithWriterClientOptions(opts ...func(*s3.Options)) ObjectWriterOption

WithWriterClientOptions adds s3 client options to the writer opperations

func WithWriterConcurrency 

func WithWriterConcurrency(i int) ObjectWriterOption

WithWriterConcurrency sets the concurrency amount for this writer.

func WithWriterLogger 

func WithWriterLogger(logger *slog.Logger) ObjectWriterOption

WithWriterLogger adds a logger for this writer.

type UploadAPIClient 

type UploadAPIClient interface {
	PutObject(context.Context, *s3.PutObjectInput, ...func(*s3.Options)) (*s3.PutObjectOutput, error)
	UploadPart(context.Context, *s3.UploadPartInput, ...func(*s3.Options)) (*s3.UploadPartOutput, error)
	CreateMultipartUpload(context.Context, *s3.CreateMultipartUploadInput, ...func(*s3.Options)) (*s3.CreateMultipartUploadOutput, error)
	CompleteMultipartUpload(context.Context, *s3.CompleteMultipartUploadInput, ...func(*s3.Options)) (*s3.CompleteMultipartUploadOutput, error)
	AbortMultipartUpload(context.Context, *s3.AbortMultipartUploadInput, ...func(*s3.Options)) (*s3.AbortMultipartUploadOutput, error)
}

UploadAPIClient is an S3 API client that can invoke PutObject, UploadPart, CreateMultipartUpload, CompleteMultipartUpload, and AbortMultipartUpload operations.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.