s3io

package module
v1.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 22, 2024 License: MIT Imports: 16 Imported by: 0

README

S3IO

Go Reference

An abstraction layer on top of the s3 sdk to do io read/write opperations on s3 objects. The s3io reader and writer stream the objects from and to your s3 instance while being memory efficient.

// Note the "WithBucket..." are options
bucket, err := s3io.OpenBucket(ctx, "my-bucket-name", s3io.WithBucketCredentials(accessKey, secretKey))
if err != nil {
  return err
}

// Note the "WithBucket..." are options specifically for this writer session
writer, err := bucket.NewWriter(ctx, "path/to/object.txt", s3io.WithWriterRetries(3))
if err != nil {
  return err
}

if _, err := io.WriteString(writer, "Hello world!"); err != nil {
  return err
}

if err := writer.Close(); err != nil {
  return err 
}

reader, err := bucket.NewReader(ctx, "path/to/object.txt")
if err != nil {
  return err
}

_, err := io.Copy(os.Stdout, reader)
...

Documentation

Overview

Package s3io is an abstraction layer on the s3 sdk.

The s3io package provides a bucket that can interact with all elements within the bucket. And can be configured with the "WithBucket..." options 

bucket, err := s3io.OpenBucket(ctx, "my-bucket-name", s3io.WithBucketCredentials("access-key", "secret-key"))

There is an ObjectReader to preform read opperations on an s3 object. 

rd, err := bucket.NewReader(ctx, "path/to/object.txt", s3io.WithReaderConcurrency(10))
if err != nil {
  return err
}

_, err := io.Copy(os.Stdout, rd)

And there is an ObjectWriter to preform write opperations on an s3 object. Note The writer MUST close to safe the object. 

wr, err := bucket.NewWriter(ctx, "path/to/object.txt")
if err != nil
  return err
}

_, err := io.WriteString(wr, "Hello world!")
if err != nil {
  return err
}

if err := wr.Close(); err != nil {
  return err
}

The s3io reader and writer stream the objects from and to your s3 instance while being memory efficient.

Index

Examples

Constants

View Source 
const (
	MinChunkSize     = 1024 * 1024 * 5
	DefaultChunkSize = MinChunkSize
)

Variables

View Source 
var (
	ErrMinChunkSize = errors.New("given value is less than minimum chunksize of 5mb")
)

Functions

This section is empty.

Types

type Bucket 

type Bucket struct {
	// contains filtered or unexported fields
}

Bucket is an abstraction to interact with objects in your S3 bucket

func OpenBucket 

func OpenBucket(ctx context.Context, name string, opts ...BucketOption) (*Bucket, error)

OpenBucket returns a bucket to interact with.

func OpenBucketkwithCli 

func OpenBucketkwithCli(ctx context.Context, cli *s3.Client, name string, opts ...BucketOption) (*Bucket, error)

OpenBucketkwithCli returns a bucket to interact with.

func (*Bucket) Client 

func (b *Bucket) Client() *s3.Client

Client returns the s3 client the Bucket uses

func (*Bucket) Delete 

func (b *Bucket) Delete(ctx context.Context, keys ...string) error

Delete deletes the given object keys

func (*Bucket) Exists 

func (b *Bucket) Exists(ctx context.Context, key string) (bool, error)

Exists returns a a boolean indicating whether the requested object exists.

func (*Bucket) List 

func (b *Bucket) List(ctx context.Context, prefix string) ([]types.Object, error)

List returns a list of objects details.

Use the prefix "/" to list all the objects in the bucket.

func (*Bucket) NewReader 

func (b *Bucket) NewReader(ctx context.Context, key string, opts ...ObjectReaderOption) (io.Reader, error)

NewReader returns a new ObjectReader to do io.Reader opperations with your s3 object

func (*Bucket) NewWriter 

func (b *Bucket) NewWriter(ctx context.Context, key string, opts ...ObjectWriterOption) (io.WriteCloser, error)

NewWriter returns a new ObjectWriter to do io.Write opparations with your s3 object

func (*Bucket) NewWriterIfNotExists 

func (b *Bucket) NewWriterIfNotExists(ctx context.Context, key string, opts ...ObjectWriterOption) (io.WriteCloser, error)

NewWriterIfNotExists returns a writer if the object doesn't already exist

type BucketOption 

type BucketOption func(*bucketBuilder) error

func BucketOptions 

func BucketOptions(opts ...BucketOption) BucketOption

BucketOptions bundles bucket options

func WithBucketCli 

func WithBucketCli(cli *s3.Client) BucketOption

WithCli directly sets the s3 client for the bucket.

func WithBucketCliLoaderOptions 

func WithBucketCliLoaderOptions(opts ...func(*config.LoadOptions) error) BucketOption

WithBucketCliLoaderOptions sets the config.LoaderOptions for the aws config. Only works if the cli is not already provided.

func WithBucketConcurrency 

func WithBucketConcurrency(size uint8) BucketOption

WithBucketConcurrency sets the default read/write concurrency

func WithBucketCreateIfNotExists 

func WithBucketCreateIfNotExists() BucketOption

WithBucketCreateIfNotExitsts will create the bucket if it doesn't already exist.

func WithBucketCredentials 

func WithBucketCredentials(accessKey, secretKey string) BucketOption

WithBucketCredentials sets the access key and secret key for the cli. Only works if the cli is not already provided.

func WithBucketHost 

func WithBucketHost(url, region string, usePathStyle bool) BucketOption

WithBucketHost sets the endpoint, region and if it uses a pathstyle for the cli. Only works if the cli is not already provided.

func WithBucketLogger 

func WithBucketLogger(logger *slog.Logger) BucketOption

WithBucketLogger sets the default logger for any opperation. Setting the logger provides debug logs.

func WithBucketReadChunkSize 

func WithBucketReadChunkSize(size uint) BucketOption

WithBucketReadChunkSize sets the default read chunksize

func WithBucketRetries 

func WithBucketRetries(i uint8) BucketOption

WithBucketRetries sets the default amount of retries for any given opperation

func WithBucketS3Options 

func WithBucketS3Options(opts ...func(*s3.Options)) BucketOption

WithBucketS3Options sets the s3 options for t.he s3 client. Only works if the cli is not already provided.

func WithBucketWriteChunkSize 

func WithBucketWriteChunkSize(size uint) BucketOption

WithBucketWriteChunkSize sets the default write chunksize

type ObjectReader 

type ObjectReader struct {
	// contains filtered or unexported fields
}

ObjectReader is an io.Reader implementation for an S3 Object

func NewObjectReader added in v1.0.0 

func NewObjectReader(ctx context.Context, cli *s3.Client, bucketName, key string, opts ...ObjectReaderOption) (*ObjectReader, error)

NewObjectReader returns a new ObjectReader to do io.Reader opperations on your s3 object

func (*ObjectReader) Read 

func (r *ObjectReader) Read(p []byte) (int, error)

Read is the io.Reader implementation for the ObjectReader.

It returns an fs.ErrNotExists if the object doesn't exist in the given bucket. And returns an io.EOF when all bytes are read.

Example 
ctx := context.Background()

bucket, err := OpenBucket(ctx, "bucket-name",
	WithBucketCredentials("access-key", "access-secret"),
	WithBucketCreateIfNotExists(),
)
if err != nil {
	log.Fatal(err)
}

rd, err := bucket.NewReader(ctx, "path/to/object.txt")
if err != nil {
	log.Fatal(err)
}

if _, err := io.Copy(os.Stdout, rd); err != nil {
	log.Fatal(err)
}

Output:

type ObjectReaderOption added in v1.0.0 

type ObjectReaderOption func(*ObjectReader) error

ObjectReaderOption is an option for the given read operation

func ObjectReaderOptions added in v1.0.0 

func ObjectReaderOptions(opts ...ObjectReaderOption) ObjectReaderOption

ObjectReaderOptions is a collection of ObjectReaderOption's

func WithReaderChunkSize 

func WithReaderChunkSize(size uint) ObjectReaderOption

WithReaderChunkSize sets the chunksize for this reader

func WithReaderConcurrency 

func WithReaderConcurrency(i uint8) ObjectReaderOption

WithReaderConcurrency set the concurency amount for this reader

func WithReaderLogger 

func WithReaderLogger(logger *slog.Logger) ObjectReaderOption

WithReaderLogger sets the logger for this reader

func WithReaderRetries 

func WithReaderRetries(i uint8) ObjectReaderOption

WithReaderRetries sets the retry count for this reader

func WithReaderS3Options added in v1.0.1 

func WithReaderS3Options(opts ...func(*s3.Options)) ObjectReaderOption

WithReaderS3Options adds s3 options to the reader opperations

type ObjectWriter 

type ObjectWriter struct {
	// contains filtered or unexported fields
}

ObjectWriter is an io.WriteCloser implementation for an s3 Object

func NewObjectWriter added in v1.0.0 

func NewObjectWriter(ctx context.Context, cli *s3.Client, bucketName, key string, opts ...ObjectWriterOption) (*ObjectWriter, error)

NewObjectWriter returns a new ObjectWriter to do io.Writer opperations on your s3 object

func (*ObjectWriter) Close 

func (w *ObjectWriter) Close() error

Close completes the write opperation.

If the byte size is less than writer's chunk size then a simply PutObject opperation is preformed. Otherwise a multipart upload complete opperation is preformed. The error returned is the error from this store opperation.

If an error occured while uploading parts this error might also be a upload part error joined with a AbortMultipartUpload error.

func (*ObjectWriter) Write 

func (w *ObjectWriter) Write(p []byte) (int, error)

Write is the io.Writer implementation of the ObjectWriter

The object is stored when the Close method is called.

Example 
ctx := context.Background()

bucket, err := OpenBucket(ctx, "bucket-name",
	WithBucketCredentials("access-key", "access-secret"),
	WithBucketCreateIfNotExists(),
)
if err != nil {
	log.Fatal(err)
}

wr, err := bucket.NewWriter(ctx, "path/to/object.txt")
if err != nil {
	log.Fatal(err)
}

if _, err := io.WriteString(wr, "hello world"); err != nil {
	log.Fatal(err)
}

if err := wr.Close(); err != nil {
	log.Fatal(err)
}

Output:

type ObjectWriterOption added in v1.0.0 

type ObjectWriterOption func(*ObjectWriter) error

ObjectWriterOption is an option for the given write operation

func ObjectWriterOptions added in v1.0.0 

func ObjectWriterOptions(opts ...ObjectWriterOption) ObjectWriterOption

ObjectWriterOptions is a collection of ObjectWriterOption's

func WithWriteRetries 

func WithWriteRetries(i uint8) ObjectWriterOption

WithWriterRetries sets the retry count for this writer

func WithWriterACL 

func WithWriterACL(acl types.ObjectCannedACL) ObjectWriterOption

WithWriterACL sets the ACL for the object thats written

func WithWriterChunkSize 

func WithWriterChunkSize(size uint) ObjectWriterOption

WithWriterChunkSize sets the chunksize for this writer

func WithWriterConcurrency 

func WithWriterConcurrency(i uint8) ObjectWriterOption

WithWriterConcurrency sets the concurrency amount for this writer

func WithWriterLogger 

func WithWriterLogger(logger *slog.Logger) ObjectWriterOption

WithWriterLogger adds a logger for this writer

func WithWriterS3Options added in v1.0.1 

func WithWriterS3Options(opts ...func(*s3.Options)) ObjectWriterOption

WithWriterS3Options adds s3 options to the writer opperations

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.