s3iofs

README

s3iofs

This package provides an S3 implementation for Go1.16 filesystem interface.

Overview

This package provides an S3 implementation for the Go1.16 filesystem interface using the AWS SDK for Go v2.

The S3FS is currently read-only, and implements the following interfaces:

• fs.FS
• fs.StatFS
• fs.ReadDirFS

The s3File implements the following interfaces:

• fs.FileInfo
• fs.DirEntry
• io.ReaderAt
• io.Seeker

This enables libraries such as apache arrow to read parts of a parquet file from S3, without downloading the entire file.

Usage

	// Load the Shared AWS Configuration (~/.aws/config) and enable request logging
	awscfg, err := config.LoadDefaultConfig(context.TODO(),
		config.WithClientLogMode(aws.LogRetries|aws.LogRequest),
		config.WithLogger(logging.NewStandardLogger(os.Stdout)),
	)
	if err != nil {
		// ...
	}

	s3fs := s3iofs.New(os.Getenv("TEST_BUCKET_NAME"), awscfg)

	err = fs.WalkDir(s3fs, ".", func(path string, d fs.DirEntry, err error) error {
		if err != nil {
			return err
		}

		if d.IsDir() {
			fmt.Println("dir:", path)
			return nil
		}
		fmt.Println("file:", path)
		return nil
	})
	if err != nil {
		// ...
	}

Integration Tests

The integration tests for this package are in a separate module under the integration directory, this to avoid polluting the main module with docker based testing dependencies used to run the tests locally against minio.

Links

S3 implements ranges based on HTTP Request ranges, it well worth reading up on this if your using the io.ReadSeek interface.

License

This application is released under Apache 2.0 license and is copyright Mark Wolfe.

Documentation

Index

• type S3API
• type S3FS
  • func New(bucket string, awscfg aws.Config) *S3FS
  • func NewWithClient(bucket string, client S3API) *S3FS
  • func (s3fs *S3FS) Open(name string) (fs.File, error)
  • func (s3fs *S3FS) ReadDir(name string) ([]fs.DirEntry, error)
  • func (s3fs *S3FS) Stat(name string) (fs.FileInfo, error)

Types

type S3API

type S3API interface {
	GetObject(ctx context.Context, params *s3.GetObjectInput, optFns ...func(*s3.Options)) (*s3.GetObjectOutput, error)
	ListObjectsV2(ctx context.Context, params *s3.ListObjectsV2Input, optFns ...func(*s3.Options)) (*s3.ListObjectsV2Output, error)
	HeadObject(ctx context.Context, params *s3.HeadObjectInput, optFns ...func(*s3.Options)) (*s3.HeadObjectOutput, error)
}

S3API s3 calls used to build this library, this is used to enable testing.

type S3FS

type S3FS struct {
	// contains filtered or unexported fields
}

S3FS is a filesystem implementation using S3.

func New

func New(bucket string, awscfg aws.Config) *S3FS

New returns a new filesystem which provides access to the specified s3 bucket.

func NewWithClient

func NewWithClient(bucket string, client S3API) *S3FS

NewWithClient returns a new filesystem which provides access to the specified s3 bucket.

func (*S3FS) Open

func (s3fs *S3FS) Open(name string) (fs.File, error)

Open opens the named file.

func (*S3FS) ReadDir

func (s3fs *S3FS) ReadDir(name string) ([]fs.DirEntry, error)

ReadDir reads the named directory.

func (*S3FS) Stat

func (s3fs *S3FS) Stat(name string) (fs.FileInfo, error)

Stat returns a FileInfo describing the file.