s3

package module
v0.4.20 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 20, 2022 License: MIT Imports: 16 Imported by: 2

README

S3 Backend for Afero

About

It provides an afero filesystem implementation of an S3 backend.

This was created to provide a backend to the ftpserver but can definitely be used in any other code.

I'm very opened to any improvement through issues or pull-request that might lead to a better implementation or even better testing.

Key points

  • Download & upload file streaming
  • 75% coverage (all APIs are tested, but not all errors are reproduced)
  • Very carefully linted

Known limitations

  • File appending / seeking for write is not supported because S3 doesn't support it, it could be simulated by rewriting entire files.
  • Chtimes is not supported because S3 doesn't support it, it could be simulated through metadata.
  • Chmod support is very limited

How to use

Note: Errors handling is skipped for brevity but you definitely have to handle it.


import(
	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/credentials"
	"github.com/aws/aws-sdk-go/aws/session"
  
	s3 "github.com/legion-zver/afero-s3"
)

func main() {
  // You create a session
  sess, _ := session.NewSession(&aws.Config{
    Region:      aws.String(region),
    Credentials: credentials.NewStaticCredentials(keyID, secretAccessKey, ""),
  })

  // Initialize the file system
  s3Fs := s3.NewFs(bucket, sess)

  // And do your thing
  file, _ := fs.OpenFile(name, os.O_WRONLY, 0777)
  file.WriteString("Hello world !")
  file.Close()
}

Thanks

The initial code (which was massively rewritten) comes from:

Documentation

Overview

Package s3 brings S3 files handling to afero

Package s3 brings S3 files handling to afero

Package s3 brings S3 files handling to afero

Index

Constants

This section is empty.

Variables

View Source 
var ErrAlreadyOpened = errors.New("already opened")

ErrAlreadyOpened is returned when the file is already opened

View Source 
var ErrInvalidSeek = errors.New("invalid seek offset")

ErrInvalidSeek is returned when the seek operation is not doable

View Source 
var ErrNotImplemented = errors.New("not implemented")

ErrNotImplemented is returned when this operation is not (yet) implemented

View Source 
var ErrNotSupported = errors.New("s3 doesn't support this operation")

ErrNotSupported is returned when this operations is not supported by S3

Functions

This section is empty.

Types

type CachedFilesInfo added in v0.4.17 

type CachedFilesInfo struct {
	// contains filtered or unexported fields
}

func (CachedFilesInfo) Del added in v0.4.17 

func (c CachedFilesInfo) Del(key string)

Del file info by key

func (CachedFilesInfo) Get added in v0.4.17 

func (c CachedFilesInfo) Get(key string) os.FileInfo

Get file info from cache

func (CachedFilesInfo) HasKey added in v0.4.17 

func (c CachedFilesInfo) HasKey(key string) bool

HasKey check info by cache

func (CachedFilesInfo) Set added in v0.4.17 

func (c CachedFilesInfo) Set(key string, info os.FileInfo) os.FileInfo

Set file info by key

type File 

type File struct {
	// contains filtered or unexported fields
}

File represents a file in S3. nolint: maligned

func NewFile 

func NewFile(fs *Fs, name string) *File

NewFile initializes an File object.

func (*File) Close 

func (f *File) Close() error

Close closes the File, rendering it unusable for I/O. It returns an error, if any.

func (*File) Name 

func (f *File) Name() string

Name returns the filename, i.e. S3 path without the bucket name.

func (*File) Read 

func (f *File) Read(p []byte) (int, error)

Read reads up to len(b) bytes from the File. It returns the number of bytes read and an error, if any. EOF is signaled by a zero count with err set to io.EOF.

func (*File) ReadAt 

func (f *File) ReadAt(p []byte, off int64) (n int, err error)

ReadAt reads len(p) bytes from the file starting at byte offset off. It returns the number of bytes read and the error, if any. ReadAt always returns a non-nil error when n < len(b). At end of file, that error is io.EOF.

func (*File) ReadAtWithSize 

func (f *File) ReadAtWithSize(p []byte, off, size int64) (n int, err error)

func (*File) Readdir 

func (f *File) Readdir(n int) ([]os.FileInfo, error)

Readdir reads the contents of the directory associated with file and returns a slice of up to n FileInfo values, as would be returned by ListObjects, in directory order. Subsequent calls on the same file will yield further FileInfos.

If n > 0, Readdir returns at most n FileInfo structures. In this case, if Readdir returns an empty slice, it will return a non-nil error explaining why. At the end of a directory, the error is io.EOF.

If n <= 0, Readdir returns all the FileInfo from the directory in a single slice. In this case, if Readdir succeeds (reads all the way to the end of the directory), it returns the slice and a nil error. If it encounters an error before the end of the directory, Readdir returns the FileInfo read until that point and a non-nil error.

func (*File) ReaddirAll 

func (f *File) ReaddirAll() ([]os.FileInfo, error)

ReaddirAll provides list of file cachedInfo.

func (*File) Readdirnames 

func (f *File) Readdirnames(n int) ([]string, error)

Readdirnames reads and returns a slice of names from the directory f.

If n > 0, Readdirnames returns at most n names. In this case, if Readdirnames returns an empty slice, it will return a non-nil error explaining why. At the end of a directory, the error is io.EOF.

If n <= 0, Readdirnames returns all the names from the directory in a single slice. In this case, if Readdirnames succeeds (reads all the way to the end of the directory), it returns the slice and a nil error. If it encounters an error before the end of the directory, Readdirnames returns the names read until that point and a non-nil error.

func (*File) Seek 

func (f *File) Seek(offset int64, whence int) (int64, error)

Seek sets the offset for the next Read or Write on file to offset, interpreted according to whence: 0 means relative to the origin of the file, 1 means relative to the current offset, and 2 means relative to the end. It returns the new offset and an error, if any. The behavior of Seek on a file opened with O_APPEND is not specified.

func (*File) SeekWithSize 

func (f *File) SeekWithSize(offset, size int64, whence int) (int64, error)

func (*File) Stat 

func (f *File) Stat() (os.FileInfo, error)

Stat returns the FileInfo structure describing file. If there is an error, it will be of type *PathError.

func (*File) Sync 

func (f *File) Sync() error

Sync is a noop.

func (*File) Truncate 

func (f *File) Truncate(int64) error

Truncate changes the size of the file. It does not change the I/O offset. If there is an error, it will be of type *PathError.

func (*File) Write 

func (f *File) Write(p []byte) (int, error)

Write writes len(b) bytes to the File. It returns the number of bytes written and an error, if any. Write returns a non-nil error when n != len(b).

func (*File) WriteAt 

func (f *File) WriteAt(p []byte, off int64) (n int, err error)

WriteAt writes len(p) bytes to the file starting at byte offset off. It returns the number of bytes written and an error, if any. WriteAt returns a non-nil error when n != len(p).

func (*File) WriteString 

func (f *File) WriteString(s string) (int, error)

WriteString is like Write, but writes the contents of string s rather than a slice of bytes.

type FileInfo 

type FileInfo struct {
	// contains filtered or unexported fields
}

FileInfo implements os.FileInfo for a file in S3.

func NewFileInfo 

func NewFileInfo(name string, directory bool, sizeInBytes int64, modTime time.Time) FileInfo

NewFileInfo creates file cachedInfo.

func (FileInfo) IsDir 

func (fi FileInfo) IsDir() bool

IsDir provides the abbreviation for Mode().IsDir()

func (FileInfo) ModTime 

func (fi FileInfo) ModTime() time.Time

ModTime provides the last modification time.

func (FileInfo) Mode 

func (fi FileInfo) Mode() os.FileMode

Mode provides the file mode bits. For a file in S3 this defaults to 664 for files, 775 for directories. In the future this may return differently depending on the permissions available on the bucket.

func (FileInfo) Name 

func (fi FileInfo) Name() string

Name provides the base name of the file.

func (FileInfo) Size 

func (fi FileInfo) Size() int64

Size provides the length in bytes for a file.

func (FileInfo) Sys 

func (fi FileInfo) Sys() interface{}

Sys provides the underlying data source (can return nil)

type Fs 

type Fs struct {
	FileProps *UploadedFileProperties // FileProps define the file properties we want to set for all new files
	// contains filtered or unexported fields
}

Fs is an FS object backed by S3.

func NewFs 

func NewFs(bucket string, session *session.Session) *Fs

NewFs creates a new Fs object writing files to a given S3 bucket.

func (Fs) Chmod 

func (fs Fs) Chmod(name string, mode os.FileMode) error

Chmod doesn't exist in S3 but could be implemented by analyzing ACLs

func (Fs) Chown 

func (Fs) Chown(string, int, int) error

Chown doesn't exist in S3 should probably NOT have been added to afero as it's POSIX-only concept.

func (Fs) Chtimes 

func (Fs) Chtimes(string, time.Time, time.Time) error

Chtimes could be implemented if needed, but that would require to override object properties using metadata, which makes it a non-standard solution

func (Fs) Create 

func (fs Fs) Create(name string) (afero.File, error)

Create a file.

func (Fs) Mkdir 

func (fs Fs) Mkdir(name string, perm os.FileMode) error

Mkdir makes a directory in S3.

func (Fs) MkdirAll 

func (fs Fs) MkdirAll(path string, perm os.FileMode) error

MkdirAll creates a directory and all parent directories if necessary.

func (Fs) Name 

func (Fs) Name() string

Name returns the type of FS object this is: Fs.

func (*Fs) Open 

func (fs *Fs) Open(name string) (afero.File, error)

Open a file for reading.

func (*Fs) OpenFile 

func (fs *Fs) OpenFile(name string, flag int, mode os.FileMode) (afero.File, error)

OpenFile opens a file .

func (*Fs) OpenFileWithRange added in v0.4.6 

func (fs *Fs) OpenFileWithRange(name string, flag int, mode os.FileMode, offset, size int64) (afero.File, error)

OpenFileWithRange opens a file with size.

func (*Fs) OpenWithRange added in v0.4.6 

func (fs *Fs) OpenWithRange(name string, offset, size int64) (afero.File, error)

OpenWithRange a file for reading with size.

func (Fs) Remove 

func (fs Fs) Remove(name string) error

Remove a file

func (*Fs) RemoveAll 

func (fs *Fs) RemoveAll(name string) error

RemoveAll removes a path.

func (Fs) Rename 

func (fs Fs) Rename(oldname, newname string) error

Rename a file. There is no method to directly rename an S3 object, so the Rename will copy the file to an object with the new name and then delete the original.

func (Fs) SetEnableCachedInfo added in v0.4.17 

func (fs Fs) SetEnableCachedInfo(v bool)

SetEnableCachedInfo ...

func (Fs) Stat 

func (fs Fs) Stat(name string) (os.FileInfo, error)

Stat returns a FileInfo describing the named file. If there is an error, it will be of type *os.PathError.

type UploadedFileProperties 

type UploadedFileProperties struct {
	ACL          *string // ACL defines the right to apply
	CacheControl *string // CacheControl defines the Cache-Control header
	ContentType  *string // ContentType define the Content-Type header
}

UploadedFileProperties defines all the set properties applied to future files

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.