fsys

README

fsys

Local and remote Filesystem abstraction

Badges

Introduction

Fsys in an abstraction layer for interacting with local and remote Filesystems.

Implementations

• nop: Filesystem with sane defaults that does absolutely nothing.
• virtual: Filesystem that utilises memory
• local: Filesystem that utilises the local file system.
  • As a side note mmap can be used as an option for the local setup.

Documentation

Index

• func ErrNotFound(err error) bool
• type Config
  • func Build(opts ...Option) (*Config, error)
• type ConfigOption
  • func WithBucket(bucket string) ConfigOption
  • func WithEncryption(encryption bool) ConfigOption
  • func WithID(id string) ConfigOption
  • func WithKMSKey(kmsKey string) ConfigOption
  • func WithRegion(region string) ConfigOption
  • func WithSecret(secret string) ConfigOption
  • func WithServerSideEncryption(serverSideEncryption string) ConfigOption
  • func WithToken(token string) ConfigOption
• type File
• type Filesystem
  • func New(config *Config) (fsys Filesystem, err error)
  • func NewLocalFilesystem(mmap bool) Filesystem
  • func NewNopFilesystem() Filesystem
  • func NewRemoteFilesystem(config *RemoteConfig) (Filesystem, error)
  • func NewVirtualFilesystem() Filesystem
• type Option
  • func With(name string) Option
  • func WithConfig(remoteConfig *RemoteConfig) Option
  • func WithMMAP(mmap bool) Option
• type Releaser
• type RemoteAccessType
• type RemoteConfig
  • func BuildConfig(opts ...ConfigOption) (*RemoteConfig, error)
  • func (c RemoteConfig) String() string

Functions

func ErrNotFound

func ErrNotFound(err error) bool

ErrNotFound tests to see if the error passed is a not found error or not.

Types

type Config

type Config struct {
	// contains filtered or unexported fields
}

Config encapsulates the requirements for generating a Filesystem

func Build

func Build(opts ...Option) (*Config, error)

Build ingests configuration options to then yield a Config and return an error if it fails during setup.

type ConfigOption

type ConfigOption func(*RemoteConfig) error

ConfigOption defines a option for generating a RemoteConfig

func WithBucket

func WithBucket(bucket string) ConfigOption

WithBucket adds an Bucket option to the configuration

func WithEncryption

func WithEncryption(encryption bool) ConfigOption

WithEncryption adds an ID option to the configuration

func WithID

func WithID(id string) ConfigOption

WithID adds an ID option to the configuration

func WithKMSKey

func WithKMSKey(kmsKey string) ConfigOption

WithKMSKey adds a KMSKey option to the configuration

func WithRegion

func WithRegion(region string) ConfigOption

WithRegion adds an Region option to the configuration

func WithSecret

func WithSecret(secret string) ConfigOption

WithSecret adds an Secret option to the configuration

func WithServerSideEncryption

func WithServerSideEncryption(serverSideEncryption string) ConfigOption

WithServerSideEncryption adds a ServerSideEncryption option to the configuration

func WithToken

func WithToken(token string) ConfigOption

WithToken adds an Token option to the configuration

type File

type File interface {
	io.Reader
	io.Writer
	io.Closer

	// Name returns the name of the file
	Name() string

	// Size returns the size of the file
	Size() int64

	// Sync attempts to sync the file with the underlying storage or errors if it
	// can't not succeed.
	Sync() error

	// ContentType returns the ContentType of file. If the underlying storage type
	// doesn't support storing the ContentType then it will return the default
	// content type - application/octet-stream
	ContentType() string

	// SetContentType sets the contentType of the file. In some underlying
	// storage types this is a nop, and reading content from the file system will
	// be ignored.
	SetContentType(t string) error
}

File is an abstraction for reading, writing and also closing a file. These interfaces already exist, it's just a matter of composing them to be more usable by other components.

type Filesystem

type Filesystem interface {
	// Create takes a path, creates the file and then returns a File back that
	// can be used. This returns an error if the file can not be created in
	// some way.
	Create(string) (File, error)

	// Open takes a path, opens a potential file and then returns a File if
	// that file exists, otherwise it returns an error if the file wasn't found.
	Open(string) (File, error)

	// Rename takes a current destination path and a new destination path and will
	// rename the a File if it exists, otherwise it returns an error if the file
	// wasn't found.
	Rename(string, string) error

	// Exists takes a path and checks to see if the potential file exists or
	// not.
	// Note: If there is an error trying to read that file, it will return false
	// even if the file already exists.
	Exists(string) bool

	// Remote takes a path, removes a potential file, if no file doesn't exist it
	// will return not found.
	Remove(string) error

	// MkdirAll takes a path and generates a directory structure from that path,
	// if there is a failure it will return an error.
	MkdirAll(string) error

	// Chtimes updates the modifier times for a given path or returns an error
	// upon failure
	Chtimes(string, time.Time, time.Time) error

	// Walk over a specific directory and will return an error if there was an
	// error whilst walking.
	Walk(string, filepath.WalkFunc) error

	// Lock attempts to create a locking file for a given path.
	Lock(string) (Releaser, bool, error)
}

Filesystem is an abstraction over the native filesystem that allows us to create mock implementations for better testing.

func New

func New(config *Config) (fsys Filesystem, err error)

New creates a filesystem from a configuration or returns error if on failure.

func NewLocalFilesystem

func NewLocalFilesystem(mmap bool) Filesystem

NewLocalFilesystem yields a local disk filesystem.

func NewNopFilesystem

func NewNopFilesystem() Filesystem

NewNopFilesystem has methods that always succeed, but do nothing.

func NewRemoteFilesystem

func NewRemoteFilesystem(config *RemoteConfig) (Filesystem, error)

NewRemoteFilesystem creates a new remote file system that abstracts over a S3 bucket.

func NewVirtualFilesystem

func NewVirtualFilesystem() Filesystem

NewVirtualFilesystem yields an in-memory filesystem.

type Option

type Option func(*Config) error

Option defines a option for generating a filesystem Config

func With

func With(name string) Option

With adds a type of filesystem to use for the configuration.

func WithConfig

func WithConfig(remoteConfig *RemoteConfig) Option

WithConfig adds a remote filesystem config to the configuration

func WithMMAP

func WithMMAP(mmap bool) Option

WithMMAP defines if we should use mmap or not.

type Releaser

type Releaser interface {

	// Release given lock or returns error upon failure.
	Release() error
}

Releaser is returned by Lock calls.

type RemoteAccessType

type RemoteAccessType int

RemoteAccessType defines what type of access to the remote S3 is going to be utilized.

const (
	// KMSRemoteAccessType uses the KMS access
	KMSRemoteAccessType RemoteAccessType = iota

	// KeySecretRemoteAccessType uses the traditional Key Secret access
	KeySecretRemoteAccessType
)

type RemoteConfig

type RemoteConfig struct {
	Type                         RemoteAccessType
	KMSKey, ServerSideEncryption string
	ID, Secret, Token            string
	Region, Bucket               string
}

RemoteConfig creates a configuration to create a RemoteFilesystem.

func BuildConfig

func BuildConfig(opts ...ConfigOption) (*RemoteConfig, error)

BuildConfig ingests configuration options to then yield a RemoteConfig, and return an error if it fails during configuring.

func (RemoteConfig) String

func (c RemoteConfig) String() string