storage

README

Storage utils

The goal with this package was a basic wrapper for manipulating storage backends without needing to implement every client type and operation in our core code base. Today we have a pretty decent pattern around instantiating "Managers" in the server opts -> handlers, as well as the ability to add the dependencies in entc to be accessed within our ent storage layer.

All of the interfaces in this package will be for remote object storage or basic file system storage / manipulation - not stored within our actual schemas. This means additional work will need to be done to create pointers to where the objects are so they can be referenced alongside our organization ID's or user ID's with the paths.

Usage

With S3:

storage, _ := s3.NewStorage(s3.Config{
    AccessKeyID:     accessKeyID,
    SecretAccessKey: secretAccessKey,
    Region:          region,
    Bucket:          bucket,
})

// Saving a file named mitb
storage.Save(ctx, strings.NewReader("HELLOOOOOOOOOOO"), "mitb")

// Deleting the new file
storage.Delete(ctx, "mitb")

With a file system:

dir := os.TempDir()
storage := fs.NewStorage(fs.Config{Root: dir})

// Saving a file named test
storage.Save(ctx, strings.NewReader("(╯°□°）╯︵ ┻━┻"), "test")

// Deleting the new file
storage.Delete(ctx, "test")

TODO

Aside from, like, supporting every storage interface available (super reasonable to do) the main TO-DO's related to some solid usage with our ent template schema and the main ecosystem of tools we have:

• "Fetch" functionality (open+write to destination)
• Additional abstraction with spf13/afero

Nice to have's

• Benchmarks
• Throughput tests
• Supported content types / file types filters
• Uploader utilities + wrappers
• Ent document ID + object ID mappers
• Compression?
• Encryption

Documentation

Overview

Package storage provides a methods and interfaces to access various storage systems and objects

Index

• Variables
• func ComputeChecksum(data io.ReadSeeker) (string, error)
• func DetectContentType(data io.ReadSeeker) (string, error)
• func UnmarshalTo(body io.ReadCloser, to interface{}) error
• func UnmarshalToRawMessages(body io.ReadCloser) ([]json.RawMessage, error)
• type Stat
• type Storage

Variables

var (
	// ErrNotExist is returned when a path does not exist
	ErrNotExist = errors.New("does not exist")

	// ErrCouldNotSeekFile is returned when a file could not be seeked
	ErrCouldNotSeekFile = errors.New("could not seek file")

	// ErrCouldNotReadFile is returned when a file could not be read
	ErrCouldNotReadFile = errors.New("could not read file")

	// ErrCouldNotWriteFile is returned when a file could not be written
	ErrCouldNotWriteFile = errors.New("could not write file")

	// ErrCouldNotDetectContentType is returned when a content type could not be detected
	ErrCouldNotDetectContentType = errors.New("could not detect content type")
)

Functions

func ComputeChecksum

func ComputeChecksum(data io.ReadSeeker) (string, error)

ComputeChecksum calculates the MD5 checksum for the provided data. It expects that the passed io object will be seeked to its beginning and will seek back to the beginning after reading its content

func DetectContentType

func DetectContentType(data io.ReadSeeker) (string, error)

DetectContentType leverages http.DetectContentType to identify the content type of the provided data. It expects that the passed io object will be seeked to its beginning and will seek back to the beginning after reading its content

func UnmarshalTo

func UnmarshalTo(body io.ReadCloser, to interface{}) error

Unmarshals the content of the given body and stores it in the provided pointer

func UnmarshalToRawMessages

func UnmarshalToRawMessages(body io.ReadCloser) ([]json.RawMessage, error)

unmarshalToRawMessages unmarshals the content of the given body and stores it in a juicy slice of json.RawMessage

Types

type Stat

type Stat struct {
	ModifiedTime time.Time
	Size         int64
}

Stat contains metadata about content stored in storage size and last modified time aren't fetched every time for lower processing overhead

type Storage

type Storage interface {
	// Delete deletes the content at the given path
	Delete(ctx context.Context, path string) error
	// FileSystem returns the underlying filesystem
	FileSystem() *afero.Afero
	// Open opens a reader for the content at the given path
	Open(ctx context.Context, path string) (io.ReadCloser, error)
	// OpenWithStat opens a reader for the content at the given path and returns the metadata
	OpenWithStat(ctx context.Context, path string) (io.ReadCloser, *Stat, error)
	// Save saves content to path
	Save(ctx context.Context, content io.Reader, path string) error
	// Stat returns path metadata
	Stat(ctx context.Context, path string) (*Stat, error)
	// Tags returns the tags for the given path
	Tags(string) (map[string]string, error)
	// TempFileSystem returns a temporary filesystem
	TempFileSystem() *afero.Afero
}

Storage is the package storage interface wrapper for absctracting the various storage backends