filestore

README

filestore: Go file storage implementations

Features

• Identify uploaded files as hashes (e.g. for reference in a database)
• Implementations
  • Local file storage (in a directory with subdirectories derived from hash prefix)
  • S3 based file storage (tested with MinIO and AWS S3)
  • Memory based file storage (for testing)

Scope

An abstracted file storage in webapps is the main use case. It is well suited to handle file uploads, retrieval and deletion. With the integration of image processing like imgproxy it provides a simple and efficient way to handle files and images in webapps. When storing files in S3 a webapp can be run stateless which simplifies container deployments.

Install

go get github.com/networkteam/filestore

Usage

Local filestore

package main

import (
	"context"
	"fmt"
	"io"
	"log"
	"strings"

	"github.com/networkteam/filestore/local"
)

func main() {
	ctx := context.Background()

	fStore, err := local.NewFilestore("./tmp", "./assets")
	if err != nil {
		log.Fatal(err)
	}

	// Storing
	hash, err := fStore.Store(ctx, strings.NewReader("Hello World"))
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(hash)

	// Fetching
	r, err := fStore.Fetch(ctx, hash)
	if err != nil {
		log.Fatal(err)
	}
	defer r.Close()
	content, err := io.ReadAll(r)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(string(content))
}

S3 filestore

package main

import (
  "context"
  "fmt"
  "io"
  "log"
  "strings"

  "github.com/networkteam/filestore/s3"
)

func main() {
  ctx := context.Background()

  fStore, err := s3.NewFilestore(
    ctx,
    "s3.eu-central-1.amazonaws.com",
    "my-bucket",
    s3.WithCredentialsV4("my-access-key", "********", ""),
  )
  if err != nil {
    log.Fatal(err)
  }

  // Storing
  text := "Hello World"
  sr := strings.NewReader(text)
  // Wrap reader with s3.SizedReader to set content length in advance
  hash, err := fStore.Store(ctx, s3.SizedReader(sr, int64(len(text))))
  if err != nil {
    log.Fatal(err)
  }
  fmt.Println(hash)

  // Fetching
  r, err := fStore.Fetch(ctx, hash)
  if err != nil {
    log.Fatal(err)
  }
  defer r.Close()
  content, err := io.ReadAll(r)
  if err != nil {
    log.Fatal(err)
  }
  fmt.Println(string(content))
}

Dependencies

The filestore module provides each implementation in its own package to reduce the amount of transitive dependencies (e.g. you don't need a S3 client if not using s3.Filestore).

Development

Tests

S3 filestore

• S3 filestore will use a fake S3 server by default
• An external S3 server and bucket can be used by setting the environment variables S3_ENDPOINT, S3_BUCKET, S3_ACCESS_KEY and S3_SECRET_KEY
• Tests can be run against AWS S3 like this (make sure to allow sufficient access to bucket):

  S3_ENDPOINT=s3.eu-central-1.amazonaws.com S3_BUCKET=my-bucket-name S3_ACCESS_KEY=my-access-key S3_SECRET_KEY=******** go test
  

• Tests can be run against a MinIO server like this:

  S3_ENDPOINT=localhost:9000 S3_BUCKET=my-bucket-name S3_ACCESS_KEY=my-access-key S3_SECRET_KEY=******** go test
  

License

MIT

Documentation

Index

• Variables
• type Exister
• type Fetcher
• type FileStore
• type HashedStorer
• type ImgproxyURLSourcer
• type Iterator
• type Remover
• type Sizer
• type Storer

Variables

var ErrNotExist = errors.New("file does not exist")

ErrNotExist is returned when a stored file does not exist.

Types

type Exister

type Exister interface {
	Exists(ctx context.Context, hash string) (bool, error)
}

An Exister checks if the given hash exists in the store.

type Fetcher

type Fetcher interface {
	Fetch(ctx context.Context, hash string) (io.ReadCloser, error)
}

A Fetcher fetches the content of the given hash in the form of an io.Reader.

type FileStore

type FileStore interface {
	Storer
	HashedStorer
	Exister
	Fetcher
	Iterator
	Remover
	Sizer
	ImgproxyURLSourcer
}

A FileStore bundles all the interfaces above.

type HashedStorer

type HashedStorer interface {
	StoreHashed(ctx context.Context, r io.Reader, hash string) error
}

A HashedStorer stores the content of the given reader (e.g. a file) with a pre-calculated hash. The hash can be chosen freely and is not checked against the reader content.

type ImgproxyURLSourcer

type ImgproxyURLSourcer interface {
	// ImgproxyURLSource gets the source URL to original file (e.g. for use with imgproxy).
	ImgproxyURLSource(hash string) (string, error)
}

An ImgproxyURLSourcer can return the source URL to original file for imgproxy.

type Iterator

type Iterator interface {
	// Iterate calls callback with a maxBatch number of asset hashes.
	// If callback returns an error, iteration stops and the error is returned.
	Iterate(ctx context.Context, maxBatch int, callback func(hashes []string) error) error
}

An Iterator iterates over all stored files and returns their hashes in batches.

type Remover

type Remover interface {
	Remove(ctx context.Context, hash string) error
}

A Remover can remove a file with the given hash.

type Sizer

type Sizer interface {
	Size(ctx context.Context, hash string) (int64, error)
}

A Sizer can return the size of a file with the given hash.

type Storer

type Storer interface {
	Store(ctx context.Context, r io.Reader) (hash string, err error)
}

A Storer stores the content of the given reader (e.g. a file) and returns a consistent hash for later retrieval.