fs

Documentation

Index

• Constants
• Variables
• func Dir(path string) fs.FS
• func MkdirAll(fsys fs.FS, name string) error
• func OpenFile(fsys fs.FS, name string, flag int, perm fs.FileMode) (fs.File, error)
• func Remove(fsys fs.FS, name string) error
• func Rename(fsys fs.FS, oldpath, newpath string) error
• func ValidName[T Name](name T) bool
• type BlobRepository
  • func NewBlobRepository[ID Name](fsys fs.FS) (*BlobRepository[ID], error)
  • func (r *BlobRepository[ID]) Count(ctx context.Context) (int64, error)
  • func (r *BlobRepository[ID]) Delete(ctx context.Context, id ID) error
  • func (r *BlobRepository[ID]) DeleteAll(ctx context.Context) error
  • func (r *BlobRepository[ID]) FindAll(ctx context.Context) (iter.Iterator[ID], error)
  • func (r *BlobRepository[ID]) FindByPrefix(ctx context.Context, prefix string) (iter.Iterator[ID], error)
  • func (r *BlobRepository[ID]) Read(ctx context.Context, id ID) (io.ReadCloser, error)
  • func (r *BlobRepository[ID]) Write(ctx context.Context, id ID) (io.WriteCloser, error)
• type MakeDirFileFS
• type Name
• type OpenFileFS
• type RemoveFileFS
• type RenameFileFS
• type Repository
  • func NewRepository[T any, ID comparable](fs fs.FS) (*Repository[T, ID], error)
• type SyncableFile
• type WriteFS
• type WriteableFile

Constants

const (
	MAX_PATH = 4096 - 1 // default linux limit excluding nul terminator
	NAME_MAX = 255
)

Variables

var FileOpenNotSupported = errors.New("fs does not support OpenFile")

var InvalidFilename = errors.New("invalid file name")

var MkDirNotSupported = errors.New("fs does not support mkdir")

var RemoveNotSupported = errors.New("fs does not support remove")

var RenameFileNotSupported = errors.New("fs does not support rename")

var WriteableFileNotSupported = errors.New("fs file does not write")

Functions

func Dir

func Dir(path string) fs.FS

Dir opens a read and writeable filesystem which implements a WriteFile method with transactional semantics.

func MkdirAll

func MkdirAll(fsys fs.FS, name string) error

MkdirAll tries to create the given hierarchy of directories.

func OpenFile

func OpenFile(fsys fs.FS, name string, flag int, perm fs.FileMode) (fs.File, error)

OpenFile tries open a file using posix style.

func Remove

func Remove(fsys fs.FS, name string) error

Remove tries to remove the named file.

func Rename

func Rename(fsys fs.FS, oldpath, newpath string) error

func ValidName

func ValidName[T Name](name T) bool

ValidName returns false, if name does not apply to our rules of a safe name:

• 255 bytes per segment (NAME_MAX)
• no multibyte, no unicode, only a-z | 0-9 | . | _ | - to avoid normalization case sensitivity issues
• at most 4096 (MAX_PATH - 1) // including nul
• prefix / directory separator is /

Our safe name rules represent more a or less the lowest subset of file names, which most common operating systems and storage apis support. This rules should be safe for windows, macos, linux and S3. For sure, Windows has some funny reserved names like LPT and COM etc. which are not checked.

Types

type BlobRepository

type BlobRepository[ID Name] struct {
	// contains filtered or unexported fields
}

BlobRepository provides a simple fs based (not yet standardized) repository. This repository works on POSIX systems and just relies on the filesystem for concurrent and atomic read/write semantics which probably do not work on Windows. Behavior is undefined, if a directory is shared between multiple repository instances. The binary bytes of the ID is taken, hex encoded and used as the file name. This works best with plain integers or byte arrays (like UUID). Storing plain strings works, but is inefficient. Actually, the file is saved in a one-level fanout structure using the first byte of the sha256 hash of the encoded ID, to support repository sizes with a million objects (boils down to 4000 files per fanout dir):

hex(sha256(binary(id)))[0])/hex(binary(id))".bin"

This implementation is mostly useful for prototyping and testing and shall not replace any serious SQL or NOSQL database. However, even though it may be slow, at least on POSIX it is considered to provide ACID properties.

func NewBlobRepository

func NewBlobRepository[ID Name](fsys fs.FS) (*BlobRepository[ID], error)

func (*BlobRepository[ID]) Count

func (r *BlobRepository[ID]) Count(ctx context.Context) (int64, error)

func (*BlobRepository[ID]) Delete

func (r *BlobRepository[ID]) Delete(ctx context.Context, id ID) error

func (*BlobRepository[ID]) DeleteAll

func (r *BlobRepository[ID]) DeleteAll(ctx context.Context) error

func (*BlobRepository[ID]) FindAll

func (r *BlobRepository[ID]) FindAll(ctx context.Context) (iter.Iterator[ID], error)

FindAll returns all blob identifiers. The current implementation buffers first the entire list of ids before the iterator becomes available. Files with a leading . are ignored.

func (*BlobRepository[ID]) FindByPrefix

func (r *BlobRepository[ID]) FindByPrefix(ctx context.Context, prefix string) (iter.Iterator[ID], error)

FindByPrefix is a special functions for this filesystem based implementation and allows to return a folder based prefix. To list the root, use '.' otherwise any ValidName denoting a directory is allowed. All contained files are returned recursively. Files with a leading . are ignored.

func (*BlobRepository[ID]) Read

func (r *BlobRepository[ID]) Read(ctx context.Context, id ID) (io.ReadCloser, error)

func (*BlobRepository[ID]) Write

func (r *BlobRepository[ID]) Write(ctx context.Context, id ID) (io.WriteCloser, error)

type MakeDirFileFS

type MakeDirFileFS interface {
	fs.FS

	// MkdirAll creates all folders, if required. If name denotes already directories, returns nil.
	MkdirAll(name string) error
}

MakeDirFileFS is the interface implemented by a file system that provides an optimized implementation of WriteFile.

type Name

type Name interface {
	~string
}

type OpenFileFS

type OpenFileFS interface {
	fs.FS

	// OpenFile is the Posix-style fopen thing.
	OpenFile(name string, flag int, perm fs.FileMode) (fs.File, error)
}

type RemoveFileFS

type RemoveFileFS interface {
	fs.FS

	// Remove unlinks or deletes the given file or directory.
	Remove(name string) error
}

type RenameFileFS

type RenameFileFS interface {
	fs.FS

	// Rename tries to perform an atomic rename if possible.
	Rename(oldpath, newpath string) error
}

type Repository

type Repository[T any, ID comparable] struct {
	// contains filtered or unexported fields
}

func NewRepository

func NewRepository[T any, ID comparable](fs fs.FS) (*Repository[T, ID], error)

type SyncableFile

type SyncableFile interface {
	WriteableFile
	Sync() error
}

type WriteFS

type WriteFS interface {
	fs.FS

	// Write writes the named file.
	Write(name string, w func(w io.Writer) error) error
}

WriteFS is the interface implemented by a file system that provides an optimized implementation of Write.

type WriteableFile

type WriteableFile interface {
	fs.File
	io.Writer
}