marshalfs

README

MarshalFS [a go package]

Simulate a 'readonly' filesystem, backed by serializable objects. Supply a marshaler(s) so that calling code can read your files via a standard io.Reader.

Note that although fs.FS is a read-only interface ,you can update marshalfs's backing objects via non-standard methods (SetFile/Remove/ReplaceAll). Also, each time you open a file, it will re-marshal the backing object. MarshalFS also uses a sync.RWMutex to provide some concurrency safety.

marshalfs only works with Go 1.16+. It can be thought of as a riff on fstest.MapFS.

Why for?

Haven't you heard, everything is a file?

Mainly, for testing, and for accessing any data source as though it were a file.

I can think of a bunch of uses for a read-only filesystem:

• Testing:
• Testing config parsing.
• Injecting config into tests.
• Simulate file changes over time.
• Imitate a serial interface or some other filesystem-based resource.
• Reading a completely different data source, as though it were a file (TODO: example needed - prolly some helpers too).
• Optionally, overlay this filesystem over a real os.DirFS filesystem, or any other fs.FS, using mergefs.

Last but not least, if you just want to implement an exotic fs.FS filesystem, then marshalfs does some of the harder stuff for you.

For testing Config

Test your config parsing without actually storing 'fixture' files on the filesystem. ...

• e.g.: testing config files ... See Example_forConfig() for a demonstration
• e.g.2: injecting config data without writing directly to the filesystem:

  mfs, err := marshalfs.New(json.Marshal, marshalfs.FilePaths{
      "config.json": marshalfs.NewFile(&myconfig{Env: "production", I: 3}),
      "config-staging.json": marshalfs.NewFile(&myconfig{Env: "staging", I: 2}),
      "config.yaml": marshalfs.NewFile(&myconfig{S: "production", I: 3}, marshalfs.WithMarshaler(yaml.Marshal)),
    })

Marshalers

Known usages or examples of use. ...

Please contribute by sending a PR with a link to an example.

Marshaler	Verified	Notes
json	[x]
yaml	[x]
xml	[ ]
asn1	[ ]
toml	[ ]
toml	[ ]
ini	[ ]
csv	[ ]

Caveats

• This implementation is NOT computationally efficient. It keeps entire objects in RAM, and bytes in RAM too.
• fs.FS is a read-only API. In the standard sense, so is this, currently.
• The backing objects can change each time you open them, though
• You can update the backing objects using marshalfs.FS.SetFile()/marshalfs.FS.Remove()/marshalfs.FS.ReplaceAll()
  • ReplaceAll is a good option if you want to maintain your map outside of marshalfs.

Incomplete plans

• Support for a writable FS will likely be postponed until fs.FS supports writable files. The eventual design is unknown.
  • Probably something like WithUnmarshaler(json.Unmarshal).
• Helpers for 'dynamically updating objects':
  • Maybe some helpers for "file generators"
• Maybe somehow copy mergefs into here?

Related Works

• Standard Library:
  • os.DirFS contains os.DirFS - this 'default' implementation is backed by an actual filesystem.
  • fstest.MapFS contains a memory-map implementation and a testing tool. The standard library contains a few other fs.FS implementations (like 'zip')
  • embed.FS provides access to files embedded in the running Go program.
• An earlier work, afero is a filesystem abstraction for Go, which has been the standard for filesystem abstractions up until go1.15. It's read-write in the usual sense (io.Writer), and it's a mature project. The interfaces look very different (lots of methods), so it's not really compatible.
• s3fs is a fs.FS backed by the AWS S3 client
• mergefs merge fs.FS filesystems together so that your FS can easily read from multiple sources.
• hashfs appends SHA256 hashes to filenames to allow for aggressive HTTP caching.

Documentation

Overview

Example (ForConfig)

package main

import (
	"encoding/json"
	"io/fs"
	"io/ioutil"
	"log"
	"reflect"

	"github.com/laher/marshalfs"
)

func main() {
	// Given a config which is usually loaded from a file, ...
	type myconfig struct {
		I int    `json:"i"`
		S string `json:"s"`
	}

	// Here is the code under test
	// NOTE: production code would invoke it with os.DirFS
	// `config := loadConfig(os.DirFS("./config"))`
	var loadMyconfig = func(myfs fs.FS) (*myconfig, error) {
		f, err := myfs.Open("config.json")
		if err != nil {
			return nil, err
		}
		b, err := ioutil.ReadAll(f)
		if err != nil {
			return nil, err
		}
		c := &myconfig{}
		err = json.Unmarshal(b, c)
		return c, err
	}

	// Set up ...
	input := &myconfig{S: "string", I: 3}
	mfs, err := marshalfs.New(json.Marshal, marshalfs.FileSpecs{"config.json": marshalfs.NewFile(input)})
	if err != nil {
		log.Fatalf("unexpected error: %v", err)
	}

	// Run the code
	output, err := loadMyconfig(mfs)
	// Verify file is loaded OK and content matches ...
	if err != nil {
		log.Fatalf("unexpected error: %v", err)
	}
	if !reflect.DeepEqual(input, output) {
		log.Fatal("loadConfig did not parse files as expected")
	}
}

Index

• Variables
• type FS
  • func New(defaultMarshaler MarshalFunc, files FileSpecs) (*FS, error)
  • func (mfs *FS) Glob(pattern string) ([]string, error)
  • func (mfs *FS) Open(name string) (fs.File, error)
  • func (mfs *FS) ReadDir(name string) ([]fs.DirEntry, error)
  • func (mfs *FS) ReadFile(name string) ([]byte, error)
  • func (mfs *FS) Remove(filename string)
  • func (mfs *FS) ReplaceAll(files FileSpecs) error
  • func (mfs *FS) SetFile(filename string, item FileSpec) error
  • func (mfs *FS) Stat(name string) (fs.FileInfo, error)
  • func (mfs *FS) Sub(dir string) (fs.FS, error)
• type FileCommon
• type FileOption
  • func WithMarshaler(mf MarshalFunc) FileOption
  • func WithModTime(t time.Time) FileOption
  • func WithMode(mode fs.FileMode) FileOption
• type FileSpec
  • func NewFile(value interface{}, opts ...FileOption) FileSpec
• type FileSpecs
• type MarshalFunc

Examples

• Package (ForConfig)

Variables

var ErrPathConflict = errors.New("path conflict")

Types

type FS

type FS struct {
	// contains filtered or unexported fields
}

An FS is a simple filesystem backed by objects and some serialization function. Given that files are backed by objects, this is only writable using non-standard methods - see "SetFile"/"Remove"/"ReplaceAll"

func New

func New(defaultMarshaler MarshalFunc, files FileSpecs) (*FS, error)

func (*FS) Glob

func (mfs *FS) Glob(pattern string) ([]string, error)

func (*FS) Open

func (mfs *FS) Open(name string) (fs.File, error)

Open opens the named file.

func (*FS) ReadDir

func (mfs *FS) ReadDir(name string) ([]fs.DirEntry, error)

func (*FS) ReadFile

func (mfs *FS) ReadFile(name string) ([]byte, error)

func (*FS) Remove

func (mfs *FS) Remove(filename string)

func (*FS) ReplaceAll

func (mfs *FS) ReplaceAll(files FileSpecs) error

func (*FS) SetFile

func (mfs *FS) SetFile(filename string, item FileSpec) error

SetFile is similar to os.WriteFile, except it takes a FileSpec instead of `[]byte, mode`

func (*FS) Stat

func (mfs *FS) Stat(name string) (fs.FileInfo, error)

func (*FS) Sub

func (mfs *FS) Sub(dir string) (fs.FS, error)

type FileCommon

type FileCommon struct {
	Mode    fs.FileMode // FileInfo.Mode
	ModTime time.Time   // FileInfo.ModTime
	Sys     interface{} // FileInfo.Sys
	// contains filtered or unexported fields
}

type FileOption

type FileOption func(*FileCommon)

func WithMarshaler

func WithMarshaler(mf MarshalFunc) FileOption

func WithModTime

func WithModTime(t time.Time) FileOption

func WithMode

func WithMode(mode fs.FileMode) FileOption

type FileSpec

type FileSpec interface {
	Common() FileCommon
	// contains filtered or unexported methods
}

func NewFile

func NewFile(value interface{}, opts ...FileOption) FileSpec

NewFile creates a new FileSpec

type FileSpecs

type FileSpecs map[string]FileSpec

type MarshalFunc

type MarshalFunc func(i interface{}) ([]byte, error)