memfs

package
v0.60.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 1, 2025 License: BSD-3-Clause Imports: 25 Imported by: 6

Documentation

Overview

Package memfs provide a library for mapping file system into memory and/or to embed it inside go source file.

Usage

The first step is to create new instance of memfs using New function. The following example embed all files inside directory named "include" or any files with extension ".css", ".html", and ".js"; but exclude any files inside directory named "exclude". 

opts := &Options{
	Root: "./mydir",
	Includes: []string{
		`.*/include`,
		`.*\.(css|html|js)$`,
	},
	Excludes: []string{
		`.*/exclude`,
	},
}
mfs, err := memfs.New(opts)

By default only file with size less or equal to 5 MB will be included in memory. To increase the default size set the memfs.Options.MaxFileSize (in bytes). For example, to change maximum file size to 10 MB, 

var opts = memfs.Options{
	MaxFileSize: 1024 * 1024 * 10,
}

Later, if we want to get the file from memory, call MemFS.Get which will return the Node object with content can be accessed from field "Content". If file size is larger than maximum, the content will need to be read manually (as long as the file exist on system), 

node, err := mfs.Get("/")
if err != nil {
	// Handle file not exist
}
if node.mode.IsDir() {
	// Handle directory.
}
if node.Content == nil {
	// Handle large file.
	node.V, err = os.ReadFile(child.SysPath)
}
// Do something with content of file system.

Go embed

The memfs package support embedding the files into Go generated source code. After we create the MemFS instance, call the memfs.GoEmbed method to dump all directory and files into Go source code.

First, define global variable as container for all files to be embedded in the same package as generated code, 

package mypackage

var myFS *memfs.MemFS

Next, create new instance of MemFS with memfs.Options.Embed is set, 

var opts = &Options{
	Embed: EmbedOptions{
		PackageName:     `mypackage`,
		VarName:         `myFS`,
		GoFileName:      `mypackage/embed.go`,
	},
	Root: `./mydir`,
	Includes: []string{
		`.*/include`,
		`.*\.(css|html|js)$`,
	},
	Excludes: []string{
		`.*/exclude`,
	},
}

var mfs *memfs.MemFS
mfs, err = memfs.New(opts)
...

Finally, call method MemFS.GoEmbed from the instance, 

mfs.GoEmbed()

This method will create Go file "mypackage/embed.go" that contains all path and content of files inside the mfs instance, under package named "mypackage". Code that can read "myFS" variable then can access any files using MemFS.Get method, with "/" as root prefix (not "./mydir").

Comparison with builtin go:embed

This section list the disadvantages of "go:embed" directive.

The memfs package created on November 2018, based on my experiences maintains the fork of go-bindata project. The "go:embed" directive introduced into Go tools since Go version 1.16, released February 2021, three years after the first release of memfs package.

Given the following directory structure, 

module-root/
+-- cmd/prog/main.go
+-- _content/
     +-- index.adoc
     +-- index.html
     +-- static/
         +-- index.png
         +-- index.png

We want to embed the directory "_content" but only html files and all files inside the "static/" directory.

Cons #1: The "go:embed" only works if files or directory to be embedded is in the same parent directory.

The "go:embed" directive define in "cmd/prog/main.go" will not able to embed files in their parent. The following code will not compile, 

//go:embed ../../_content/*.html
//go:embed ../../_content/static
var contentFS embed.FS

// go build output,
// pattern ../../_content/*.html: invalid pattern syntax

If we remove the ".." and execute "go build" from module-root, it will still not compile, 

//go:embed _content/*.html
//go:embed _content/static
var contentFS embed.FS

// go build or run output,
// pattern _content/*.html: no matching files found

The only solution is to create and export the variable "ContentFS" in the same parent directory as "_content".

The memfs package does not have this limitation. As long as the Go commands are executed from the module-root directory, you can define the variable in any packages.

Cons #2: Accessing the embedded file require the original path.

Let say we have embeded the "_content" directory using the following syntax, 

//go:embed _content/*.html
//go:embed _content/static
var ContentFS embed.FS

To access the file "index.html" you need to pass their full path, in this case "_content/index.html". The path "_content" leaked to the parent FS and not portable.

In the memfs package, the content of [Options.Root] directory can be accessed with "/", so it would become "/index.html". This design allow flexibility and consistency between modules and packages. If an external, third-party package accept the MemFS instance and the first thing they do is to read all contents of "/" directory, the caller can embed any path without have specific prefix or name.

Case example, when we embed SQL files for migration under directory "db/migration" using the "go:embed" directive, 

//go:embed db/migration/*.sql
var DBMigrationFS embed.FS

and then call the Migrate function, it cannot found any ".sql" files inside the "/" directory because the files is stored under "db/migration/" prefix.

Cons #3: No development mode.

Let say we run our server that served the content from FS instance. If we changes the html files, and refresh the browser, the new content will not reflected because it serve the content on the first embed.

The memfs package have [Options.TryDirect] that bypass file in memory and read directly to the file system. This allow quick development when changes only template or non-code files.

Index

Examples

Constants

View Source 
const DefaultEmbedGoFileName = `memfs_generate.go`

DefaultEmbedGoFileName default file output for GoEmbed.

View Source 
const DefaultEmbedPackageName = `main`

DefaultEmbedPackageName default package name for GoEmbed.

View Source 
const DefaultEmbedVarName = `memFS`

DefaultEmbedVarName default variable name for GoEmbed.

View Source 
const DefaultWatchFile = `.memfs_rescan`

DefaultWatchFile define default file name to be watch for changes. Any update to this file will trigger rescan on the memfs tree.

Variables

This section is empty.

Functions

This section is empty.

Types

type EmbedOptions 

type EmbedOptions struct {
	// CommentHeader define optional comment to be added to the header of
	// generated file, for example copyright holder and/or license.
	// The string value is not checked, whether it's a comment or not, it
	// will rendered as is.
	//
	// Due to templating, the value MUST be set using raw
	// string literal syntax “, NOT "".
	CommentHeader string

	// The generated package name for GoEmbed().
	// If its not defined it will be default to "main".
	PackageName string

	// VarName is the global variable name with type *memfs.MemFS which
	// will be initialized by generated Go source code on init().
	// If its empty it will default to "memFS".
	VarName string

	// GoFileName the path to Go generated file, where the file
	// system will be embedded.
	// If its not defined it will be default to "memfs_generate.go"
	// in current directory from where its called.
	GoFileName string

	// WithoutModTime if its true, the modification time for all
	// files and directories are not stored inside generated code, instead
	// all files will use the current time when the program is running.
	WithoutModTime bool
}

EmbedOptions define an options for GoEmbed.

type MemFS 

type MemFS struct {
	PathNodes *PathNode
	Root      *Node
	Opts      *Options
	// contains filtered or unexported fields
}

MemFS contains directory tree of file system in memory.

func New 

func New(opts *Options) (mfs *MemFS, err error)

New create and initialize new memory file system from directory Root using list of regular expresssion for Including or Excluding files.

Example 
package main

import (
	"fmt"
	"log"

	"git.sr.ht/~shulhan/pakakeh.go/lib/memfs"
)

func main() {
	/**
	Let say we have the "testdata" directory,

		testdata/
		├── direct
		│   └── add
		│       ├── file
		│       └── file2
		├── exclude
		│   ├── dir
		│   ├── index-link.css -> ../index.css
		│   ├── index-link.html -> ../index.html
		│   └── index-link.js -> ../index.js
		├── include
		│   ├── dir
		│   ├── index.css -> ../index.css
		│   ├── index.html -> ../index.html
		│   └── index.js -> ../index.js
		├── index.css
		├── index.html
		├── index.js
		└── plain

	Assume that we want to embed all files with extension .css, .html,
	and .js only; but not from directory "exclude".
	We can create the Options like below,
	*/
	opts := &memfs.Options{
		Root:     `./testdata`,
		Includes: []string{`.*/include`, `.*\.(css|html|js)$`},
		Excludes: []string{`.*/exclude`},
	}
	mfs, err := memfs.New(opts)
	if err != nil {
		log.Fatal(err)
	}

	node, err := mfs.Get("/index.html")
	if err != nil {
		log.Fatal(err)
	}

	fmt.Printf("Content of /index.html: %s", node.Content)

	fmt.Printf("List of embedded files: %+v\n", mfs.ListNames())

	_, err = mfs.Get("/exclude/index.html")
	if err != nil {
		fmt.Println(`Error:`, err)
	}

}

Output:

Content of /index.html: <html></html>
List of embedded files: [/ /direct /direct/add /include /include/dir /include/index.css /include/index.html /include/index.js /index.css /index.html /index.js]
Error: Get "/exclude/index.html": file does not exist

func (*MemFS) AddChild 

func (mfs *MemFS) AddChild(parent *Node, fi os.FileInfo) (child *Node, err error)

AddChild add FileInfo fi as new child of parent node.

It will return nil without an error if,

  • the system path of parent+fi.Name() is excluded by one of Options.Excludes pattern, or
  • fi is symlink to not existen node.

func (*MemFS) AddFile 

func (mfs *MemFS) AddFile(internalPath, externalPath string) (node *Node, err error)

AddFile add the external file directly as internal file. If the internal file is already exist it will be replaced. Any directories in the internal path will be generated automatically if its not exist.

func (*MemFS) Get 

func (mfs *MemFS) Get(path string) (node *Node, err error)

Get the node representation of file in memory. If path is not exist it will return fs.ErrNotExist.

func (*MemFS) GoEmbed 

func (mfs *MemFS) GoEmbed() (err error)

GoEmbed write the tree nodes as Go generated source file. This method assume that the files inside the mfs instance is already up-to-date. If you are not sure, call Remount.

func (*MemFS) Init 

func (mfs *MemFS) Init() (err error)

Init initialize the MemFS instance. This method provided to initialize MemFS if its Options is set directly, not through New() function.

func (*MemFS) ListNames 

func (mfs *MemFS) ListNames() (paths []string)

ListNames list all files in memory sorted by name.

func (*MemFS) MarshalJSON 

func (mfs *MemFS) MarshalJSON() ([]byte, error)

MarshalJSON encode the MemFS object into JSON format.

The field that being encoded is the Root node.

func (*MemFS) Merge 

func (mfs *MemFS) Merge(sub *MemFS)

Merge other MemFS instance as sub file system.

When Get method called, each sub fs will be evaluated in order of Merge.

func (*MemFS) MustGet 

func (mfs *MemFS) MustGet(path string) (node *Node)

MustGet return the Node representation of file in memory by its path if its exist or nil the path is not exist.

func (*MemFS) Open 

func (mfs *MemFS) Open(path string) (http.File, error)

Open the named file for reading. This is an alias to Get() method, to make it implement http.FileSystem.

func (*MemFS) Remount 

func (mfs *MemFS) Remount() (err error)

Remount reset the memfs instance to force rescanning the files again from file system.

func (*MemFS) RemoveChild 

func (mfs *MemFS) RemoveChild(parent *Node, child *Node) (removed *Node)

RemoveChild remove a child on parent, including its map on PathNode. If child is not part if node's childrens it will return nil.

func (*MemFS) Search 

func (mfs *MemFS) Search(words []string, snippetLen int) (results []SearchResult)

Search one or more strings in each content of files.

Example 
package main

import (
	"fmt"
	"log"

	"git.sr.ht/~shulhan/pakakeh.go/lib/memfs"
)

func main() {
	opts := &memfs.Options{
		Root: `./testdata`,
	}
	mfs, err := memfs.New(opts)
	if err != nil {
		log.Fatal(err)
	}

	q := []string{`body`}
	results := mfs.Search(q, 0)

	for _, result := range results {
		fmt.Println(`Path:`, result.Path)
		fmt.Printf("Snippets: %q\n", result.Snippets)
	}
}

Output:

Path: /include/index.css
Snippets: ["body {\n}\n"]
Path: /exclude/index-link.css
Snippets: ["body {\n}\n"]
Path: /index.css
Snippets: ["body {\n}\n"]

func (*MemFS) StopWatch 

func (mfs *MemFS) StopWatch()

StopWatch stop watching for update, from calling Watch.

func (*MemFS) Update 

func (mfs *MemFS) Update(node *Node, newInfo os.FileInfo)

Update the node content and information in memory based on new file information. This method only check if the node name is equal with file name, but it's not checking whether the node is part of memfs (node is parent or have the same Root node).

func (*MemFS) Watch 

func (mfs *MemFS) Watch(watchopts WatchOptions) (
	changes <-chan []*Node, err error,
)

Watch create and start the watchfs/v2.DirWatcher that re-scan the content of Root directory recursively on every memfs.WatchOptions.Interval, triggered by changes on memfs.WatchOptions.File.

The watcher will remove or update the tree and node content automatically if the included files is being deleted, created, or updated.

The returned channel changes return list of Node that has been deleted, created, or updated. To stop watching for update call the MemFS.StopWatch.

Example 
var (
	opts memfs.Options
	err  error
)

opts.Root, err = os.MkdirTemp(``, `ExampleMemFS_Watch`)
if err != nil {
	log.Fatal(err)
}

defer func() {
	_ = os.RemoveAll(opts.Root)
}()

var mfs *memfs.MemFS

mfs, err = memfs.New(&opts)
if err != nil {
	log.Fatal(err)
}

var fileToWatch = filepath.Join(opts.Root, memfs.DefaultWatchFile)
var watchOpts = memfs.WatchOptions{
	FileWatcherOptions: watchfs.FileWatcherOptions{
		File:     fileToWatch,
		Interval: 200 * time.Millisecond,
	},
	Verbose: true,
}

var changesq <-chan []*memfs.Node

changesq, err = mfs.Watch(watchOpts)
if err != nil {
	log.Fatal(err)
}

var testFile = filepath.Join(opts.Root, `file`)
err = os.WriteFile(testFile, []byte(`dummy content`), 0600)
if err != nil {
	log.Fatal(err)
}

err = os.WriteFile(fileToWatch, []byte(`x`), 0600)
if err != nil {
	log.Fatal(err)
}
<-changesq

_, err = mfs.Get(`/file`)
if err != nil {
	log.Fatal(err)
}

err = os.Remove(testFile)
if err != nil {
	log.Fatal(err)
}

err = os.WriteFile(fileToWatch, []byte(`xx`), 0600)
if err != nil {
	log.Fatal(err)
}
<-changesq
mfs.StopWatch()
<-changesq

Output:

MemFS: file created: "/file"
MemFS: file deleted: "/file"

type Node 

type Node struct {
	Parent *Node // Pointer to parent directory.

	SysPath string // The original file path in system.
	Path    string // Absolute file path in memory.

	ContentType string // File type per MIME, for example "application/json".
	GenFuncName string // The function name for embedded Go code.

	Childs []*Node // List of files in directory.

	Content []byte // Content of file.
	// contains filtered or unexported fields
}

Node represent a single file.

This Node implement os.FileInfo and http.File.

func NewNode 

func NewNode(parent *Node, fi os.FileInfo, maxFileSize int64) (node *Node, err error)

NewNode create a new node based on file information "fi".

The parent parameter is required to allow valid system path generated for new node.

If maxFileSize is greater than zero, the file content and its type will be saved in node as Content and ContentType.

func (*Node) AddChild 

func (node *Node) AddChild(child *Node)

AddChild add the other node as child of this node.

func (*Node) Child 

func (node *Node) Child(name string) (cnode *Node)

Child return the child node based on its name.

func (*Node) Close 

func (node *Node) Close() error

Close reset the offset position back to zero.

func (*Node) GenerateIndexHTML 

func (node *Node) GenerateIndexHTML()

GenerateIndexHTML generate simple directory listing as HTML for all childs in this node. This method is only applicable if node is a directory.

func (*Node) IsDir 

func (node *Node) IsDir() bool

IsDir return true if the node is a directory.

func (*Node) JSON 

func (node *Node) JSON(depth int, withContent, withModTime bool) (rawjson []byte, err error)

JSON encode the Node into JSON. This method provides an alternative to MarshalJSON with granular options.

The depth set the level of childs to be encoded. The depth=0 only encode the Root Node itself (with its childs), depth=1 encode the Root node and its subdirectories, and so on.

If the withContent set to true, all of the Node content will not be included in the output.

If the withModTime is set to true, all of the node ModTime will not be included in the output.

func (*Node) MarshalJSON 

func (node *Node) MarshalJSON() ([]byte, error)

MarshalJSON encode the node into JSON format. If the node is a file it will return the content of file; otherwise it will return the node with list of childs, but not including childs of childs.

func (*Node) ModTime 

func (node *Node) ModTime() time.Time

ModTime return the node modification time.

func (*Node) Mode 

func (node *Node) Mode() os.FileMode

Mode return the node file mode.

func (*Node) Name 

func (node *Node) Name() string

Name return the node (file) name.

func (*Node) Read 

func (node *Node) Read(p []byte) (n int, err error)

Read the content of node into p.

func (*Node) Readdir 

func (node *Node) Readdir(count int) (fis []os.FileInfo, err error)

Readdir reads the contents of the directory associated with file and returns a slice of up to n FileInfo values, as would be returned by os.Stat.

func (*Node) Save 

func (node *Node) Save(content []byte) (err error)

Save the content to file system and update the content of Node.

func (*Node) Seek 

func (node *Node) Seek(offset int64, whence int) (int64, error)

Seek sets the offset for the next Read offset, interpreted according to whence: SeekStart means relative to the start of the file, SeekCurrent means relative to the current offset, and SeekEnd means relative to the end. Seek returns the new offset relative to the start of the file and an error, if any.

func (*Node) SetModTime 

func (node *Node) SetModTime(modTime time.Time)

SetModTime set the file modification time.

func (*Node) SetModTimeUnix 

func (node *Node) SetModTimeUnix(seconds, nanoSeconds int64)

SetModTimeUnix set the file modification time using seconds and nanoseconds since January 1, 1970 UTC.

func (*Node) SetMode 

func (node *Node) SetMode(mode os.FileMode)

SetMode set the mode of file.

func (*Node) SetName 

func (node *Node) SetName(name string)

SetName set the name of file.

func (*Node) SetSize 

func (node *Node) SetSize(size int64)

SetSize set the file size.

func (*Node) Size 

func (node *Node) Size() int64

Size return the file size information.

func (*Node) Stat 

func (node *Node) Stat() (os.FileInfo, error)

Stat return the file information.

func (*Node) Sys 

func (node *Node) Sys() any

Sys return the underlying data source (can return nil).

func (*Node) Update 

func (node *Node) Update(newInfo os.FileInfo, maxFileSize int64) (err error)

Update the node metadata or content based on new file information.

The newInfo parameter is optional, if its nil, it will read the file information based on node's SysPath.

The maxFileSize parameter is also optional. If its negative, the node content will not be updated. If its zero, it will default to 5 MB.

There are two possible changes that will happen: its either change on mode or change on content (size and modtime). Change on mode will not affect the content of node.

func (*Node) UpdateContent added in v0.59.0 

func (node *Node) UpdateContent(maxFileSize int64) (err error)

UpdateContent read the content of file.

type Options 

type Options struct {
	// Embed options for GoEmbed method.
	Embed EmbedOptions

	// Root define the path to directory where its contents will be mapped
	// to memory or to be embedded as Go source code using GoEmbed.
	Root string

	// The includes and excludes pattern applied relative to the system
	// path.
	// The Excludes patterns will be applied first before the Includes.
	// If the path is not excluded and Includes is empty, it will be
	// assumed as included.
	Includes []string
	Excludes []string

	// MaxFileSize define maximum file size that can be stored on memory.
	// The default value is 5 MB.
	// If its value is negative, the content of file will not be mapped to
	// memory, the MemFS will behave as directory tree.
	MaxFileSize int64

	// TryDirect define a flag to bypass file in memory.
	// If its true, any call to Get will try direct read to file system.
	// This flag has several use cases.
	// First, to test serving file system directly from disk during
	// development.
	// Second, to combine embedded MemFS instance with non-embedded
	// instance.
	// One is reading content from memory, one is reading content from
	// disk directly.
	TryDirect bool
	// contains filtered or unexported fields
}

Options to create and initialize the MemFS.

type PathNode 

type PathNode struct {
	// contains filtered or unexported fields
}

PathNode contains a mapping between path and Node.

func NewPathNode 

func NewPathNode() *PathNode

NewPathNode create and initialize new PathNode.

func (*PathNode) Delete 

func (pn *PathNode) Delete(path string)

Delete the the node by its path.

func (*PathNode) Get 

func (pn *PathNode) Get(path string) (node *Node)

Get the node by path, or nil if path is not exist.

func (*PathNode) MarshalJSON 

func (pn *PathNode) MarshalJSON() ([]byte, error)

MarshalJSON encode the PathNode into JSON value.

func (*PathNode) Nodes 

func (pn *PathNode) Nodes() (nodes []*Node)

Nodes return all the nodes.

func (*PathNode) Paths 

func (pn *PathNode) Paths() (paths []string)

Paths return all the nodes paths sorted in ascending order.

func (*PathNode) Set 

func (pn *PathNode) Set(path string, node *Node)

Set mapping of path to Node.

type SearchResult 

type SearchResult struct {
	Path     string
	Snippets []string
}

SearchResult contains the result of searching where the Path will be filled with absolute path of file system in memory and the Snippet will filled with part of the text before and after the search string.

type WatchOptions 

type WatchOptions struct {
	watchfs.FileWatcherOptions

	// Verbose if true print the file changes information to stdout.
	Verbose bool
}

WatchOptions define an options for the MemFS Watch method.

If the watchfs.FileWatcherOptions.File is empty it will default to DefaultWatchFile inside the memfs.Options.Root. The watchfs.FileWatcherOptions.Interval must be greater than 10 milliseconds, otherwise it will default to 5 seconds.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.