memfs

Documentation

Overview

Package memfs provide a library for mapping file system into memory and to generate a go file.

Usage

By default only file with size less or equal to 5 MB will be included in memory. To increase the default size set the MaxFileSize (in bytes). For example, to set maximum file size to 10 MB,

memfs.MaxFileSize = 1024 * 1024 * 10

The first step is to create new instance of memfs using "New()".

incs := []string{
	`.*/include`,
	`.*\.(css|html|js)$`,
}
excs := []string{
	`.*/exclude`,
}
mfs, err := memfs.New(incs, excs)

and then we mount the directory that we want into memory using "Mount()",

err := mfs.Mount("./testdata")

Later, if we want to get the file from memory, call "Get()" and access the content with "node.V". Remember that if file size is larger that maximum, the content will need to be read manually,

node, err := mfs.Get()
if err != nil {
	// Handle file not exist
}
if node.mode.IsDir() {
	// Handle directory.
}
if node.V == nil {
	// Handle large file.
	node.V, err = ioutil.ReadFile(child.SysPath)
}
// Do something with content of file system.

Thats it!

Go Generate

memfs also support generating the files into Go generated source file. After we Mount the directory, we can call,

mfs.GoGenerate("mypackage", "output/path/file.go", memfs.EncodingGzip)

The Go generated file will be defined with package named "mypackage" in file "output/path/file.go" with each content encoded (compressed) using gzip.

Index

• Constants
• Variables
• type MemFS
  • func New(includes, excludes []string, withContent bool) (mfs *MemFS, err error)
  • func (mfs *MemFS) AddChild(parent *Node, fi os.FileInfo) (child *Node, err error)
  • func (mfs *MemFS) AddFile(path string) (*Node, error)
  • func (mfs *MemFS) ContentEncode(encoding string) (err error)
  • func (mfs *MemFS) Get(path string) (node *Node, err error)
  • func (mfs *MemFS) GoGenerate(pkgName, out, contentEncoding string) (err error)
  • func (mfs *MemFS) IsMounted() bool
  • func (mfs *MemFS) ListNames() (paths []string)
  • func (mfs *MemFS) Mount(dir string) error
  • func (mfs *MemFS) Open(path string) (http.File, error)
  • func (mfs *MemFS) RemoveChild(parent *Node, child *Node) (removed *Node)
  • func (mfs *MemFS) Search(words []string, snippetLen int) (results []SearchResult)
  • func (mfs *MemFS) Unmount()
  • func (mfs *MemFS) Update(node *Node, newInfo os.FileInfo)
• type Node
  • func NewNode(parent *Node, fi os.FileInfo, withContent bool) (node *Node, err error)
  • func (leaf *Node) Close() error
  • func (leaf *Node) Decode() ([]byte, error)
  • func (leaf *Node) IsDir() bool
  • func (leaf *Node) ModTime() time.Time
  • func (leaf *Node) Mode() os.FileMode
  • func (leaf *Node) Name() string
  • func (leaf *Node) Read(p []byte) (n int, err error)
  • func (leaf *Node) Readdir(count int) (fis []os.FileInfo, err error)
  • func (leaf *Node) Seek(offset int64, whence int) (int64, error)
  • func (leaf *Node) SetModTime(modTime time.Time)
  • func (leaf *Node) SetMode(mode os.FileMode)
  • func (leaf *Node) SetName(name string)
  • func (leaf *Node) SetSize(size int64)
  • func (leaf *Node) Size() int64
  • func (leaf *Node) Stat() (os.FileInfo, error)
  • func (leaf *Node) Sys() interface{}
• type PathNode
  • func NewPathNode() *PathNode
  • func (pn *PathNode) Get(path string) *Node
  • func (pn *PathNode) Set(path string, node *Node)
• type SearchResult

Examples

• MemFS.Search
• New

Constants

const (
	EncodingGzip = "gzip"
)

List of valid content encoding for ContentEncode().

Variables

var (
	// MaxFileSize define maximum file size that can be stored on memory.
	// The default value is 5 MB.
	MaxFileSize int64 = 1024 * 1024 * 5

	// Development define a flag to bypass file in memory.  If its
	// true, any call to Get will result in direct read to file system.
	Development bool

	// GeneratedPathNode contains the mapping of path and node.  Its will
	// be used and initialized by ".go" file generated from GoGenerate().
	GeneratedPathNode *PathNode
)

Types

type MemFS

type MemFS struct {
	http.FileSystem
	// contains filtered or unexported fields
}

MemFS contains directory tree of file system in memory.

func New

func New(includes, excludes []string, withContent bool) (mfs *MemFS, err error)

New create and initialize new memory file system using list of regular expresssion for including or excluding files.

The includes and excludes pattern applied to path of file in file system, not to the path in memory.

The "withContent" parameter tell the MemFS to read the content of file and detect its content type. If this paramater is false, the content of file will not be mapped to memory, the MemFS will behave as directory tree.

On directory that contains output from GoGenerate(), the includes and excludes does not have any effect, since the content of path and nodes will be overwritten by GeneratedPathNode.

Example

incs := []string{
	`.*/include`,
	`.*\.(css|html|js)$`,
}
excs := []string{
	`.*/exclude`,
}

mfs, err := New(incs, excs, true)
if err != nil {
	log.Fatal(err)
}

err = mfs.Mount("./testdata")
if err != nil {
	log.Fatal(err)
}

node, err := mfs.Get("/index.html")
if err != nil {
	log.Fatal(err)
}

fmt.Printf("%s", node.V)

Output:

<html></html>

func (*MemFS) AddChild

func (mfs *MemFS) AddChild(parent *Node, fi os.FileInfo) (child *Node, err error)

AddChild add new child to parent node.

func (*MemFS) AddFile

func (mfs *MemFS) AddFile(path string) (*Node, error)

AddFile add the file directly as child of root. The directory and subdirectories in the path will be keep as separated nodes,

func (*MemFS) ContentEncode

func (mfs *MemFS) ContentEncode(encoding string) (err error)

ContentEncode encode each node's content into specific encoding, in other words this method can be used to compress the content of file in memory or before being served or written.

Only file with size greater than 0 will be encoded.

List of known encoding is "gzip".

func (*MemFS) Get

func (mfs *MemFS) Get(path string) (node *Node, err error)

Get the node representation of file in memory. If path is not exist it will return os.ErrNotExist.

func (*MemFS) GoGenerate

func (mfs *MemFS) GoGenerate(pkgName, out, contentEncoding string) (err error)

GoGenerate write the tree nodes as Go generated source file. If pkgName is not defined it will be default to "main". If out is not defined it will be default "memfs_generate.go" and saved in current directory.

If contentEncoding is not empty, it will encode the content of node and set the node ContentEncoding. List of available encoding is "gzip". For example, if contentEncoding is "gzip" it will compress the content of file using gzip and set "ContentEncoding" to "gzip".

func (*MemFS) IsMounted

func (mfs *MemFS) IsMounted() bool

IsMounted will return true if a directory in file system has been mounted to memory; otherwise it will return false.

func (*MemFS) ListNames

func (mfs *MemFS) ListNames() (paths []string)

ListNames list all files in memory sorted by name.

func (*MemFS) Mount

func (mfs *MemFS) Mount(dir string) error

Mount the directory recursively into the memory as root directory. For example, if we mount directory "/tmp" and "/tmp" contains file "a", to access file "a" we call Get("/a"), not Get("/tmp/a").

Mount does not have any effect if current directory contains ".go" file generated from GoGenerate().

func (*MemFS) Open

func (mfs *MemFS) Open(path string) (http.File, error)

Open the named file for reading. This is an alias to Get() method, to make it implement http.FileSystem.

func (*MemFS) RemoveChild

func (mfs *MemFS) RemoveChild(parent *Node, child *Node) (removed *Node)

RemoveChild remove a child on parent, including its map on PathNode. If child is not part if node's childrens it will return nil.

func (*MemFS) Search

func (mfs *MemFS) Search(words []string, snippetLen int) (results []SearchResult)

Search one or more strings in each content of files.

Example

mfs, err := New(nil, nil, true)
if err != nil {
	log.Fatal(err)
}

err = mfs.Mount("./testdata")
if err != nil {
	log.Fatal(err)
}

q := []string{"body"}
results := mfs.Search(q, 0)

for _, result := range results {
	fmt.Printf("Path: %s\n", result.Path)
	fmt.Printf("Snippets: %q\n", result.Snippets)
}

Output:

Path: /include/index.css
Snippets: ["body {\n}\n"]
Path: /exclude/index.css
Snippets: ["body {\n}\n"]
Path: /index.css
Snippets: ["body {\n}\n"]

func (*MemFS) Unmount

func (mfs *MemFS) Unmount()

Unmount the root directory from memory.

func (*MemFS) Update

func (mfs *MemFS) Update(node *Node, newInfo os.FileInfo)

Update the node content and information in memory based on new file information. This method only check if the node name is equal with file name, but it's not checking whether the node is part of memfs (node is parent or have the same root node).

type Node

type Node struct {
	os.FileInfo
	http.File

	SysPath string // The original file path in system.
	Path    string // Absolute file path in memory.

	ContentType     string // File type per MIME, for example "application/json".
	ContentEncoding string // File type encoding, for example "gzip".

	V      []byte  // Content of file.
	Parent *Node   // Pointer to parent directory.
	Childs []*Node // List of files in directory.
	// contains filtered or unexported fields
}

Node represent a single file.

func NewNode

func NewNode(parent *Node, fi os.FileInfo, withContent bool) (node *Node, err error)

NewNode create a new node based on file information "fi". If withContent is true, the file content and its type will be saved in node as V and ContentType.

func (*Node) Close

func (leaf *Node) Close() error

Close reset the offset position back to zero.

func (*Node) Decode

func (leaf *Node) Decode() ([]byte, error)

Decode the contents of node (for example, uncompress with gzip) and return it.

func (*Node) IsDir

func (leaf *Node) IsDir() bool

func (*Node) ModTime

func (leaf *Node) ModTime() time.Time

func (*Node) Mode

func (leaf *Node) Mode() os.FileMode

func (*Node) Name

func (leaf *Node) Name() string

func (*Node) Read

func (leaf *Node) Read(p []byte) (n int, err error)

Read the content of node into p.

func (*Node) Readdir

func (leaf *Node) Readdir(count int) (fis []os.FileInfo, err error)

Readdir reads the contents of the directory associated with file and returns a slice of up to n FileInfo values, as would be returned by Lstat, in directory order. Subsequent calls on the same file will yield further FileInfos.

func (*Node) Seek

func (leaf *Node) Seek(offset int64, whence int) (int64, error)

Seek sets the offset for the next Read offset, interpreted according to whence: SeekStart means relative to the start of the file, SeekCurrent means relative to the current offset, and SeekEnd means relative to the end. Seek returns the new offset relative to the start of the file and an error, if any.

func (*Node) SetModTime

func (leaf *Node) SetModTime(modTime time.Time)

SetModTime set the file modification time.

func (*Node) SetMode

func (leaf *Node) SetMode(mode os.FileMode)

SetMode set the mode of file.

func (*Node) SetName

func (leaf *Node) SetName(name string)

SetName set the name of file.

func (*Node) SetSize

func (leaf *Node) SetSize(size int64)

SetSize set the file size.

func (*Node) Size

func (leaf *Node) Size() int64

Size return the file size information.

func (*Node) Stat

func (leaf *Node) Stat() (os.FileInfo, error)

Stat return the file information.

func (*Node) Sys

func (leaf *Node) Sys() interface{}

Sys return the underlying data source (can return nil).

type PathNode

type PathNode struct {
	// contains filtered or unexported fields
}

PathNode contains a mapping between path and Node.

func NewPathNode

func NewPathNode() *PathNode

NewPathNode create and initialize new PathNode.

func (*PathNode) Get

func (pn *PathNode) Get(path string) *Node

Get the node by path, or nil if path is not exist.

func (*PathNode) Set

func (pn *PathNode) Set(path string, node *Node)

Set mapping of path to Node.

type SearchResult

type SearchResult struct {
	Path     string
	Snippets []string
}

SearchResult contains the result of searching where the Path will be filled with absolute path of file system in memory and the Snippet will filled with part of the text before and after the search string.