Documentation ¶
Overview ¶
Package memfs provide a library for mapping file system into memory and/or to embed it inside go source file.
Usage ¶
The first step is to create new instance of memfs using `New()`. The following example embed all files inside directory named "include" or any files with extension ".css", ".html", and ".js"; but exclude any files inside directory named "exclude".
opts := &Options{ Root: "./mydir", Includes: []string{ `.*/include`, `.*\.(css|html|js)$`, }, Excludes: []string{ `.*/exclude`, }, } mfs, err := memfs.New(opts)
By default only file with size less or equal to 5 MB will be included in memory. To increase the default size set the MaxFileSize (in bytes). For example, to set maximum file size to 10 MB,
opts.MaxFileSize = 1024 * 1024 * 10
Later, if we want to get the file from memory, call Get() which will return the node object with content can be accessed from field "Content". Remember that if file size is larger that maximum, the content will need to be read manually,
node, err := mfs.Get("/") if err != nil { // Handle file not exist } if node.mode.IsDir() { // Handle directory. } if node.Content == nil { // Handle large file. node.V, err = os.ReadFile(child.SysPath) } // Do something with content of file system.
Go embed ¶
The memfs package also support embedding the files into Go generated source file. After we create memfs instance, we call GoEmbed() to dump all directory and files as Go source code.
First, define global variable as container for all files later in the same package as generated code,
package mypackage var myFS *memfs.MemFS
Second, create new instance of MemFS with Options.Embed is set, and write the content of memfs instance as Go source code file,
opts := &Options{ Root: "./mydir", Includes: []string{ `.*/include`, `.*\.(css|html|js)$`, }, Excludes: []string{ `.*/exclude`, }, Embed: EmbedOptions{ PackageName: "mypackage", VarName: "myFS", GoFileName: "mypackage/embed.go", }, } mfs, _ := memfs.New(opts) mfs.GoEmbed()
The Go generated file will be defined with package named "mypackage" using global variable "myFS" as container stored in file "mypackage/file.go" with each content encoded (compressed) using gzip.
Thats it!
Index ¶
- Constants
- type DirWatcher
- type EmbedOptions
- type FileState
- type MemFS
- func (mfs *MemFS) AddChild(parent *Node, fi os.FileInfo) (child *Node, err error)
- func (mfs *MemFS) AddFile(internalPath, externalPath string) (node *Node, err error)
- func (mfs *MemFS) Get(path string) (node *Node, err error)
- func (mfs *MemFS) GoEmbed() (err error)
- func (mfs *MemFS) Init() (err error)
- func (mfs *MemFS) ListNames() (paths []string)
- func (mfs *MemFS) MarshalJSON() ([]byte, error)
- func (mfs *MemFS) Merge(sub *MemFS)
- func (mfs *MemFS) MustGet(path string) (node *Node)
- func (mfs *MemFS) Open(path string) (http.File, error)
- func (mfs *MemFS) Remount() (err error)
- func (mfs *MemFS) RemoveChild(parent *Node, child *Node) (removed *Node)
- func (mfs *MemFS) Search(words []string, snippetLen int) (results []SearchResult)
- func (mfs *MemFS) StopWatch()
- func (mfs *MemFS) Update(node *Node, newInfo os.FileInfo)
- func (mfs *MemFS) Watch(opts WatchOptions) (dw *DirWatcher, err error)
- type Node
- func (node *Node) AddChild(child *Node)
- func (node *Node) Child(name string) (cnode *Node)
- func (node *Node) Close() error
- func (node *Node) GenerateIndexHTML()
- func (node *Node) IsDir() bool
- func (node *Node) JSON(depth int, withContent, withModTime bool) (rawjson []byte, err error)
- func (node *Node) MarshalJSON() ([]byte, error)
- func (node *Node) ModTime() time.Time
- func (node *Node) Mode() os.FileMode
- func (node *Node) Name() string
- func (node *Node) Read(p []byte) (n int, err error)
- func (node *Node) Readdir(count int) (fis []os.FileInfo, err error)
- func (node *Node) Save(content []byte) (err error)
- func (node *Node) Seek(offset int64, whence int) (int64, error)
- func (node *Node) SetModTime(modTime time.Time)
- func (node *Node) SetModTimeUnix(seconds, nanoSeconds int64)
- func (node *Node) SetMode(mode os.FileMode)
- func (node *Node) SetName(name string)
- func (node *Node) SetSize(size int64)
- func (node *Node) Size() int64
- func (node *Node) Stat() (os.FileInfo, error)
- func (node *Node) Sys() interface{}
- func (node *Node) Update(newInfo os.FileInfo, maxFileSize int64) (err error)
- type NodeState
- type Options
- type PathNode
- type SearchResult
- type WatchCallback
- type WatchOptions
- type Watcher
Examples ¶
Constants ¶
const DefaultEmbedGoFileName = `memfs_generate.go`
DefaultEmbedGoFileName default file output for GoEmbed.
const DefaultEmbedPackageName = `main`
DefaultEmbedPackageName default package name for GoEmbed.
const DefaultEmbedVarName = `memFS`
DefaultEmbedVarName default variable name for GoEmbed.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type DirWatcher ¶
type DirWatcher struct { // C channel on which the changes are delivered to user. C <-chan NodeState // This struct embed Options to map the directory to be watched // into memory. // // The Root field define the directory that we want to watch. // // Includes contains list of regex to filter file names that we want // to be notified. // // Excludes contains list of regex to filter file names that we did // not want to be notified. Options // Delay define a duration when the new changes will be fetched from // system. // This field is optional, minimum is 100 milli second and default // is 5 seconds. Delay time.Duration // contains filtered or unexported fields }
DirWatcher is a naive implementation of directory change notification.
Example ¶
package main import ( "fmt" "log" "os" "path/filepath" "time" "git.sr.ht/~shulhan/pakakeh.go/lib/memfs" ) func main() { var ( rootDir string err error ) rootDir, err = os.MkdirTemp(``, `libmemfs`) if err != nil { log.Fatal(err) } // In this example, we watch sub directory "assets" and its // contents, including only files with ".adoc" extension and // excluding files with ".html" extension. var dw = &memfs.DirWatcher{ Options: memfs.Options{ Root: rootDir, Includes: []string{ `assets/.*`, `.*\.adoc$`, }, Excludes: []string{ `.*\.html$`, }, }, Delay: 100 * time.Millisecond, } err = dw.Start() if err != nil { log.Fatal(err) } fmt.Println(`Deleting the root directory:`) err = os.Remove(rootDir) if err != nil { log.Fatal(err) } var ns = <-dw.C fmt.Println(`--`, ns.State, ns.Node.Path) // Create the root directory back with sub directory // This will trigger one FileStateCreated event, for "/". fmt.Println(`Re-create root directory with sub-directory:`) var dirAssets = filepath.Join(rootDir, `assets`) err = os.MkdirAll(dirAssets, 0770) if err != nil { log.Fatal(err) } ns = <-dw.C fmt.Println(`--`, ns.State, ns.Node.Path) // Modify the permission on root directory fmt.Println(`Chmod on root directory:`) err = os.Chmod(rootDir, 0700) if err != nil { log.Fatal(err) } ns = <-dw.C fmt.Println(`--`, ns.State, ns.Node.Path, ns.Node.Mode()) fmt.Println(`Create new file on root directory: /new.adoc`) var newFile = filepath.Join(rootDir, `new.adoc`) err = os.WriteFile(newFile, nil, 0600) if err != nil { log.Fatal(err) } ns = <-dw.C fmt.Println(`--`, ns.State, ns.Node.Path, ns.Node.Mode()) fmt.Println(`Remove file on root directory: /new.adoc`) err = os.Remove(newFile) if err != nil { log.Fatal(err) } ns = <-dw.C fmt.Println(`--`, ns.State, ns.Node.Path, ns.Node.Mode()) fmt.Println(`Create new sub-directory: /subdir`) var subDir = filepath.Join(rootDir, `subdir`) err = os.Mkdir(subDir, 0770) if err != nil { log.Fatal(err) } ns = <-dw.C fmt.Println(`--`, ns.State, ns.Node.Path, ns.Node.Mode()) // Add new file in sub directory. newFile = filepath.Join(subDir, `new.adoc`) fmt.Println(`Create new file in sub directory: /subdir/new.adoc`) err = os.WriteFile(newFile, nil, 0600) if err != nil { log.Fatal(err) } ns = <-dw.C fmt.Println(`--`, ns.State, ns.Node.Path, ns.Node.Mode()) fmt.Println(`Remove file in sub directory: /subdir/new.adoc`) err = os.Remove(newFile) if err != nil { log.Fatal(err) } ns = <-dw.C fmt.Println(`--`, ns.State, ns.Node.Path, ns.Node.Mode()) // Creating file that is excluded should not trigger event. fmt.Println(`Create excluded file in sub directory: /subdir/new.html`) newFile = filepath.Join(subDir, `new.html`) err = os.WriteFile(newFile, nil, 0600) if err != nil { log.Fatal(err) } // Create file without extension in directory "assets" should trigger // event. newFile = filepath.Join(dirAssets, `new`) fmt.Println(`Create new file under assets: /assets/new`) err = os.WriteFile(newFile, nil, 0600) if err != nil { log.Fatal(err) } ns = <-dw.C fmt.Println(`--`, ns.State, ns.Node.Path, ns.Node.Mode()) dw.Stop() }
Output: Deleting the root directory: -- FileStateDeleted / Re-create root directory with sub-directory: -- FileStateCreated / Chmod on root directory: -- FileStateUpdateMode / drwx------ Create new file on root directory: /new.adoc -- FileStateCreated /new.adoc -rw------- Remove file on root directory: /new.adoc -- FileStateDeleted /new.adoc -rw------- Create new sub-directory: /subdir -- FileStateCreated /subdir drwxr-x--- Create new file in sub directory: /subdir/new.adoc -- FileStateCreated /subdir/new.adoc -rw------- Remove file in sub directory: /subdir/new.adoc -- FileStateDeleted /subdir/new.adoc -rw------- Create excluded file in sub directory: /subdir/new.html Create new file under assets: /assets/new -- FileStateCreated /assets/new -rw-------
func (*DirWatcher) Start ¶
func (dw *DirWatcher) Start() (err error)
Start watching changes in directory and its content.
type EmbedOptions ¶
type EmbedOptions struct { // CommentHeader define optional comment to be added to the header of // generated file, for example copyright holder and/or license. // The string value is not checked, whether it's a comment or not, it // will rendered as is. // // Due to templating, the value MUST be set using raw // string literal syntax “, NOT "". CommentHeader string // The generated package name for GoEmbed(). // If its not defined it will be default to "main". PackageName string // VarName is the global variable name with type *memfs.MemFS which // will be initialized by generated Go source code on init(). // If its empty it will default to "memFS". VarName string // GoFileName the path to Go generated file, where the file // system will be embedded. // If its not defined it will be default to "memfs_generate.go" // in current directory from where its called. GoFileName string // WithoutModTime if its true, the modification time for all // files and directories are not stored inside generated code, instead // all files will use the current time when the program is running. WithoutModTime bool }
EmbedOptions define an options for GoEmbed.
type FileState ¶
type FileState byte
FileState define the state of file. There are four states of file: created, updated on mode, updated on content or deleted.
const ( FileStateCreated FileState = iota // FileStateCreated when new file is created. FileStateUpdateContent // FileStateUpdateContent when the content of file is modified. FileStateUpdateMode // FileStateUpdateMode when the mode of file is modified. FileStateDeleted // FileStateDeleted when the file has been deleted. )
type MemFS ¶
type MemFS struct { http.FileSystem PathNodes *PathNode Root *Node Opts *Options // contains filtered or unexported fields }
MemFS contains directory tree of file system in memory.
func New ¶
New create and initialize new memory file system from directory Root using list of regular expresssion for Including or Excluding files.
Example ¶
package main import ( "fmt" "log" "git.sr.ht/~shulhan/pakakeh.go/lib/memfs" ) func main() { /** Let say we have the "testdata" directory, testdata/ ├── direct │ └── add │ ├── file │ └── file2 ├── exclude │ ├── dir │ ├── index-link.css -> ../index.css │ ├── index-link.html -> ../index.html │ └── index-link.js -> ../index.js ├── include │ ├── dir │ ├── index.css -> ../index.css │ ├── index.html -> ../index.html │ └── index.js -> ../index.js ├── index.css ├── index.html ├── index.js └── plain Assume that we want to embed all files with extension .css, .html, and .js only; but not from directory "exclude". We can create the Options like below, */ opts := &memfs.Options{ Root: `./testdata`, Includes: []string{`.*/include`, `.*\.(css|html|js)$`}, Excludes: []string{`.*/exclude`}, } mfs, err := memfs.New(opts) if err != nil { log.Fatal(err) } node, err := mfs.Get("/index.html") if err != nil { log.Fatal(err) } fmt.Printf("Content of /index.html: %s", node.Content) fmt.Printf("List of embedded files: %+v\n", mfs.ListNames()) _, err = mfs.Get("/exclude/index.html") if err != nil { fmt.Println(`Error:`, err) } }
Output: Content of /index.html: <html></html> List of embedded files: [/ /direct /direct/add /include /include/dir /include/index.css /include/index.html /include/index.js /index.css /index.html /index.js] Error: Get "/exclude/index.html": file does not exist
func (*MemFS) AddChild ¶
AddChild add FileInfo fi as new child of parent node.
It will return nil without an error if,
- the system path of parent+fi.Name() is excluded by one of Options.Excludes pattern, or
- fi is symlink to not existen node.
func (*MemFS) AddFile ¶
AddFile add the external file directly as internal file. If the internal file is already exist it will be replaced. Any directories in the internal path will be generated automatically if its not exist.
func (*MemFS) Get ¶
Get the node representation of file in memory. If path is not exist it will return fs.ErrNotExist.
func (*MemFS) GoEmbed ¶
GoEmbed write the tree nodes as Go generated source file. This method assume that the files inside the mfs instance is already up-to-date. If you are not sure, call Remount.
func (*MemFS) Init ¶
Init initialize the MemFS instance. This method provided to initialize MemFS if its Options is set directly, not through New() function.
func (*MemFS) MarshalJSON ¶
MarshalJSON encode the MemFS object into JSON format.
The field that being encoded is the Root node.
func (*MemFS) Merge ¶
Merge other MemFS instance as sub file system.
When Get method called, each sub fs will be evaluated in order of Merge.
func (*MemFS) MustGet ¶
MustGet return the Node representation of file in memory by its path if its exist or nil the path is not exist.
func (*MemFS) Open ¶
Open the named file for reading. This is an alias to Get() method, to make it implement http.FileSystem.
func (*MemFS) Remount ¶
Remount reset the memfs instance to force rescanning the files again from file system.
func (*MemFS) RemoveChild ¶
RemoveChild remove a child on parent, including its map on PathNode. If child is not part if node's childrens it will return nil.
func (*MemFS) Search ¶
func (mfs *MemFS) Search(words []string, snippetLen int) (results []SearchResult)
Search one or more strings in each content of files.
Example ¶
package main import ( "fmt" "log" "git.sr.ht/~shulhan/pakakeh.go/lib/memfs" ) func main() { opts := &memfs.Options{ Root: `./testdata`, } mfs, err := memfs.New(opts) if err != nil { log.Fatal(err) } q := []string{`body`} results := mfs.Search(q, 0) for _, result := range results { fmt.Println(`Path:`, result.Path) fmt.Printf("Snippets: %q\n", result.Snippets) } }
Output: Path: /include/index.css Snippets: ["body {\n}\n"] Path: /exclude/index-link.css Snippets: ["body {\n}\n"] Path: /index.css Snippets: ["body {\n}\n"]
func (*MemFS) StopWatch ¶
func (mfs *MemFS) StopWatch()
StopWatch stop watching for update, from calling Watch.
func (*MemFS) Update ¶
Update the node content and information in memory based on new file information. This method only check if the node name is equal with file name, but it's not checking whether the node is part of memfs (node is parent or have the same Root node).
func (*MemFS) Watch ¶
func (mfs *MemFS) Watch(opts WatchOptions) (dw *DirWatcher, err error)
Watch create and start the DirWatcher that monitor the memfs Root directory based on the list of pattern on WatchOptions.Watches and Options.Includes.
The MemFS will remove or update the tree and node content automatically if the file being watched get deleted or updated.
The returned DirWatcher is ready to use. To stop watching for update call the StopWatch.
Example ¶
package main import ( "fmt" "log" "os" "path/filepath" "time" "git.sr.ht/~shulhan/pakakeh.go/lib/memfs" ) func main() { var ( watchOpts = memfs.WatchOptions{ Delay: 200 * time.Millisecond, } mfs *memfs.MemFS dw *memfs.DirWatcher node *memfs.Node opts memfs.Options ns memfs.NodeState err error ) opts.Root, err = os.MkdirTemp(``, `memfs_watch`) if err != nil { log.Println(err) return } defer func() { _ = os.RemoveAll(opts.Root) }() mfs, err = memfs.New(&opts) if err != nil { log.Println(err) return } dw, err = mfs.Watch(watchOpts) if err != nil { log.Println(err) return } // Wait for the goroutine on Watch run. time.Sleep(200 * time.Millisecond) testFile := filepath.Join(opts.Root, `file`) err = os.WriteFile(testFile, []byte(`dummy content`), 0600) if err != nil { log.Println(err) return } ns = <-dw.C node, err = mfs.Get(`/file`) if err != nil { log.Println(err) return } fmt.Printf("Node: %s: %s\n", node.Path, ns.State) err = os.Remove(testFile) if err != nil { log.Println(err) return } ns = <-dw.C fmt.Printf("Node: %s: %s\n", ns.Node.Path, ns.State) dw.Stop() }
Output: Node: /file: FileStateCreated Node: /file: FileStateDeleted
type Node ¶
type Node struct { Parent *Node // Pointer to parent directory. SysPath string // The original file path in system. Path string // Absolute file path in memory. ContentType string // File type per MIME, for example "application/json". GenFuncName string // The function name for embedded Go code. Childs []*Node // List of files in directory. Content []byte // Content of file. // contains filtered or unexported fields }
Node represent a single file.
This Node implement os.FileInfo and http.File.
func NewNode ¶
NewNode create a new node based on file information "fi".
The parent parameter is required to allow valid system path generated for new node.
If maxFileSize is greater than zero, the file content and its type will be saved in node as Content and ContentType.
func (*Node) GenerateIndexHTML ¶
func (node *Node) GenerateIndexHTML()
GenerateIndexHTML generate simple directory listing as HTML for all childs in this node. This method is only applicable if node is a directory.
func (*Node) JSON ¶
JSON encode the Node into JSON. This method provides an alternative to MarshalJSON with granular options.
The depth set the level of childs to be encoded. The depth=0 only encode the Root Node itself (with its childs), depth=1 encode the Root node and its subdirectories, and so on.
If the withContent set to true, all of the Node content will not be included in the output.
If the withModTime is set to true, all of the node ModTime will not be included in the output.
func (*Node) MarshalJSON ¶
MarshalJSON encode the node into JSON format. If the node is a file it will return the content of file; otherwise it will return the node with list of childs, but not including childs of childs.
func (*Node) Readdir ¶
Readdir reads the contents of the directory associated with file and returns a slice of up to n FileInfo values, as would be returned by os.Stat.
func (*Node) Seek ¶
Seek sets the offset for the next Read offset, interpreted according to whence: SeekStart means relative to the start of the file, SeekCurrent means relative to the current offset, and SeekEnd means relative to the end. Seek returns the new offset relative to the start of the file and an error, if any.
func (*Node) SetModTime ¶
SetModTime set the file modification time.
func (*Node) SetModTimeUnix ¶
SetModTimeUnix set the file modification time using seconds and nanoseconds since January 1, 1970 UTC.
func (*Node) Sys ¶
func (node *Node) Sys() interface{}
Sys return the underlying data source (can return nil).
func (*Node) Update ¶
Update the node metadata or content based on new file information.
The newInfo parameter is optional, if its nil, it will read the file information based on node's SysPath.
The maxFileSize parameter is also optional. If its negative, the node content will not be updated. If its zero, it will default to 5 MB.
There are two possible changes that will happen: its either change on mode or change on content (size and modtime). Change on mode will not affect the content of node.
type NodeState ¶
type NodeState struct { // Node represent the file information. Node Node // State of file, its either created, modified, or deleted. State FileState }
NodeState contains the information about the file and its state.
type Options ¶
type Options struct { // Embed options for GoEmbed method. Embed EmbedOptions // Root define the path to directory where its contents will be mapped // to memory or to be embedded as Go source code using GoEmbed. Root string // The includes and excludes pattern applied relative to the system // path. // The Excludes patterns will be applied first before the Includes. // If the path is not excluded and Includes is empty, it will be // assumed as included. Includes []string Excludes []string // MaxFileSize define maximum file size that can be stored on memory. // The default value is 5 MB. // If its value is negative, the content of file will not be mapped to // memory, the MemFS will behave as directory tree. MaxFileSize int64 // TryDirect define a flag to bypass file in memory. // If its true, any call to Get will try direct read to file system. // This flag has several use cases. // First, to test serving file system directly from disk during // development. // Second, to combine embedded MemFS instance with non-embedded // instance. // One is reading content from memory, one is reading content from // disk directly. TryDirect bool }
Options to create and initialize the MemFS.
type PathNode ¶
type PathNode struct {
// contains filtered or unexported fields
}
PathNode contains a mapping between path and Node.
func (*PathNode) MarshalJSON ¶
MarshalJSON encode the PathNode into JSON value.
type SearchResult ¶
SearchResult contains the result of searching where the Path will be filled with absolute path of file system in memory and the Snippet will filled with part of the text before and after the search string.
type WatchCallback ¶
type WatchCallback func(*NodeState)
WatchCallback is a function that will be called when Watcher or DirWatcher detect any changes on its file or directory. The watcher will pass the file information and its state.
type WatchOptions ¶
type WatchOptions struct { // Watches contain list of regular expressions for files to be watched // inside the Root, as addition to Includes pattern. // If this field is empty, only files pass the Includes filter will be // watched. Watches []string // Delay define the duration when the new changes will be checked from // system. // This field set the DirWatcher.Delay returned from Watch(). // This field is optional, default is 5 seconds. Delay time.Duration }
WatchOptions define an options for the MemFS Watch method.
type Watcher ¶
type Watcher struct { C <-chan NodeState // The channel on which the changes are delivered. // contains filtered or unexported fields }
Watcher is a naive implementation of file event change notification.
func NewWatcher ¶
NewWatcher return a new file watcher that will inspect the file for changes for `path` with period specified by duration `d` argument.
If duration is less or equal to 100 millisecond, it will be set to default duration (5 seconds).
The changes can be consumed from the channel C. If the consumer is slower, channel is full, the changes will be dropped.
Example ¶
package main import ( "fmt" "log" "os" "time" "git.sr.ht/~shulhan/pakakeh.go/lib/memfs" ) func main() { var ( content = `Content of file` f *os.File watcher *memfs.Watcher ns memfs.NodeState err error ) // Create a file to be watched. f, err = os.CreateTemp(``, `watcher`) if err != nil { log.Fatal(err) } watcher, err = memfs.NewWatcher(f.Name(), 150*time.Millisecond) if err != nil { log.Fatal(err) } // Update file mode. err = f.Chmod(0700) if err != nil { log.Fatal(err) } ns = <-watcher.C fmt.Printf("State: %s\n", ns.State) fmt.Printf("File mode: %s\n", ns.Node.Mode()) fmt.Printf("File size: %d\n", ns.Node.Size()) // Update content of file. _, err = f.WriteString(content) if err != nil { log.Fatal(err) } ns = <-watcher.C fmt.Printf("State: %s\n", ns.State) fmt.Printf("File mode: %s\n", ns.Node.Mode()) fmt.Printf("File size: %d\n", ns.Node.Size()) err = f.Close() if err != nil { log.Fatal(err) } // Remove the file. err = os.Remove(f.Name()) if err != nil { log.Fatal(err) } ns = <-watcher.C fmt.Printf("State: %s\n", ns.State) fmt.Printf("File mode: %s\n", ns.Node.Mode()) fmt.Printf("File size: %d\n", ns.Node.Size()) }
Output: State: FileStateUpdateMode File mode: -rwx------ File size: 0 State: FileStateUpdateContent File mode: -rwx------ File size: 15 State: FileStateDeleted File mode: -rwx------ File size: 15