stuffbin

README

stuffbin

stuffbin is a utility + package to compress and embed static files and assets into Go binaries for distribution. It supports falling back to the local file system when no embedded assets are available, for instance, in development mode. stuffbin is inspired by zgok but is simpler and has better abstractions.

stuffbin vs. Go 1.16 embed

Go 1.16 introduced the //go:embed directive that allows embedding of files into Go binaries without any external utilities. stuffbin offers a few key advantages over native embedding in its current form.

• All files are ZIP compressed.
• Custom path aliases (eg: embed /home/local/path/file.txt as /app/file.txt).
• Dynamically embed files instead of static //go:embed directives.
• Embed parent directories, sub-directories, and arbitrary paths (eg: ../../path). Go embed does not permit embedding of files outside of a .go file's directory. This makes it difficult for programs structured in a cmd directory to embed files outside it.
• Better filesystem abstraction for virtual filesystem manipulation, including merging.

How does it work?

stuffbin compresses and embeds arbitrary files to the end of Go binaries. This does not affect the normal execution of the binary as the compressed data that is appended beyond the binary's original size is simply ignored by the operating system. When a stuffed application is executed, stuffbin reads the compressed bytes from self (the executable), uncompresses the files on the fly into an in-memory filesystem, and provides a FileSystem interface to access them. This enables Go applications that have external file dependencies to be shipped a single fat binary, commonly, web applications that have static file and template dependencies.

• Built in ZIP compression
• A virtual filesystem abstraction to access embedded files
• Add static assets from nested directories recursively
• Re-path files and whole directories with the :suffix format, eg: ../my/original/file.txt:/my/virtual/file.txt and /my/nested/dir:/virtual/dir
• Template parsing helper similar to template.ParseGlob() to parse templates from the virtual filesystem
• Launch an http.FileServer for serving static files
• Gracefully failover to the local file system in the absence of embedded assets
• CLI to stuff, unstuff and extract, and list stuffed files in binaries

Installation

go install github.com/knadh/stuffbin/...@latest

Homebrew

For macOS/Linux users, you can install via brew

$ brew install stuffbin

Usage

Stuffing and embedding

# -a, -in, and -out params followed by the paths of files to embed.
# To normalize paths, aliases can be suffixed with a colon.
stuffbin -a stuff -in /path/to/exe -out /path/to/new.exe \
    static/file1.css static/file2.pdf /somewhere/else/file3.txt:/static/file3.txt

List files in a stuffed binary

stuffbin -a id -in /path/to/new/exe

Extract stuffed files from a binary

stuffbin -a unstuff -in /path/to/new/exe -out assets.zip

In the application

To test this, cd into ./mock and run go run mock.go

package main

import (
	"fmt"
	"log"
	"net/http"
	"os"

	"github.com/knadh/stuffbin"
)

func main() {
	// Get self executable path.
	path, err := os.Executable()
	if err != nil {
		log.Fatalf("error getting executable path: %v", err)
	}
	// Read stuffed data from self.
	fs, err := stuffbin.UnStuff(path)
	if err != nil {
		// Binary is unstuffed or is running in dev mode.
		// Can halt here or fall back to the local filesystem.
		if err == stuffbin.ErrNoID {
			// First argument is to the root to mount the files in the FileSystem
			// and the rest of the arguments are paths to embed.
			fs, err = stuffbin.NewLocalFS("/",
				"./", "bar.txt:/virtual/path/bar.txt")
			if err != nil {
				log.Fatalf("error falling back to local filesystem: %v", err)
			}
		} else {
			log.Fatalf("error reading stuffed binary: %v", err)
		}
	}

	fmt.Println("loaded files", fs.List())
	// Read the file 'foo'.
	f, err := fs.Get("foo.txt")
	if err != nil {
		log.Fatalf("error reading foo.txt: %v", err)
	}
	log.Println("foo.txt =", string(f.ReadBytes()))

	// Read the file 'bar'.
	f, err = fs.Get("/virtual/path/bar.txt")
	if err != nil {
		log.Fatalf("error reading /virtual/path/bar.txt: %v", err)
	}
	log.Println("/virtual/path/bar.txt =", string(f.ReadBytes()))

	fmt.Println("stuffed files:")
	for _, f := range fs.List() {
		fmt.Println("\t", f)
	}

	// Compile templates with the helpers:
	// err, tpl := stuffbin.ParseTemplatesGlob(nil, fs, "/templates/*.html")
	//
	// Template func map.
	// mp := map[string]interface{}{
	// 	"Foo": func() string {
	// 		return "func"
	// 	},
	// }
	// err, tpl := stuffbin.ParseTemplates(mp, fs, "/templates/index.html", "/templates/hello.html")

	// Expose an HTTP file server.
	// Try http://localhost:8000/static/foo.txt
	// Try http://localhost:8000/static/virtual/path/bar.txt
	// Try http://localhost:8000/static/subdir/baz.txt
	http.Handle("/static/", http.StripPrefix("/static/", fs.FileServer()))
	log.Println("listening on :8000")
	http.ListenAndServe(":8000", nil)
}

License

Licensed under the MIT License.

Documentation

Index

• Variables
• func GetStuff(in string) ([]byte, error)
• func MergeFS(dest FileSystem, src FileSystem) error
• func ParseTemplates(f template.FuncMap, fs FileSystem, path ...string) (*template.Template, error)
• func ParseTemplatesGlob(f template.FuncMap, fs FileSystem, pattern string) (*template.Template, error)
• func Stuff(in, out, rootPath string, files ...string) (int64, int64, error)
• type File
  • func NewFile(path string, info os.FileInfo, b []byte) *File
  • func (f *File) Close() error
  • func (f *File) Path() string
  • func (f *File) Read(b []byte) (int, error)
  • func (f *File) ReadBytes() []byte
  • func (f *File) Readdir(count int) ([]os.FileInfo, error)
  • func (f *File) Seek(offset int64, whence int) (int64, error)
  • func (f *File) Stat() (os.FileInfo, error)
• type FileSystem
  • func NewFS() (FileSystem, error)
  • func NewLocalFS(rootPath string, paths ...string) (FileSystem, error)
  • func UnStuff(path string) (FileSystem, error)
  • func UnZip(b []byte) (FileSystem, error)
• type ID
  • func GetFileID(fName string) (ID, error)
• type WalkFunc

Variables

var ErrNoID = errors.New("no ID found in the file")

ErrNoID is used to indicate if an ID was found in a file or not.

var ErrNotSupported = errors.New("this method is not supported")

ErrNotSupported indicates interface methods that are implemented but not supported.

Functions

func GetStuff

func GetStuff(in string) ([]byte, error)

GetStuff takes the path to a stuffed binary and extracts the packed data.

func MergeFS

func MergeFS(dest FileSystem, src FileSystem) error

MergeFS merges FileSystem b into a, overwriting conflicting paths.

func ParseTemplates

func ParseTemplates(f template.FuncMap, fs FileSystem, path ...string) (*template.Template, error)

ParseTemplates takes a file system, a list of file paths, and parses them into a template.Template.

func ParseTemplatesGlob

func ParseTemplatesGlob(f template.FuncMap, fs FileSystem, pattern string) (*template.Template, error)

ParseTemplatesGlob takes a file system, a file path pattern, and parses matching files into a template.Template with an optional template.FuncMap that will be applied to the compiled templates.

func Stuff

func Stuff(in, out, rootPath string, files ...string) (int64, int64, error)

Stuff takes the path to a binary, a list of file paths to stuff, and compresses the files and appends them to the end of the binary's body and writes everything to a new binary.

Types

type File

type File struct {
	// contains filtered or unexported fields
}

File represents an abstraction over http.File.

func NewFile

func NewFile(path string, info os.FileInfo, b []byte) *File

NewFile creates and returns a new instance of File.

func (*File) Close

func (f *File) Close() error

Close emulates http.File's Close but internally, it simply seeks the File's reader to 0.

func (*File) Path

func (f *File) Path() string

Path returns the path of the file.

func (*File) Read

func (f *File) Read(b []byte) (int, error)

Read reads the file contents.

func (*File) ReadBytes

func (f *File) ReadBytes() []byte

ReadBytes returns the bytes of the given file.

func (*File) Readdir

func (f *File) Readdir(count int) ([]os.FileInfo, error)

Readdir is a dud.

func (*File) Seek

func (f *File) Seek(offset int64, whence int) (int64, error)

Seek seeks the given offset in the file.

func (*File) Stat

func (f *File) Stat() (os.FileInfo, error)

Stat returns the file's os.FileInfo.

type FileSystem

type FileSystem interface {
	Add(f *File) error
	List() []string
	Len() int
	Size() int64
	Get(path string) (*File, error)
	Glob(pattern string) ([]string, error)
	Read(path string) ([]byte, error)
	Open(path string) (http.File, error)
	Delete(path string) error
	Merge(f FileSystem) error
	FileServer() http.Handler
}

FileSystem represents a simple filesystem abstraction that implements the http.fileSystem interface.

func NewFS

func NewFS() (FileSystem, error)

NewFS returns a new instance of FileSystem.

func NewLocalFS

func NewLocalFS(rootPath string, paths ...string) (FileSystem, error)

NewLocalFS returns a new instance of FileSystem with the given list of local files and directories mapped to it.

func UnStuff

func UnStuff(path string) (FileSystem, error)

UnStuff takes the path to a stuffed binary, unstuffs it, and returns a FileSystem.

func UnZip

func UnZip(b []byte) (FileSystem, error)

UnZip unzips zipped bytes and returns a FileSystem with the files mapped to it.

type ID

type ID struct {
	Name    [8]byte
	BinSize uint64
	ZipSize uint64
}

ID represents an identifier that is appended to binaries for identifying stuffbin binaries. The fields are appended as bytes totalling 8 + 12 + 8 + 8 = 36 bytes in the order Name BinSize ZipSize.

func GetFileID

func GetFileID(fName string) (ID, error)

GetFileID attempts to get the stuffbin identifier from the end of the file and returns the identifier name and file sizes.

type WalkFunc

type WalkFunc func(srcPath, targetPath string, fInfo os.FileInfo) error

WalkFunc is an abstraction over filepath.WalkFunc that's used as a callback to receive the real file path and their corresponding target (alias) paths from a real filepath.Walk() traversal of a list of file and directory paths.