zassets

README

Zassets, An Elegant Resource Bundler for Go

v0.0.1 Disclaimer: The resource debugger and compile APIs are unstable. The project carries some technical debt from some minifiers, which are not adapted to work well with virtual file systems, and a particular ESNext compiler ESBuild, which is incredibly fast but quite hacky itself. Pull requests for more general usage and optimization are welcome.

The program generates static asset packs as embedded Zip archives. It allows for multiple asset packs to exist harmoniously side by side in the same module, unlike most other packers. Giving up individual gzip encoding, common in other resource packers, for a Zip bundle speeds up compilation for larger resource packs. It also allows easy loading of assets from an external archive by calling zassets.FromArchive("zipFile.zip") when needed. The loss of speed for directory transversal operations is justified by the assumption that static assets are rarely accessed directly. They should be shielded by a proper caching layer or mirror to a CDN. This project is inspired by the following excellent packages:

• https://github.com/shurcooL/vfsgen
• https://github.com/tdewolff/minify
• https://github.com/evanw/esbuild
• https://github.com/wellington/go-libsass

Installation

There are build recipes for Linux, Windows, and MacOS in the Makefile. If you have the Go environment configured with $GOROOT/bin included in $PATH, enter the following command in your terminal:

go install github.com/dkotik/zassets/cmd/zassets

Usage

zassets [files or directories]... [flags]...

Flags:
  -c, --comment [lines]      Include a comment with the variable definition.
  -d, --debug                Readable refined output.
  -e, --embed                Embed provided files and directories.
  -i, --ignore [pattern]     Skip files and directories that match provided pattern.
                             You can use this flag multiple times. If not set, ignore paths that begin with "." or "_" or "node_modules".
  -o, --output [file]        Write program output to this location.
  -p, --package [name]       Assets will belong to this package.
  -r, --refine               Apply default refiners to assets before embedding.
  -s, --sum [algorithm]      Include a hash table sum.* in the embedded archive.
                             Choose from xxh64, md5, and sha256.
  -t, --tag [name]           Specify a build tag. You can use this flag multiple times.
  -v, --var [name]           Assets will be accessible using this variable name.
      --version              version for zassets

Point Zassets at resource files or directories by adding the following line under the package declaration in one of your Go files. When you run go generate, the program will create the assets.gen.go file, which will contain the specified Assets variable acting as http.FileSystem. To read an asset call Assets.Open("assetpath.txt").

//go:generate zassets dir1 dir2 file1 --output assets.gen.go --package mypackage --var Assets --embed

You may use shell redirection for the output in this manner: //go:generate sh -c "zassets dir1 dir2 file1 --var Assets --package mypackage --embed > assets.gen.go". But if an error occurs due to a missing file or during refinement, the shell will truncate assets.gen.go and cause package execution errors because of a missing variable. For this reason, it is wiser to specify --output parameter explicitly every time instead of using shell redirection.

Transformations

The following default file transformations happen when the --refine flag is activated. You may configure the compiler with custom refiners manually by using zassets/compile package.

• *.js files are compiled to ESNext *.js bundles.
• *.sass and *.scss files are compiled to *.css.
• *.html, *.svg, and *.css files are minified.
• *.jpg, *.jpeg, *.png, *.webp images are resized to 1080p and re-compressed for the Web.
• *.min.* files are served without any changes.

Public Assets

If you would like to include a hash map for a given asset pack, use the --sum [algorithm] parameter. The resulting archive will contain a hash map file sum.[algorithm], which you can use for validation or for serving assets by hash name. The second is handy for serving multiple versions of the same file in continual deployment setups or for a CDN or when multiple packages share some of the same assets. Files with identical content will be associated with the same hash name, when the sum.[algorithm] file is included. Experimental helpers you may try:

• zassets.Public: An http.FileSystem object that will open assets by an associated hash name. Serve files over HTTP from this object.
• zassets.PublicRegister: Link all assets from a given pack with zassets.Public. If the pack contains a supported sum.[algorithm] file, the assets are associated using the hash map and their original file extension. Otherwise, the debug mode is assumed and a flimsy hash is generated from name space and path.
• zassets.PublicName: Returns the hash name associated with a resource. Use this function with your template engine to point to correct assets.
• zassets.Handler: An http.Handler object that serves files from zassets.Public.

Roadmap

• Explain major benefit of multiple resource packs: a huge pile of icons takes a long time to zip up and they do not need to be re-compressed.
• Solidify Refiner API:
  • Javascript io.Pipe is clumsy. Needs re-writing.
  • Add image quality setting for the Compiler.
  • Add a refiner for Typescript.
  • Add a refiner for Clojure.
  • What is zassets points to the same folder "." - the pathing gets screwed up with a panic? recursive inclusion - should I fight it? Filter out .go files by default?!
• Solidify Debugger API:
  • Watched directories do not respond to new files being created.
• When the APIs are solid, hedge them with test suites.

License

Zassets is released under the MIT license. The author would also like to add the SQLite blessing:

May you do good and not evil. May you find forgiveness for yourself and forgive others. May you share freely, never taking more than you give.

Documentation

Index

• Variables
• func MustOpen(f http.File, err error) http.File
• func PublicName(namespace, path string) (string, error)
• func PublicRegister(namespace string, d http.FileSystem) (err error)
• func Walk(s http.FileSystem, p string, f filepath.WalkFunc) error
• type Embed
  • func (e *Embed) Iterator(w io.Writer, i *compile.Iterator) (err error)
  • func (e *Embed) Reader(w io.Writer, r io.Reader) error
  • func (e *Embed) SetTemplate(t string)
• type Store
  • func FromArchive(p string) (*Store, error)
  • func FromBytes(b []byte) (*Store, error)
  • func Must(s *Store, err error) *Store
  • func (s *Store) Open(p string) (http.File, error)

Variables

var Handler http.Handler = new(handler)

Handler serves files only from Public. URL.Path does not have to be stripped.

var Public http.FileSystem = &publicSystem{}

Public provides access to assets distributed among different FileSystems by hash name. This is useful for serving static assets from a single http.Handler sitting behing a caching layer or from a CDN. To find out the hash name for a given asset, call the PublicName function.

var PublicPrefix = `/asset/`

PublicPrefix is added to all the outputs of PublicName.

Functions

func MustOpen

func MustOpen(f http.File, err error) http.File

MustOpen panics if there is an error opening an asset. Use it as a convenience method when loading required assets. Example: asset := zassets.MustOpen(Assets.Open(`schema.sql`))

func PublicName

func PublicName(namespace, path string) (string, error)

PublicName returns the hash name associated with an asset namespace and path. This function is useful for pointing asset URLs in any templating engine contained in the Public virtual file system.

func PublicRegister

func PublicRegister(namespace string, d http.FileSystem) (err error)

PublicRegister connects a FileSystem to Public, differentiated by name space. If the FileSystem includes "sum.xxhash", "sum.md5", or "sum.sha256" file, the hash map is constructed by using values from this file. Otherwise, the hash map is constructed by hashing name space and asset path, which is useful for debugging Public.

func Walk

func Walk(s http.FileSystem, p string, f filepath.WalkFunc) error

Walk emulates filepath.Walk()

Types

type Embed

type Embed struct {
	Variable      string
	Package       string
	Comment       string
	Refine        bool
	Debug         bool
	Entries       []string
	Ignore        []string
	Tags          []string
	HashAlgorythm string
	HashTable     map[string][]byte
	// contains filtered or unexported fields
}

Embed converts a binary stream or a path set to a Go asset Store.

func (*Embed) Iterator

func (e *Embed) Iterator(w io.Writer, i *compile.Iterator) (err error)

Iterator zips and embeds the contents of all paths.

func (*Embed) Reader

func (e *Embed) Reader(w io.Writer, r io.Reader) error

Reader writes binary data through a template.

func (*Embed) SetTemplate

func (e *Embed) SetTemplate(t string)

SetTemplate sets up definition blocks for output rendering. If an empty string is given, the default template is used instead.

type Store

type Store struct {
	Size int64
	// contains filtered or unexported fields
}

Store presents as zipped archive as http.FileSystem.

func FromArchive

func FromArchive(p string) (*Store, error)

FromArchive serves assets from a zip archive.

func FromBytes

func FromBytes(b []byte) (*Store, error)

FromBytes serves assets from bytes encoding a zip archive. Used for accessing assets embedded into a Go binary.

func Must

func Must(s *Store, err error) *Store

Must panics if there is an error associated with loading assets.

func (*Store) Open

func (s *Store) Open(p string) (http.File, error)

Open returns a handle to the underlying file or directory. The directory operations are slow! Zip is not the right file format for frequent tree transversal. Put Store behind a a proper caching layer, if speed is a requirement.