caviar

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 13, 2014 License: MIT Imports: 15 Imported by: 0

README

Caviar

Caviar is a resource packer for Go. It will essentially pack a bunch of resources/assets that you would normally deploy along with your executable into a custom ZIP file which will then bundled with your program's executable (or not, your choice).

NOTE: UNDER DEVELOPMENT. NOT READY FOR PRIMETIME.

Install

Just run:

$ go get github.com/mvillalba/caviar $ go get github.com/mvillalba/caviar/cmd/cavundle $ go get github.com/mvillalba/caviar/cmd/caviarize

Usage

Converting your program to use Caviar is very straightforward, just import "github.com/mvillalba/caviar", change any calls to os.Open/OpenFile to caviar.Open/OpenFile, and run the bundled cavundle utility on your compiled executables. That's it.

During runtime, Caviar will attempt to load bundled resources from the running executable and failing that from a detached container (executable-name.cvr).

Your program will still run in the event a Caviar bundle can't be loaded. Caviar will simply pass through your Open/OpenFile calls to the os package transparently. This is very useful for development. And the same goes for opening files not present in the bundle.

Got dependencies that need to read various files and won't take an io.Reader (I'm looking at you, Revel framework)? You can use the bundled tool caviarize which will attempt to automatically patch any package (and all its dependencies) to load their files via Caviar's API like this:

$ caviarize github.com/revel/revel

See the examples directory for a handful of working toy program examples.

NOTE: In order to generate attached bundles (program = program + asset bundle), cavundle needs the to execute zip program due to a shortcoming with Go's ZIP library

NOTE: Caviar is designed with long-running processes (such as Web apps) that need to have quick access to their assets/resources in mind and this has some consequences. Namely, Caviar will load all assets to RAM on startup and it will keep them there.

NOTE: This is an early version of Caviar and no cross-platform testing has been done. It works on Linux (and probably other UNIX variants), but using it on Windows will likely require some work (case-insensitive matches, possible hard-coded paths, etc.). You are welcome to submit a patch.

GoDoc

Contact

Martín Raúl Villalba martin@martinvillalba.com http://www.martinvillalba.com/

TODO

  • Support for unpacking to a TMP dir instead of RAM (for things like Gtk where we depend on C libraries that can't be patched with caviarize). Don't forget to implement a clean-up function to be run when the program exits!
  • Support for a custom, dead-simple binary format to replace ZIP files (Go's Zips don't support writing uncompressed data and TARs don't feel like a good replacement. The replacement format would be as follows: [EXECUTABLE][MAGIC-1][MANIFEST][ASSETS][MANIFEST-LEN][ASSETS-LEN][MAGIC-2] Reading it is a simple matter of:
    1. Read and verify MAGIC-2.
    2. Add ASSETS-LEN and MANIFEST-LEN and use them to calculate start offset of Caviar container (MAGIC-1 start).
    3. Read and verify MAGIC-1.
    4. Read both MANIFEST and ASSETS.
    5. Store ASSETS.
    6. Process MANIFEST. I may optionally consider adding a FLAGS byte somewhere to specify the compression algorithm used for MANIFEST and ASSETS. Options will probably be DEFLATE, GZIP, and NONE. No format versioning or backwards compatibility is needed as it's assumed the running version of Caviar will both be producing the asset bundle and reading it during runtime. The supported optimization profiles could be: FAST MANIFEST (NONE), ASSETS (NONE) TINY MANIFEST (GZIP), ASSETS (GZIP) NORM MANIFEST (NONE), ASSETS (DEFLATE)
  • Make the manifest use Protocol Buffers instead of Gob.
  • Cross-platform support.
  • Some functions in the "os" file-related API return an os.PathError instead of an error type. Caviar should mimick this behavior.
  • Practical examples.
  • Tests, tests, tests.
  • Preserve metadata other than the file name. I'm thinking creation time, modification time, and permission bits.
  • Handle absolute paths better in caviarOpen(). At the moment, any absolute path must match the path string returned by osext.ExecutableFolder() exactly to be considered to be inside the asset root path, so something like '/home/martin/wks/../go/myprogram/data.bin' will be passed to os.Open (and error out with a file not found error) while '/home/martin/wks/go/myprogram/data.bin' will succeed.
  • The code needs a general clean-up, but fo it after mostly everything else on this list has been implemented.
  • Can't be bothered now, but there is a hack (find “BEGIN HACK”) in init.go to patch a bug somewhere (probably in processManifest() or cavundle) that creates a sort of pseuso-root dir in state.assets that's kinda broken.
  • Invoking cavundle with multiple paths that contain files and/or directories with repeated names (i.e revel/README, martini/README, etc.) and cherrypicking disabled is likely broken. Fix it.
  • Implement a Caviar FS walk function to shadow path/filepath.Walk().
  • There is a bit of a type casting mess. Make everything use int64 and be done with it.
  • Port caviarize to Go.
  • Get auxiliary os methods working (Stat, Lstat, etc.).
  • Make Revel work.
  • Make Martini work.
  • Allow directories to be open (Martini seems to need this).
  • Make sure all functions only operate when Caviar is ready and error out otherwise.
  • Setup Travis CI/Wercker and Godoc.
  • Make sure no CaviarFile related function allows itself to be called on a closed file. Should return os.ErrInvalid.
  • Apparently, in for…range…{} constructs, range makes a copy of the object being looped, which is really bad for the object tree which could potentially be pretty big. Go through the code and make the loops not use range when iterating over the object tree. Or perhaps this is pointless, as the objects themselves hold children on a slice which is a reference type, right?
  • Functions such as CaviarFile.Readdir() should merge their own output with that of the native OS and merge them together.
  • Run some benchmarks on just how much faster (or slower) Caviar is relative to the native OS both for files in and out of the kernel's disk cache.
  • More documentation.
  • Move issues to GitHub's bug tracker.
  • Make Caviar faster. Maybe augment findObject() to use a hash table for path search. Should really write a decent set of benchmarks that run for a set of cases and then dumps the raw data for comparison to a CSV I can graph. The basic cases I'm interested in are: Data source:
    • RAM disk via OS.
    • RAM disk via Caviar.
    • Normal HDD via OS.
    • Normal HDD via Caviar.
    • Caviar bundle. Combines with one of the following options:
    • Few assets (100).
    • Many assets (10000).
    • Fuckton of assets (1000000).
    • Small assets (~10 KiB).
    • Large assets (~10 MiB).
    • Huge assets (~100 MiB).
  • Add ciavirize-like verbose output to cavundle and make both of them produce no output by default (with a -v switch to turn it on or something).
  • BUG: with a large, ~100 MiB bundle, the toy examples use around 230 MiB of RAM after initialization, which makes no sense. It's as if I was making 2 copies of the payload. Forcing garbage collection via runtime.GC() makes no difference. Investigate.

Documentation

Overview

Caviar is an asset/resource packer for Go.

Index

Constants

View Source 
const (
	// Load everything to RAM
	EXTRACT_MEMORY = iota
	// Unpack everything to a temp dir and proxy OS Open/OpenFile calls to make
	// the resources available from the expected FS locations. Files are
	// deleted cleanly when the program exits.
	EXTRACT_TEMP
	// Same as EXTRACT_TEMP, but extract to the executable's root directory.
	EXTRACT_EXECUTABLE
)
View Source 
const CAVIAR_EXTENSION = "cvr"

File extension for Caviar containers.

View Source 
const MANIFEST_MAGIC = "CAVIAR"

Manifest-level magic value

View Source 
const OBJECTROOT_MAGIC = "ROOT"

Asset-root-level magic value

View Source 
const READ_MAX = 1024 * 32

Maximum number of bytes to read when calling Read().

Variables

This section is empty.

Functions

func DetachedName 

func DetachedName(p string) string

Returns the name of a detached container for a given program “p”.

func Glob 

func Glob(pattern string) (matches []string, err error)

func Init 

func Init() (err error)

Init sets up Caviar's internal state and loads the bundle, if any.

func Lstat 

func Lstat(name string) (fi os.FileInfo, err error)

Lstat mimicks os.Lstat(). Please note Caviar does not currently support symlinks inside the bundle so Lstat() will be have identically to Stat() for paths matching files and directories inside the bundle.

func PayloadSize 

func PayloadSize() int64

Returns the total number of bytes for all loaded assets.

func ReadDir 

func ReadDir(dirname string) ([]os.FileInfo, error)

TODO

func ReadFile 

func ReadFile(filename string) ([]byte, error)

TODO

func Stat 

func Stat(name string) (fi os.FileInfo, err error)

Stat mimicks os.Stat()

func Walk 

func Walk(root string, walkFn filepath.WalkFunc) error

Types

type BundleOptions 

type BundleOptions struct {
	// The asset root will be placed in a fixed location (i.e.
	// “/usr/share/myprogram”) instead of the executable's directory, whatever
	// that happens to be if CustomPrefix is set to anything other than an
	// empty string.
	CustomPrefix string
	// This will make Caviar print out various log messages to the terminal.
	Debug bool
	// See EXTRACT_* constants above.
	ExtractionMode int
}

Various options to be set by the program creating the bundle. They will affect Caviar's run-time behaviour.

type CaviarFile 

type CaviarFile struct {
	// contains filtered or unexported fields
}

CaviarFile implements caviar.File and serves as a replacement for os.File.

func (*CaviarFile) Chdir 

func (f *CaviarFile) Chdir() error

Chdir mimicks os.File.Chdir(). It will return with an error unless EXTRACT_EXECUTABLE is used as the extraction mode for the bundle as it's not possible to chdir to a virtual, in-memory directory (EXTRACT_MEMORY) and doing so for EXTRACT_TEMP would screw up relative paths causing subtle bugs.

func (*CaviarFile) Chmod 

func (f *CaviarFile) Chmod(mode os.FileMode) error

Chmod mimicks os.File.Chmod(). It always returns an error as Caviar files are read-only.

func (*CaviarFile) Chown 

func (f *CaviarFile) Chown(uid, gid int) error

Chown mimicks os.File.Chown(). It always returns an error as Caviar files are read-only.

func (*CaviarFile) Close 

func (f *CaviarFile) Close() error

Close mimicks os.File.Close()

func (*CaviarFile) Fd 

func (f *CaviarFile) Fd() uintptr

Fd mimicks os.File.Fd(). The returned file descriptor is a dummy value that is unlikely to repeat across Open files (but no guarantees).

func (*CaviarFile) Name 

func (f *CaviarFile) Name() string

Name mimicks os.File.Name().

func (*CaviarFile) Read 

func (f *CaviarFile) Read(b []byte) (int, error)

Read mimicks os.File.Read().

func (*CaviarFile) ReadAt 

func (f *CaviarFile) ReadAt(b []byte, off int64) (int, error)

ReadAt mimicks os.File.ReadAt().

func (*CaviarFile) Readdir 

func (f *CaviarFile) Readdir(n int) (fi []os.FileInfo, err error)

Readdir mimicks os.File.Readdir().

func (*CaviarFile) Readdirnames 

func (f *CaviarFile) Readdirnames(n int) (names []string, err error)

Readdirnames mimicks os.File.Readdirnames().

func (*CaviarFile) Seek 

func (f *CaviarFile) Seek(offset int64, whence int) (pos int64, err error)

Seek mimicks os.File.Seek().

func (*CaviarFile) Stat 

func (f *CaviarFile) Stat() (os.FileInfo, error)

Stat mimicks os.File.Stat().

func (*CaviarFile) Sync 

func (f *CaviarFile) Sync() (err error)

Sync mimicks os.File.Sync(). It always returns an error as Caviar files are read-only.

func (*CaviarFile) Truncate 

func (f *CaviarFile) Truncate(size int64) error

Truncate mimicks os.File.Truncate(). It always returns an error as Caviar files are read-only.

func (*CaviarFile) Write 

func (f *CaviarFile) Write(b []byte) (n int, err error)

Write mimicks os.File.Write(). It always returns an error as Caviar files are read-only.

func (*CaviarFile) WriteAt 

func (f *CaviarFile) WriteAt(b []byte, off int64) (int, error)

WriteAt mimicks os.File.WriteAt(). It always returns an error as Caviar files are read-only.

func (*CaviarFile) WriteString 

func (f *CaviarFile) WriteString(s string) (int, error)

WriteString mimicks os.File.WriteString(). It always returns an error as Caviar files are read-only.

type CaviarFileInfo 

type CaviarFileInfo struct {
	// contains filtered or unexported fields
}

CaviarFileInfo implements os.FileInfo and should behave exactly the same as the os package's implementation for native OS files.

func (*CaviarFileInfo) IsDir 

func (fi *CaviarFileInfo) IsDir() bool

func (*CaviarFileInfo) ModTime 

func (fi *CaviarFileInfo) ModTime() time.Time

func (*CaviarFileInfo) Mode 

func (fi *CaviarFileInfo) Mode() os.FileMode

func (*CaviarFileInfo) Name 

func (fi *CaviarFileInfo) Name() string

func (*CaviarFileInfo) Size 

func (fi *CaviarFileInfo) Size() int64

func (*CaviarFileInfo) Sys 

func (fi *CaviarFileInfo) Sys() interface{}

type Dir 

type Dir string

Replacement for net/http.Dir.

func (Dir) Open 

func (d Dir) Open(name string) (http.File, error)

Caviarized copy of net/http.Dir.Open()

type File 

type File interface {
	io.Reader
	io.ReaderAt
	io.Writer
	io.WriterAt
	io.Seeker
	io.Closer
	Stat() (os.FileInfo, error)
	Name() string
	Chdir() error
	Sync() error
	Fd() uintptr
	Truncate(int64) error
	WriteString(string) (int, error)
	Chmod(os.FileMode) error
	Chown(int, int) error
	Readdir(n int) (fi []os.FileInfo, err error)
	Readdirnames(n int) (names []string, err error)
}

File mimicks the os.File type's entire public API so Caviar can serve as a drop-in replacement.

func CaviarOpen 

func CaviarOpen(name string) (File, error)

CaviarOpen behaves the same way as Open but it will only attempt to open files and directories contained within the bundle and will not pass along the call to os.Open() on failure.

func CaviarOpenFile 

func CaviarOpenFile(name string, flag int, perm os.FileMode) (File, error)

CaviarOpenFile is to OpenFile what CaviarOpen is to Open.

func Open 

func Open(name string) (File, error)

Open mimicks os.Open. It will first attempt to open the file as an internal Caviar file and failing that it will pass along the call to the os package.

func OpenFile 

func OpenFile(name string, flag int, perm os.FileMode) (File, error)

OpenFile mimicks os.OpenFile. It will first attempt to open the file as an internal Caviar file and failing that it will pass along the call to the os package.

type Manifest 

type Manifest struct {
	// Magic should be set to MANIFEST_MAGIC
	Magic string
	// Free-formatted comment string added by the program generating the
	// bundle. Caviar's own cavundle adds information about Caviar and a
	// copyright statement.
	Comment string
	// Additional options set when creating the bundle. These are needed as
	// Caviar's initialization routine is called from within an init() method
	// and thus can take no input from the program itself.
	Options BundleOptions
	// The root directory object.
	ObjectRoot Object
}

Manifest describes the contents of a Caviar bundle and it's serialized by cavundle along with the raw data of all assets when creating a bundle.

type Object 

type Object struct {
	// Name of the file or directory. This is NOT a full path but rather one
	// segment (for instance, the file /home/martin/DATA would require 3
	// objects named “home”, “martin”, and “DATA”).
	Name string
	// FileMode as returned by os.File.Stat(). Note that os.ModeDir must be set
	// if the object represents a directory.
	ModeBits os.FileMode
	// Modification time.
	ModTime int64
	// Size of the file. Set to 0 for directories.
	Size int64
	// All assets inside a bundle are packed inside a single flat binary file
	// next to the serialized Manifest. Offset represents the offset from the
	// beginning of the asset binary file to the beginning of the data for the
	// file the given Object represents. Must be set to 0 for directories and
	// empty files.
	Offset int64
	// CRC32 checksum for the file's contents. Set to 0 for directories.
	Checksum uint32
	// Child objects (sub-directories and contained files). File objects must
	// not have any children.
	Objects []Object
}

Object represents either a file or a directory inside the bundle.

Source Files

View all Source files

Directories

Path Synopsis
cmd
caviarize
Caviarize is meant to help patch dependencies that need to access files and directories you'd like to put in a Caviar bundle.
 Caviarize is meant to help patch dependencies that need to access files and directories you'd like to put in a Caviar bundle.
cavundle
Cavundle creates asset bundles for Caviar-enabled programs.
 Cavundle creates asset bundles for Caviar-enabled programs.
examples
martini
This example showcases a trivial Martini app mostly serving static files from a bundle.
 This example showcases a trivial Martini app mostly serving static files from a bundle.
serve
This little program creates a very simple (and very fast) Web server to serve all files in the current working directory.
 This little program creates a very simple (and very fast) Web server to serve all files in the current working directory.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.