libzipfs

package module
v0.0.0-...-13d7bfb Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 30, 2015 License: MIT Imports: 17 Imported by: 1

README

libzipfs

-------------------       ---------------
| go executable   |       |  zip file   |
-------------------       ---------------
        \                        /
         --> libzipfs-combiner <-
                      |
                      v
----------------------------------------------------
| go executable   |  zip file   |  256-byte footer | = a new single executable with all media inside
----------------------------------------------------

Libzipfs lets you ship a filesystem of media resources inside your golang application. This is done by attaching a Zip file containing the directories of your choice to the end of your application, and following it with a short footer to allow the go executable to locate the files and serve them via a fuse mount point. The client will typically be either be Go or C embedded in the go executable (see for example testfiles/api.go), but can be another application altogether. For the later use, see for example the libzipfs/cmd/mountzip/mountzip.go example source code.

Use cases

  1. You have a bunch of images/scripts/files to be served from a webserving application, and you want to bundle those resources alongside your Go webapp. libzipfs lets you easily create a single executable that contains all your resources in one place.

  2. If you are using CGO to call C code, and that C code expects to be able to read files from somewhere on the filesystem, you can package up all those files, ship them with the executable, and they can be read from the fuse mountpoint -- where the C code can find and use them. For example, my https://github.com/glycerine/rmq project embeds R inside a Go binary, and libzipfs allows R libraries to be easily shipped all together in a single binary.

status

Excellent. Works well and is very useful. I only use it on OSX and Linux. On OSX you need to have OSX Fuse installed first. On Linux you'll need to either sudo yum install fuse or sudo apt-get install fuse-utils to obtain the /bin/fusermount utility.

installation

$ go get -t -u -v github.com/glycerine/libzipfs
$ cd $GOPATH/src/github.com/glycerine/libzipfs && make
$ ## note 1: the libzipfs-combiner and mountzip utilities are now in your $GOPATH/bin
$ ## note 2: be sure that $GOPATH/bin is added to your PATH env variable
$ ##         e.g. in your ~/.bashrc you have: export PATH=$GOPATH/bin:$PATH
$ go test -v  # to run the test suite
$ make demo   # to see the demo code run

origins

The libzipfs library is derived from Tommi Virtanen's work https://github.com/bazil/zipfs, which is fantastic and provides a fuse-mounted read-only filesystem from a Zip file. The zipfs library and https://github.com/bazil/fuse are doing the heavy lifting behind the scenes.

The project was inspired by https://github.com/bazil/zipfs and https://github.com/shurcooL/vfsgen

In particular, vfsgen is a similar approach, but I needed the ability to serve files to legacy code that expects to read from a file system.

libzipfs: a fully integrated solution

Libzipfs builds on top of zipfs to allow developers the ability to assemble a "combined" file from an executable and a Zip file containing the directories you wish to have available to your application at runtime. The structure of the combined file looks like this:

----------------------------------------------------
| executable      |  zip file   |  256-byte footer |
----------------------------------------------------
^                                                  ^
byte 0                                           byte N

The embedded Zip file (the middle part in the diagram above) can then be made available via a fuse mountpoint. The Go executable will contain Go code to accomplish this. The 256-bite footer at the end of the file describes the location of the embedded zip file. The combined file is still an executable, and can be run directly.

creating a combined executable and Zip file

the libzipfs-combiner utility does this for you.

For example: assuming that my.go.binary and hi.zip already exist, and you wish to create a new combo executable called my.go.binary.combo, you would do:

$ libzipfs-combiner --help
libzipfs-combiner --help
Usage of libzipfs-combiner:
  -exe string
    	path to the executable file
  -o string
    	path to the combined output file to be written (or split if -split given)
  -split
    	split the output file back apart (instead of combine which is the default)
  -zip string
    	path to the Zip file to embed

$ libzipfs-combiner --exe my.go.binary -o my.go.binary.combo -zip hi.zip
api/code code inside your my.go.binary.combo binary:

type make demo and see testfiles/api.go for a full demo:

Our demo zip file testfiles/hi.zip is a simple zip file with one file hello that resides inside two nested directories:

$ unzip -Z -z testfiles/hi.zip
Archive:  testfiles/hi.zip   478 bytes   3 files
drwxr-xr-x  3.0 unx        0 bx stor 19-Dec-15 17:27 dirA/
drwxr-xr-x  3.0 unx        0 bx stor 19-Dec-15 17:27 dirA/dirB/
-rw-r--r--  3.0 unx       12 tx stor 19-Dec-15 17:27 dirA/dirB/hello
3 files, 12 bytes uncompressed, 12 bytes compressed:  0.0%

the go code:

package main

import (
	"bytes"
	"fmt"
	"io/ioutil"
	"path"

	"github.com/glycerine/libzipfs"
)

// build instructions:
//
// cd libzipfs && make
// cd testfiles; go build -o api-demo api.go
// libzipfs-combiner -exe api-demo -zip hi.zip -o api-demo-combo
// ./api-demo-combo

func main() {
	z, mountpoint, err := libzipfs.MountComboZip()
	if err != nil {
		panic(err)
	}
	defer z.Stop() // if you want to stop serving files

	// access the files from `hi.zip` at mountpoint

	by, err := ioutil.ReadFile(path.Join(mountpoint, "dirA", "dirB", "hello"))
	if err != nil {
		panic(err)
	}
	by = bytes.TrimRight(by, "\n")
	fmt.Printf("we should see our file dirA/dirB/hello from inside hi.zip, containing 'salutations'.... '%s'\n", string(by))

	if string(by) != "salutations" {
		panic("problem detected")
	}
	fmt.Printf("Excellent: all looks good.\n")
}

example number 2: mountzip

The mountzip utility (see the source code in libzipfs/cmd/mountzip/mountzip.go) mounts a zip file of your choice on a directory of your choice.

$ cd $GOPATH/src/github.com/glycerine/libzipfs
$ make # installs the mountzip utility into $GOPATH/bin
$ mountzip -help
Usage of mountzip:
  -mnt string
    	directory to fuse-mount the Zip file on
  -zip string
    	path to the Zip file to mount
$
$ mkdir /tmp/hi
$ mountzip -zip testfiles/hi.zip -mnt /tmp/hi
Zip file 'hi.zip' mounted at directory '/tmp/hi'. [press ctrl-c to exit and unmount]

license

MIT license. See enclosed LICENSE file.

Documentation

Overview

combiner appends a zip file to an executable and further appends a footer in the last 256 bytes that describes the combination. libzipfs will look for this footer and use it to determine where the internalized zipfile filesystem starts.

combiner appends a zip file to an executable and further appends a footer in the last 256 bytes that describes the combination. libzipfs will look for this footer and use it to determine where the internalized zipfile filesystem starts.

Index

Constants

View Source 
const BLAKE2_HASH_LEN = 64
View Source 
const LIBZIPFS_FOOTER_LEN = 256
View Source 
const MAGIC_NUM_LEN = 16

Variables

View Source 
var GITLASTCOMMIT string // git rev-parse HEAD
View Source 
var GITLASTTAG string // git describe --abbrev=0 --tags

track git version of this lib

View Source 
var MAGIC1 = []byte("\nLibZipFs00\n")
View Source 
var MAGIC2 = []byte("\nLibZipFsEnd\n")
View Source 
var Verbose bool

Functions

func Blake2HashFile 

func Blake2HashFile(path string) (hash []byte, length int64, err error)

func DirExists 

func DirExists(name string) bool

func DisplayVersionAndExitIfRequested 

func DisplayVersionAndExitIfRequested()

func DoCombineExeAndZip 

func DoCombineExeAndZip(cfg *CombinerConfig) error

func FileExists 

func FileExists(name string) bool

func FindMount 

func FindMount() error

func FindMountUmount 

func FindMountUmount() error

func FindUmount 

func FindUmount() error

func ShouldRetry 

func ShouldRetry(err error) bool

helper for reading in a loop. will panic on unknown error.

func TSPrintf 

func TSPrintf(format string, a ...interface{})

time-stamped printf

func TrimTrailingSlashes 

func TrimTrailingSlashes(mountpoint string) string

must trim any trailing slash from the mountpoint, or else mount can fail

func VPrintf 

func VPrintf(format string, a ...interface{})

func VersionString 

func VersionString() string

func WaitUntilMounted 

func WaitUntilMounted(mountPoint string) error

func WaitUntilUnmounted 

func WaitUntilUnmounted(mountPoint string) error

Types

type CombinerConfig 

type CombinerConfig struct {
	ExecutablePath string
	ZipfilePath    string
	OutputPath     string
	Split          bool
}

func (*CombinerConfig) DefineFlags 

func (c *CombinerConfig) DefineFlags(fs *flag.FlagSet)

call DefineFlags before myflags.Parse()

func (*CombinerConfig) ValidateConfig 

func (c *CombinerConfig) ValidateConfig() error

call c.ValidateConfig() after myflags.Parse()

type Dir 

type Dir struct {
	// contains filtered or unexported fields
}

func (*Dir) Attr 

func (d *Dir) Attr(ctx context.Context, a *fuse.Attr) error

func (*Dir) Lookup 

func (d *Dir) Lookup(ctx context.Context, req *fuse.LookupRequest, resp *fuse.LookupResponse) (fs.Node, error)

func (*Dir) ReadDirAll 

func (d *Dir) ReadDirAll(ctx context.Context) ([]fuse.Dirent, error)

type FS 

type FS struct {
	// contains filtered or unexported fields
}

func (*FS) Root 

func (f *FS) Root() (fs.Node, error)

type File 

type File struct {
	// contains filtered or unexported fields
}

func (*File) Attr 

func (f *File) Attr(ctx context.Context, a *fuse.Attr) error

func (*File) Open 

func (f *File) Open(ctx context.Context, req *fuse.OpenRequest, resp *fuse.OpenResponse) (fs.Handle, error)

type FileHandle 

type FileHandle struct {
	// contains filtered or unexported fields
}

func (*FileHandle) Read 

func (fh *FileHandle) Read(ctx context.Context, req *fuse.ReadRequest, resp *fuse.ReadResponse) error

func (*FileHandle) Release 

func (fh *FileHandle) Release(ctx context.Context, req *fuse.ReleaseRequest) error
type Footer struct {
	Reserved1          int64
	MagicFooterNumber1 [MAGIC_NUM_LEN]byte

	ExecutableLengthBytes int64
	ZipfileLengthBytes    int64
	FooterLengthBytes     int64

	ExecutableBlake2Checksum [BLAKE2_HASH_LEN]byte
	ZipfileBlake2Checksum    [BLAKE2_HASH_LEN]byte
	FooterBlake2Checksum     [BLAKE2_HASH_LEN]byte // has itself set to zero when taking the hash.

	MagicFooterNumber2 [MAGIC_NUM_LEN]byte
}

func DoSplitOutExeAndZip 

func DoSplitOutExeAndZip(cfg *CombinerConfig) (*Footer, error)

func ReadFooter 

func ReadFooter(combinedPath string) (footerStartOffset int64, ft *Footer, comb *os.File, err error)

client take responsibility for closing combFd when done with it; it is the open file handled (if err == nil) for reading from the file at combinedPath.

func ReifyFooterAndDoInexpensiveChecks 

func ReifyFooterAndDoInexpensiveChecks(by []byte, combinedPath string, footerStartOffset int64) (*Footer, error)

must return err if foot is bad

func (*Footer) FillHashes 

func (foot *Footer) FillHashes(cfg *CombinerConfig) error

func (*Footer) FromBytes 

func (f *Footer) FromBytes(by []byte)

func (*Footer) GetFooterChecksum 

func (foot *Footer) GetFooterChecksum() []byte

func (*Footer) ToBytes 

func (f *Footer) ToBytes() []byte

func (*Footer) VerifyExeZipChecksums 

func (foot *Footer) VerifyExeZipChecksums(cfg *CombinerConfig) (err error)

type FooterArray 

type FooterArray [LIBZIPFS_FOOTER_LEN]byte

type FuseZipFs 

type FuseZipFs struct {
	ZipfilePath string
	MountPoint  string

	Ready   chan bool
	ReqStop chan bool
	Done    chan bool
	// contains filtered or unexported fields
}

func MountComboZip 

func MountComboZip() (fzfs *FuseZipFs, mountpoint string, err error)

The Main API entry point for mounting a combo file vis FUSE to make the embedded Zip file directory available. Users should call fzfs.Stop() when/if they wish to stop serving files at mountpoint.

func NewFuseZipFs 

func NewFuseZipFs(zipFilePath, mountpoint string, byteOffsetToZipFileStart int64, bytesAvail int64, footerBytes int64) *FuseZipFs

Mount a possibly combined/zipfile at mountpiont. Call Start() to start servicing fuse reads.

If the file has a libzipfs footer on it, set footerBytes == LIBZIPFS_FOOTER_LEN. The bytesAvail value should describe how long the zipfile is in bytes, and byteOffsetToZipFileStart should describe how far into the (possibly combined) zipFilePath the actual zipfile starts.

func NewFuzeZipFsFromCombo 

func NewFuzeZipFsFromCombo(comboFilePath string) (fzfs *FuseZipFs, mountpoint string, err error)

mount the comboFilePath file in a temp directory mountpoint created just for this purpose, and return the mountpoint and a handle to the fuse fileserver in fzfs.

func (*FuseZipFs) Start 

func (p *FuseZipFs) Start() error

func (*FuseZipFs) Stop 

func (p *FuseZipFs) Stop() error

Source Files

View all Source files

Directories

Path Synopsis
cmd
libzipfs-combiner
combiner appends a zip file to an executable and further appends a footer in the last 256 bytes that describes the combination.
 combiner appends a zip file to an executable and further appends a footer in the last 256 bytes that describes the combination.
mountzip

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.