xbindata

package module
v0.0.13 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 16, 2020 License: BSD-2-Clause, CC0-1.0 Imports: 42 Imported by: 1

README

zbindata

This fork is maintained by Moises P. Sena. Don't expect many new features, though I will take a look at PR's. Changes made include:

  • Atomic writes; generated file cannot be read while partially complete.

  • Generated file reports file sizes.

  • Generated code is run through go fmt.

  • SHA256 hashes are computed for all files and stored in the binary. You can use this to detect in-memory corruption and to provide easy cache-busting mechanisms.

  • Added AssetString and MustAssetString functions.

  • ByName is not public.

  • Some errors in file writes were unchecked, but are now checked.

  • Asset File System

This package converts any file into manageable Go source code. Useful for embedding binary data into a go program. The file data is optionally gzip compressed before being converted to a raw byte slice.

It comes with a command line tool in the xbindata subdirectory. This tool offers a set of command line options, used to customize the output being generated.

Installation

Using GO! installation:

go get -u github.com/moisespsena-go/xbindata/xb

Usage

Conversion is done on one or more sets of files. They are all embedded in a new Go source file, along with a table of contents and an Asset function, which allows quick access to the asset, based on its name.

The simplest invocation generates a xbindata.go file in the current working directory. It includes all assets from the data directory.

$ xbindata data/

To include all input sub-directories recursively, use the ellipsis postfix as defined for Go import paths. Otherwise it will only consider assets in the input directory itself.

$ xbindata data/...

To specify the name of the output file being generated, use the -o option:

$ xbindata -o myfile.go data/

Multiple input directories can be specified if necessary.

$ xbindata dir1/... /path/to/dir2/... dir3

The following paragraphs detail some of the command line options which can be supplied to xbindata. Refer to the testdata/out directory for various output examples from the assets in testdata/in. Each example uses different command line options.

To ignore files, pass in regexes using -ignore, for example:

$ xbindata -ignore=\\.gitignore data/...

Accessing an asset

To access asset data, we use the Asset(string) ([]byte, error) function which is included in the generated output.

data, err := Asset("pub/style/foo.css")
if err != nil {
	// Asset was not found.
}

// use asset data

Debug vs Release builds

When invoking the program with the -debug flag, the generated code does not actually include the asset data. Instead, it generates function stubs which load the data from the original file on disk. The asset API remains identical between debug and release builds, so your code will not have to change.

This is useful during development when you expect the assets to change often. The host application using these assets uses the same API in both cases and will not have to care where the actual data comes from.

An example is a Go webserver with some embedded, static web content like HTML, JS and CSS files. While developing it, you do not want to rebuild the whole server and restart it every time you make a change to a bit of javascript. You just want to build and launch the server once. Then just press refresh in the browser to see those changes. Embedding the assets with the debug flag allows you to do just that. When you are finished developing and ready for deployment, just re-invoke xbindata without the -debug flag. It will now embed the latest version of the assets.

Lower memory footprint

Using the -nomemcopy flag, will alter the way the output file is generated. It will employ a hack that allows us to read the file data directly from the compiled program's .rodata section. This ensures that when we call our generated function, we omit unnecessary memcopies.

The downside of this, is that it requires dependencies on the reflect and unsafe packages. These may be restricted on platforms like AppEngine and thus prevent you from using this mode.

Another disadvantage is that the byte slice we create, is strictly read-only. For most use-cases this is not a problem, but if you ever try to alter the returned byte slice, a runtime panic is thrown. Use this mode only on target platforms where memory constraints are an issue.

The default behaviour is to use the old code generation method. This prevents the two previously mentioned issues, but will employ at least one extra memcopy and thus increase memory requirements.

For instance, consider the following two examples:

This would be the default mode, using an extra memcopy but gives a safe implementation without dependencies on reflect and unsafe:

func myfile() []byte {
    return []byte{0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a}
}

Here is the same functionality, but uses the .rodata hack. The byte slice returned from this example can not be written to without generating a runtime error.

var _myfile = "\x89\x50\x4e\x47\x0d\x0a\x1a"

func myfile() []byte {
    var empty [0]byte
    sx := (*reflect.StringHeader)(unsafe.Pointer(&_myfile))
    b := empty[:]
    bx := (*reflect.SliceHeader)(unsafe.Pointer(&b))
    bx.Data = sx.Data
    bx.Len = len(_myfile)
    bx.Cap = bx.Len
    return b
}

Optional compression

When the -nocompress flag is given, the supplied resource is not GZIP compressed before being turned into Go code. The data should still be accessed through a function call, so nothing changes in the usage of the generated file.

This feature is useful if you do not care for compression, or the supplied resource is already compressed. Doing it again would not add any value and may even increase the size of the data.

The default behaviour of the program is to use compression.

File System

When the -fs flag is given, the file system resource is available on AssetFS global variable.

import "github.com/moisespsena-go/assetfs/api"

var AssetFS api.Interface

Path prefix stripping

The keys used in the assets map, are the same as the input file name passed to bindata. This includes the path. In most cases, this is not desireable, as it puts potentially sensitive information in your code base. For this purpose, the tool supplies another command line flag -prefix. This accepts a portion of a path name, which should be stripped off from the map keys and function names.

For example, running without the -prefix flag, we get:

$ xbindata /path/to/templates/

assets["/path/to/templates/foo.html"] = path_to_templates_foo_html

Running with the -prefix flag, we get:

$ xbindata -prefix "/path/to/" /path/to/templates/

assets["templates/foo.html"] = templates_foo_html

Build tags

With the optional -tags flag, you can specify any go build tags that must be fulfilled for the output file to be included in a build. This is useful when including binary data in multiple formats, where the desired format is specified at build time with the appropriate tags.

The tags are appended to a // +build line in the beginning of the output file and must follow the build tags syntax specified by the go tool.

assetfs - implements http.FileSystem interface. Allows you to serve assets with net/http.

Documentation

Overview

xbindata converts any file into managable Go source code. Useful for embedding binary data into a go program. The file data is optionally gzip compressed before being converted to a raw byte slice.

The following paragraphs cover some of the customization options which can be specified in the Config struct, which must be passed into the Translate() call.

Debug vs Release builds

When used with the `Debug` option, the generated code does not actually include the asset data. Instead, it generates function stubs which load the data from the original file on disk. The asset API remains identical between debug and release builds, so your code will not have to change.

This is useful during development when you expect the assets to change often. The host application using these assets uses the same API in both cases and will not have to care where the actual data comes from.

An example is a Go webserver with some embedded, static web content like HTML, JS and CSS files. While developing it, you do not want to rebuild the whole server and restart it every time you make a change to a Bit of javascript. You just want to build and launch the server once. Then just press refresh in the browser to see those changes. Embedding the assets with the `debug` flag allows you to do just that. When you are finished developing and ready for deployment, just re-invoke `xbindata` without the `-debug` flag. It will now embed the latest version of the assets.

Lower memory footprint

The `NoMemCopy` option will alter the way the output file is generated. It will employ a hack that allows us to read the file data directly from the compiled program's `.rodata` section. This ensures that when we call call our generated function, we omit unnecessary memcopies.

The downside of this, is that it requires dependencies on the `reflect` and `unsafe` packages. These may be restricted on platforms like AppEngine and thus prevent you from using this mode.

Another disadvantage is that the byte slice we create, is strictly read-only. For most use-cases this is not a problem, but if you ever try to alter the returned byte slice, a runtime panic is thrown. Use this mode only on target platforms where memory constraints are an issue.

The default behaviour is to use the old code generation method. This prevents the two previously mentioned issues, but will employ at least one extra memcopy and thus increase memory requirements.

For instance, consider the following two examples:

This would be the default mode, using an extra memcopy but gives a safe implementation without dependencies on `reflect` and `unsafe`: 

func myfile() []byte {
	return []byte{0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a}
}

Here is the same functionality, but uses the `.rodata` hack. The byte slice returned from this example can not be written to without generating a runtime error. 

var _myfile = "\x89\x50\x4e\x47\x0d\x0a\x1a"

func myfile() []byte {
	var empty [0]byte
	sx := (*reflect.StringHeader)(unsafe.Pointer(&_myfile))
	b := empty[:]
	bx := (*reflect.SliceHeader)(unsafe.Pointer(&b))
	bx.Data = sx.Data
	bx.Len = len(_myfile)
	bx.Cap = bx.Len
	return b
}

Optional compression

The NoCompress option indicates that the supplied assets are *not* GZIP compressed before being turned into Go code. The data should still be accessed through a function call, so nothing changes in the API.

This feature is useful if you do not care for compression, or the supplied resource is already compressed. Doing it again would not add any value and may even increase the size of the data.

The default behaviour of the program is to use compression.

Path prefix stripping

The keys used in the `_bindata` map are the same as the input file name passed to `xbindata`. This includes the path. In most cases, this is not desireable, as it puts potentially sensitive information in your code base. For this purpose, the tool supplies another command line flag `-prefix`. This accepts a portion of a path name, which should be stripped off from the map keys and function names.

For example, running without the `-prefix` flag, we get: 

$ xbindata /path/to/templates/

_bindata["/path/to/templates/foo.html"] = path_to_templates_foo_html

Running with the `-prefix` flag, we get: 

$ xbindata -prefix "/path/to/" /path/to/templates/

_bindata["templates/foo.html"] = templates_foo_html

Build tags

With the optional Tags field, you can specify any go build tags that must be fulfilled for the output file to be included in a build. This is useful when including binary data in multiple formats, where the desired format is specified at build time with the appropriate tags.

The tags are appended to a `// +build` line in the beginning of the output file and must follow the build tags syntax specified by the go tool.

Index

Constants

View Source 
const (
	OutputToProgram = "_"
	OutputToStdout  = "-"
	DefaultDataDir  = "assets"
)
View Source 
const (
	ContextEnvKey configContextKeyType = iota
	ContextInputKey
)
View Source 
const DefaultOutput = "assets.go"

Variables

This section is empty.

Functions

func ContextEnv added in v0.0.6 

func ContextEnv(ctx context.Context) map[string]string

func ContextWithEnv added in v0.0.6 

func ContextWithEnv(ctx context.Context, env map[string]string, noInherits ...bool) context.Context

func ContextWithInputKey added in v0.0.6 

func ContextWithInputKey(ctx context.Context, key string) context.Context

func Env added in v0.0.6 

func Env(update ...map[string]string) map[string]string

func EnvS added in v0.0.6 

func EnvS(update ...map[string]string) (env []string)

func InputKey added in v0.0.6 

func InputKey(ctx context.Context, format ...string) (key string)

func Translate 

func Translate(c *Config) (count int, err error)

Translate reads assets from an input directory, converts them to Go code and writes new files to the output specified in the given configuration.

Types

type Asset 

type Asset struct {
	Path string // Full file path.
	Name string // Key used in TOC -- name by which asset is referenced.
	Func string // Function name for the procedure returning the asset contents.
	Size int64

	Prefix string
	// contains filtered or unexported fields
}

Asset holds information about a single asset to be processed.

func (*Asset) Digest 

func (a *Asset) Digest() (dig *[sha256.Size]byte, err error)

func (*Asset) Info 

func (a *Asset) Info() (info os.FileInfo, err error)

func (*Asset) InfoExport 

func (a *Asset) InfoExport(c *Config) (string, error)

func (*Asset) SourceCode 

func (a *Asset) SourceCode(c *Config, start int64) (code string, err error)

type ByteWriter 

type ByteWriter struct {
	io.Writer
	// contains filtered or unexported fields
}

ByteWriter writes the hex-encoded version of input bytes to its Writer.

func (*ByteWriter) Write 

func (w *ByteWriter) Write(p []byte) (n int, err error)

type Config 

type Config struct {
	// Name of the package to use. Defaults to 'main'.
	Package string

	// Tags specify a set of optional build tags, which should be
	// included in the generated output. The tags are appended to a
	// `// +build` line in the beginning of the output file
	// and must follow the build tags syntax specified by the go tool.
	Tags []string

	// Input defines the directory path, containing all asset files as
	// well as whether to recursively process assets in any sub directories.
	Input []InputConfig

	// Output defines the output file for the generated code.
	// If left empty, this defaults to 'xbindata.go' in the current
	// working directory.
	Output string

	// OutputWriter defines the output writer for the generated code.
	OutputWriter io.Writer

	// Prefix defines a path prefix which should be stripped from all
	// file names when generating the keys in the table of contents.
	// For example, running without the `-prefix` flag, we get:
	//
	// 	$ xbindata /path/to/templates
	// 	go_bindata["/path/to/templates/foo.html"] = _path_to_templates_foo_html
	//
	// Running with the `-prefix` flag, we get:
	//
	// 	$ xbindata -prefix "/path/to/" /path/to/templates/foo.html
	// 	go_bindata["templates/foo.html"] = templates_foo_html
	Prefix string

	// NoMemCopy will alter the way the output file is generated.
	//
	// It will employ a hack that allows us to read the file data directly from
	// the compiled program's `.rodata` section. This ensures that when we call
	// call our generated function, we omit unnecessary mem copies.
	//
	// The downside of this, is that it requires dependencies on the `reflect` and
	// `unsafe` packages. These may be restricted on platforms like AppEngine and
	// thus prevent you from using this mode.
	//
	// Another disadvantage is that the byte slice we create, is strictly read-only.
	// For most use-cases this is not a problem, but if you ever try to alter the
	// returned byte slice, a runtime panic is thrown. Use this mode only on target
	// platforms where memory constraints are an issue.
	//
	// The default behaviour is to use the old code generation method. This
	// prevents the two previously mentioned issues, but will employ at least one
	// extra memcopy and thus increase memory requirements.
	//
	// For instance, consider the following two examples:
	//
	// This would be the default mode, using an extra memcopy but gives a safe
	// implementation without dependencies on `reflect` and `unsafe`:
	//
	// 	func myfile() []byte {
	// 		return []byte{0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a}
	// 	}
	//
	// Here is the same functionality, but uses the `.rodata` hack.
	// The byte slice returned from this example can not be written to without
	// generating a runtime error.
	//
	// 	var _myfile = "\x89\x50\x4e\x47\x0d\x0a\x1a"
	//
	// 	func myfile() []byte {
	// 		var empty [0]byte
	// 		sx := (*reflect.StringHeader)(unsafe.Pointer(&_myfile))
	// 		b := empty[:]
	// 		bx := (*reflect.SliceHeader)(unsafe.Pointer(&b))
	// 		bx.Data = sx.Data
	// 		bx.Len = len(_myfile)
	// 		bx.Cap = bx.Len
	// 		return b
	// 	}
	NoMemCopy bool

	// NoCompress means the assets are /not/ GZIP compressed before being turned
	// into Go code. The generated function will automatically unzip
	// the file data when called. Defaults to false.
	NoCompress bool

	// Perform a debug build. This generates an asset file, which
	// loads the asset contents directly from disk at their original
	// location, instead of embedding the contents in the code.
	//
	// This is mostly useful if you anticipate that the assets are
	// going to change during your development cycle. You will always
	// want your code to access the latest version of the asset.
	// Only in release mode, will the assets actually be embedded
	// in the code. The default behaviour is Release mode.
	Debug bool

	// Perform a dev build, which is nearly identical to the debug option. The
	// only difference is that instead of absolute file paths in generated code,
	// it expects a variable, `rootDir`, to be set in the generated code's
	// package (the author needs to do this manually), which it then prepends to
	// an asset's name to construct the file path on disk.
	//
	// This is mainly so you can push the generated code file to a shared
	// repository.
	Dev bool

	// When true, size, mode and modtime are not preserved from files
	NoMetadata bool
	// When nonzero, use this as mode for all files.
	Mode uint
	// When nonzero, use this as unix timestamp for all files.
	ModTime int64
	// When nonzero, use this as unix timestamp for all files.
	ChangeTime int64

	// Ignores any filenames matching the regex pattern specified, e.g.
	// path/to/file.ext will ignore only that file, or \\.gitignore
	// will match any .gitignore file.
	//
	// This parameter can be provided multiple times.
	Ignore []*regexp.Regexp

	// Ignores any filenames matching the glob pattern specified, e.g.
	// path/to/*.ext will ignore only that file, or \\.gitignore
	// will match any .gitignore file.
	// See <https://github.com/gobwas/glob> for more glob info.
	//
	// This parameter can be provided multiple times.
	IgnoreGlob []glob.Glob

	// Create File System.
	//
	// This parameter provides `AssetFS` variable.
	FileSystem bool

	// Outlined assets content into binary file. Do not compile it.
	Outlined bool

	OutlinedOutputDir string

	OutlinedLocalOutputDir string

	//
	OutlineEmbeded bool

	// OutlinedApi save asset contents into him. If is blank, use the application binary.
	OutlinedApi string

	// OutlinedNoTruncate truncate existing outlined.
	OutlinedNoTruncate bool

	OutlinedProgram bool

	EmbedPreInitSource string

	OutlinedHeadersOutput string

	NoAutoLoad bool

	Hybrid bool

	NoStore bool

	OulinedSkipApi bool

	FileSystemLoadCallbacks []string
}

Config defines a set of options for the asset conversion.

func NewConfig 

func NewConfig() *Config

NewConfig returns a default configuration struct.

type Finder 

type Finder struct {
	// contains filtered or unexported fields
}

Finder recursively finds all the file paths in the given directory tree. They are added to the given map as keys. Values will be safe function names for each file, which will be used when generating the output code.

type IgnoreGlobSlice 

type IgnoreGlobSlice = ignore.IgnoreGlobSlice

type IgnoreSlice 

type IgnoreSlice = ignore.IgnoreSlice

type InputConfig 

type InputConfig struct {
	// Path defines a directory containing asset files to be included
	// in the generated output.
	Path string

	// Recusive defines whether subdirectories of Path
	// should be recursively included in the conversion.
	Recursive bool

	// Ignores any filenames matching the regex pattern specified, e.g.
	// path/to/file.ext will ignore only that file, or \\.gitignore
	// will match any .gitignore file.
	//
	// This parameter can be provided multiple times.
	Ignore []*regexp.Regexp

	// Ignores any filenames matching the glob pattern specified, e.g.
	// path/to/*.ext will ignore only that file, or \\.gitignore
	// will match any .gitignore file.
	// See <https://github.com/gobwas/glob> for more glob info.
	//
	// This parameter can be provided multiple times.
	IgnoreGlob []glob.Glob

	Prefix string

	NameSpace string

	DirReplacesCount int

	WalkFunc func(visited *map[string]bool, recursive bool, cb func(info walker.FileInfo) error) error
}

InputConfig defines options on a asset directory to be convert.

func (InputConfig) DefaultWalk added in v0.0.6 

func (i InputConfig) DefaultWalk(visited *map[string]bool, recursive bool, cb walker.WalkCallback) (err error)

func (InputConfig) Walk added in v0.0.6 

func (i InputConfig) Walk(visited *map[string]bool, cb walker.WalkCallback) (err error)

type ManyConfig 

type ManyConfig struct {
	Embedded []ManyConfigEmbedded
	Outlined []ManyConfigOutlined
}

func (*ManyConfig) InputsRelTo added in v0.0.3 

func (c *ManyConfig) InputsRelTo(baseDir string)

func (*ManyConfig) Validate 

func (c *ManyConfig) Validate() (err error)

type ManyConfigCommon 

type ManyConfigCommon struct {
	Disabled        bool
	NoAutoLoad      bool `mapstructure:"no_auto_load" yaml:"no_auto_load"`
	NoCompress      bool `mapstructure:"no_compress" yaml:"no_compress"`
	NoMetadata      bool `mapstructure:"no_metadata" yaml:"no_metadata"`
	NoMemCopy       bool `mapstructure:"no_mem_copy" yaml:"no_mem_copy"`
	Mode            uint
	ModTime         int64 `mapstructure:"mod_time" yaml:"mod_time"`
	Ignore          IgnoreSlice
	IgnoreGlob      IgnoreGlobSlice `mapstructure:"ignore_glob" yaml:"ignore_glob"`
	Inputs          ManyConfigInputSlice
	Pkg             string
	Output          string
	Prefix          string
	Hybrid          bool
	Fs              bool
	FsLoadCallbacks []string `mapstructure:"fs_load_callbacks" yaml:"fs_load_callbacks"`
	Default         ManyConfigCommonDefault
}

func (*ManyConfigCommon) Config 

func (a *ManyConfigCommon) Config(ctx context.Context) (c *Config, err error)

func (*ManyConfigCommon) UnmarshalMap 

func (a *ManyConfigCommon) UnmarshalMap(value interface{}) (err error)

func (*ManyConfigCommon) Validate 

func (a *ManyConfigCommon) Validate() (err error)

type ManyConfigCommonDefault added in v0.0.6 

type ManyConfigCommonDefault struct {
	Input ManyConfigCommonDefaultInput
}

func (*ManyConfigCommonDefault) UnmarshalMap added in v0.0.6 

func (d *ManyConfigCommonDefault) UnmarshalMap(value interface{}) (err error)

type ManyConfigCommonDefaultInput added in v0.0.6 

type ManyConfigCommonDefaultInput struct {
	Prefix    string
	NameSpace string `mapstructure:"name_space" yaml:"name_space"`
	Recursive bool
}

func (*ManyConfigCommonDefaultInput) UnmarshalMap added in v0.0.6 

func (i *ManyConfigCommonDefaultInput) UnmarshalMap(value interface{}) (err error)

type ManyConfigEmbedded 

type ManyConfigEmbedded struct {
	ManyConfigCommon
}

func (*ManyConfigEmbedded) UnmarshalMap 

func (a *ManyConfigEmbedded) UnmarshalMap(value interface{}) (err error)

func (*ManyConfigEmbedded) Validate 

func (a *ManyConfigEmbedded) Validate() (err error)

type ManyConfigInput 

type ManyConfigInput struct {
	Path             string
	Prefix           string
	NameSpace        string `mapstructure:"ns" yaml:"ns"`
	Recursive        bool
	Ignore           IgnoreSlice
	DirReplacesCount int
	IgnoreGlob       IgnoreGlobSlice `mapstructure:"ignore_glob" yaml:"ignore_glob"`
	Pkg              string
}

func (ManyConfigInput) Config 

func (i ManyConfigInput) Config(ctx context.Context) (configs []*InputConfig, err error)

func (*ManyConfigInput) UnmarshalMap added in v0.0.6 

func (i *ManyConfigInput) UnmarshalMap(value interface{}) (err error)

func (ManyConfigInput) Walked added in v0.0.6 

func (i ManyConfigInput) Walked(visited *map[string]bool, recursive bool, cb walker.WalkCallback) (err error)

type ManyConfigInputSlice 

type ManyConfigInputSlice []ManyConfigInput

func (ManyConfigInputSlice) Items 

func (s ManyConfigInputSlice) Items(ctx context.Context) (r []InputConfig, err error)

type ManyConfigOutlined 

type ManyConfigOutlined struct {
	ManyConfigCommon
	Api     string
	Program bool
}

func (*ManyConfigOutlined) Config 

func (a *ManyConfigOutlined) Config(ctx context.Context) (c *Config, err error)

func (*ManyConfigOutlined) Translate 

func (a *ManyConfigOutlined) Translate(ctx context.Context) (count int, err error)

func (*ManyConfigOutlined) UnmarshalMap 

func (a *ManyConfigOutlined) UnmarshalMap(value interface{}) (err error)

func (*ManyConfigOutlined) Validate 

func (a *ManyConfigOutlined) Validate() (err error)

type MapUnmarshaler 

type MapUnmarshaler interface {
	UnmarshalMap(value interface{}) (err error)
}

type StringWriter 

type StringWriter struct {
	io.Writer
	// contains filtered or unexported fields
}

func (*StringWriter) Write 

func (w *StringWriter) Write(p []byte) (n int, err error)

Source Files

View all Source files

Directories

Path Synopsis
embed
embed/cmd
mem
mix
mix/assets/embed_go_source/e_multiple_inputs_common_prefix
mix/assets/embed_go_source/e_multiple_inputs_mix_prefix
mix/assets/embed_go_source/e_simple_input
mix/assets/outlineds/embed_executable
Code generated by xbindata.
 Code generated by xbindata.
mix/assets/outlineds/normal
Code generated by xbindata.
 Code generated by xbindata.
mix/callbacks
srv
t
xb
cmd

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.