goblin

package module
v0.0.0-...-ce82533 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 6, 2020 License: MIT Imports: 16 Imported by: 0

README

GoDoc Build Status Coverage Maintainability Rating Go Report Card

Goblin

Goblin is a library and utility for interacting with filesystems in Go, including filesystems embedded in Go binaries. It's designed around the proposed io/fs interfaces. At some point it will likely also support the proposed embedded static assets support.

Warning: Since the interfaces this library is based on are not stable, neither is this API! It will evolve with the proposal and thus should be considered unstable. The goal is to avoid a lot of API churn but sometimes it's inevitable!

Installation

If you'd only like to use Goblin as a library you can install it with:

$ go get github.com/aphistic/goblin

If you'd like to make use of Goblin's capability to embed files in a Go binary, you'll want to install the goblin utility to create those files.

go get github.com/aphistic/goblin/cmd/goblin

General Usage

The representation of a filesystem in Goblin is known as a vault (get it? goblins? vaults? 😁). The Vault interface in Goblin implements at least a few of the io/fs interfaces and adds a few additional utility methods (some hopefully having io/fs interfaces in the future).

// Create a new memory vault
mVault := goblin.NewMemoryVault()

// Create a reader to write the file from with
// some file contents.
fileData := bytes.NewReader([]byte("I'm a file!"))

// Write the file contents to your/file/here.txt with
// a file modified time of April 8, 2020 at Midnight UTC.
_ = mVault.WriteFile(
    "your/file/here.txt", fileData,
    goblin.FileModTime(time.Date(2020, 4, 8, 0, 0, 0, 0, time.UTC)),
)

// Read all the files in the root of the memory vault. In
// our case it's only the "your" directory.
infos, _ := mVault.ReadDir(".")
fmt.Printf("Root Files:\n")
for _, info := range infos {
    fmt.Printf("  - %s\n", info.Name())
}

// Read the contents of the file we previously wrote.
readData, _ := mVault.ReadFile("your/file/here.txt")
fmt.Printf("File Data: %s\n", readData)

// Output:
// Root Files:
//   - your
// File Data: I'm a file!

Mixing Vaults at Runtime

It's sometimes desired to be able to choose between one or more vaults at runtime. Since this is a common pattern, Goblin provides a way to do it for you with a VaultSelector. The VaultSelector is a Vault, so it can be used anywhere a Vault would be, and provides a few options for selecting a Vault by default but also supports creating custom selectors.

// Use a FilesystemVault if a path is provided as an environment variable,
// but fall back to using a MemoryVault by default. This, for example,
// could be useful during development when you are iterating on embedded
// files and don't want to have to rebuild for every change.
const assetVaultPathKey = "ASSET_VAULT_PATH"

_ = os.Setenv(assetVaultPathKey, "/")

// Create a filesystem vault and a memory vault to use.
fsysAssetVault := goblin.NewFilesystemVault(os.Getenv(assetVaultPathKey))
memAssetVault := goblin.NewMemoryVault() // Loaded from an embedded vault

// Create a vault selector that will use the filesystem asset vault
// if the environment variable ASSET_VAULT_PATH has a non-empty vault,
// use the embedded asset vault if not.
assetVaultSelector := goblin.NewVaultSelector(
    goblin.SelectEnvNotEmpty(assetVaultPathKey, fsysAssetVault),
    goblin.SelectDefault(memAssetVault),
)

// Since ASSET_VAULT_PATH is not empty, we use the filesystem vault.
v, _ := assetVaultSelector.GetVault()
fmt.Printf("Vault: %s\n", v)

// Output:
// Vault: Filesystem Vault (/)

Embedding Files

To embed files in your binary using Goblin, you'll use the goblin utility to generate a Go code file with the contents of the file and a method to load the contents at runtime.

Note: The public interface for the goblin utility will be changing and is not stabilized. It'll be changing to hopefully be cleaner and more intuitive.

There are two common command line arguments that are used when create an embedded vault:

  • --name or -n: The name of the vault to embed. This will be used for any package, vault, and file names in the generated code.
  • --include-root or -r: The root path of any included files. Any file paths in the vault will be relative to this path. Defaults to the current working directory.
  • --include or -i: A glob path to include files for. Can be provided more than once.

To include all .html files in your project's web directory or a directory below it in a vault, for example, you would use:

$ goblin create --name assets --include-root /src/web --include *.html --include **/*.html

This would result in a file called goblin_assets.go in the current directory to be created with contents similar to the following:

package assets

import goblin "github.com/aphistic/goblin"

func loadVaultAssets() (goblin.Vault, error) {
	return goblin.LoadMemoryVault(goblinMemoryVaultXassets)
}

var goblinMemoryVaultXassets = []byte{ /* lots of bytes */ }

If you need to specify a package name other than the default (assets in our example), you can use the --package or -p command line option to provide a different one.

What's With the Name?

When I was trying to come up with a project name for a utility to embed binary files in Go binaries one of the names I was working with included gobin for Go Binaries and when I read it I misread it as goblin at one point and thought that would be a fun name since goblins love treasure (your files!) and keeping things in vaults.

Documentation

Overview

Package goblin provides interaction with various types of filesystems, including those embedded in the binary.

It's designed to implement the proposed Go standard library filesystem interfaces. Since these interfaces do not exist in a stable release yet, they've been included in this package for now. Eventually those interfaces will be removed in favor of the stable release. Types that are included temporarily are noted in their documentation.

Index

Examples

Constants

This section is empty.

Variables

This section is empty.

Functions

func ReadDir 

func ReadDir(fsys FS, name string) ([]os.FileInfo, error)

ReadDir mirrors the proposed io/fs.ReadDir

https://go.googlesource.com/go/+/2ad964dc3731dac3ab561ab344042dbe316dbf28/src/io/fs/readdir.go#28

func Stat 

func Stat(fsys FS, name string) (os.FileInfo, error)

Stat mirrors the proposed io/fs.Stat

https://go.googlesource.com/go/+/2ad964dc3731dac3ab561ab344042dbe316dbf28/src/io/fs/stat.go#20

func Walk 

func Walk(fsys FS, root string, walkFn filepath.WalkFunc) error

Walk mirrors the proposed io/fs.Walk

https://go.googlesource.com/go/+/2ad964dc3731dac3ab561ab344042dbe316dbf28/src/io/fs/walk.go#70

Types

type FS 

type FS interface {
	Open(name string) (File, error)
}

FS mirrors the proposed io/fs.FS

https://go.googlesource.com/go/+/2ad964dc3731dac3ab561ab344042dbe316dbf28/src/io/fs/fs.go#21

type File 

type File interface {
	Stat() (os.FileInfo, error)
	Read(buf []byte) (int, error)
	Close() error
}

File mirrors the proposed io/fs.File

https://go.googlesource.com/go/+/2ad964dc3731dac3ab561ab344042dbe316dbf28/src/io/fs/fs.go#73

type FileOption 

type FileOption func(*fileOptions)

FileOption is a common set of options used when creating or managing files.

func FileModTime 

func FileModTime(modTime time.Time) FileOption

FileModTime specifies the modified time to use for the file.

type FilesystemVault 

type FilesystemVault struct {
	// contains filtered or unexported fields
}

FilesystemVault is a vault used to interact with a local filesystem. All paths provided to a FilesystemVault are relative to the root path.

Example 
package main

import (
	"fmt"

	"github.com/aphistic/goblin"
)

func main() {
	fsysVault := goblin.NewFilesystemVault("/")
	infos, _ := fsysVault.ReadDir(".")

	fmt.Printf("Files in /\n")
	for _, info := range infos {
		fmt.Printf("  - %s\n", info.Name())
	}
}

Output:

func NewFilesystemVault 

func NewFilesystemVault(rootPath string) *FilesystemVault

NewFilesystemVault creates a FilesystemVault using the given root path as the root of the filesystem.

func (*FilesystemVault) Glob 

func (v *FilesystemVault) Glob(pattern string) ([]string, error)

Glob returns names of files in the filesystem that match the given pattern relative to the vault's root path.

func (*FilesystemVault) Open 

func (v *FilesystemVault) Open(name string) (File, error)

Open returns a file at the given path relative to the vault's root path.

func (*FilesystemVault) ReadDir 

func (v *FilesystemVault) ReadDir(dirName string) ([]os.FileInfo, error)

ReadDir returns directory contents for the given path relative to the vault's root path.

func (*FilesystemVault) ReadFile 

func (v *FilesystemVault) ReadFile(name string) ([]byte, error)

ReadFile returns the contents of the file at the given path relative to the vault's root path.

func (*FilesystemVault) Stat 

func (v *FilesystemVault) Stat(name string) (os.FileInfo, error)

Stat returns file info at the given path relative to the vault's root path.

func (*FilesystemVault) String 

func (v *FilesystemVault) String() string

type GlobFS 

type GlobFS interface {
	FS
	Glob(pattern string) ([]string, error)
}

GlobFS mirrors the proposed io/fs.GlobFS

https://go.googlesource.com/go/+/2ad964dc3731dac3ab561ab344042dbe316dbf28/src/io/fs/glob.go#13

type GlobVault 

type GlobVault interface {
	Vault
	GlobFS
}

GlobVault is an interface that provides a Vault in addition to a GlobFS.

type LoadMemoryOption 

type LoadMemoryOption func(*loadMemoryOptions)

LoadMemoryOption is an option used when loading an in-memory vault.

type MemoryBuilder 

type MemoryBuilder struct {
	// contains filtered or unexported fields
}

MemoryBuilder creates binary or code representations of a memory vault.

func NewMemoryBuilder 

func NewMemoryBuilder(opts ...MemoryBuilderOption) *MemoryBuilder

NewMemoryBuilder creates a new memory builder.

func (*MemoryBuilder) Include 

func (b *MemoryBuilder) Include(rootPath string, globs []string) error

Include iterates over all files in the root path, then includes any file matching one or more of the provided globs in the memory vault being built.

func (*MemoryBuilder) WriteBinary 

func (b *MemoryBuilder) WriteBinary(w io.Writer) error

WriteBinary writes the binary representation of the memory vault to the provided io.Writer.

func (*MemoryBuilder) WriteLoader 

func (b *MemoryBuilder) WriteLoader(packageName string, vaultName string, w io.Writer) error

WriteLoader writes code and binary data to the provided io.Writer to allow loading the memory vault being built at runtime.

type MemoryBuilderOption 

type MemoryBuilderOption func(b *MemoryBuilder)

MemoryBuilderOption is an option used when creating a memory vault builder

func MemoryBuilderExportLoader 

func MemoryBuilderExportLoader(exportLoader bool) MemoryBuilderOption

MemoryBuilderExportLoader will cause the code for loading the vault to be exported. For example, LoadVaultAssets instead of loadVaultAssets.

func MemoryBuilderLogger 

func MemoryBuilderLogger(logger logging.Logger) MemoryBuilderOption

MemoryBuilderLogger provides a logger for the builder to use.

type MemoryVault 

type MemoryVault struct {
	// contains filtered or unexported fields
}

MemoryVault is a vault stored in memory. It can be used as a temporary in-memory filesystem or as a way to load a filesystem from binary data, such as one embedded in a file.

Example 
package main

import (
	"bytes"
	"fmt"
	"time"

	"github.com/aphistic/goblin"
)

func main() {
	// Create a new memory vault
	mVault := goblin.NewMemoryVault()

	// Create a reader to write the file from with
	// some file contents.
	fileData := bytes.NewReader([]byte("I'm a file!"))

	// Write the file contents to your/file/here.txt with
	// a file modified time of April 8, 2020 at Midnight UTC.
	_ = mVault.WriteFile(
		"your/file/here.txt", fileData,
		goblin.FileModTime(time.Date(2020, 4, 8, 0, 0, 0, 0, time.UTC)),
	)

	// Read all the files in the root of the memory vault. In
	// our case it's only the "your" directory.
	infos, _ := mVault.ReadDir(".")
	fmt.Printf("Root Files:\n")
	for _, info := range infos {
		fmt.Printf("  - %s\n", info.Name())
	}

	// Read the contents of the file we previously wrote.
	readData, _ := mVault.ReadFile("your/file/here.txt")
	fmt.Printf("File Data: %s\n", readData)

}

Output:

Root Files:
  - your
File Data: I'm a file!

func NewMemoryVault 

func NewMemoryVault(opts ...MemoryVaultOption) *MemoryVault

NewMemoryVault creates a new memory vault.

func (*MemoryVault) Glob 

func (v *MemoryVault) Glob(pattern string) ([]string, error)

Glob returns names of files in the in-memory vault that match the given pattern.

func (*MemoryVault) MarshalBinary 

func (v *MemoryVault) MarshalBinary() ([]byte, error)

MarshalBinary encodes the MemoryVault into a binary representation.

func (*MemoryVault) Open 

func (v *MemoryVault) Open(name string) (File, error)

Open will open the file at the provided path from the in-memory vault.

func (*MemoryVault) ReadDir 

func (v *MemoryVault) ReadDir(dirName string) ([]os.FileInfo, error)

ReadDir returns a slice of file info for the provided directory in the in-memory vault.

func (*MemoryVault) ReadFile 

func (v *MemoryVault) ReadFile(name string) ([]byte, error)

ReadFile returns the contents of the file at the given path from the in-memory vault.

func (*MemoryVault) Stat 

func (v *MemoryVault) Stat(name string) (os.FileInfo, error)

Stat returns file info for the provided path in the in-memory vault.

func (*MemoryVault) String 

func (v *MemoryVault) String() string

func (*MemoryVault) UnmarshalBinary 

func (v *MemoryVault) UnmarshalBinary(data []byte) error

UnmarshalBinary decodes the provided data into the MemoryVault.

func (*MemoryVault) WriteFile 

func (v *MemoryVault) WriteFile(path string, r io.Reader, opts ...FileOption) error

WriteFile reads data from the provided io.Reader and then writes it to the memory vault at the provided path.

type MemoryVaultOption 

type MemoryVaultOption func(*memoryVaultOptions)

MemoryVaultOption is an option used when creating a memory vault.

type ReadDirFS 

type ReadDirFS interface {
	FS
	ReadDir(name string) ([]os.FileInfo, error)
}

ReadDirFS mirrors the proposed io/fs.ReadDirFS

https://go.googlesource.com/go/+/2ad964dc3731dac3ab561ab344042dbe316dbf28/src/io/fs/readdir.go#14

type ReadDirFile 

type ReadDirFile interface {
	File
	ReadDir(n int) ([]os.FileInfo, error)
}

ReadDirFile mirrors the proposed io/fs.ReadDirFile

https://go.googlesource.com/go/+/2ad964dc3731dac3ab561ab344042dbe316dbf28/src/io/fs/fs.go#83

type ReadFileFS 

type ReadFileFS interface {
	FS
	ReadFile(name string) ([]byte, error)
}

ReadFileFS mirrors the proposed io/fs.ReadFileFS

https://go.googlesource.com/go/+/2ad964dc3731dac3ab561ab344042dbe316dbf28/src/io/fs/readfile.go#11

type SelectOption 

type SelectOption func(*VaultSelector)

SelectOption is an option used when creating a vault selector.

func SelectDefault 

func SelectDefault(v Vault) SelectOption

SelectDefault will always return the provided vault.

Example 
package main

import (
	"fmt"

	"github.com/aphistic/goblin"
)

func main() {
	fsVault := goblin.NewFilesystemVault("/")
	memVault := goblin.NewMemoryVault()

	// The first vault will always be selected because it
	// will always return a vault value.
	vs := goblin.NewVaultSelector(
		goblin.SelectDefault(fsVault),
		goblin.SelectDefault(memVault),
	)

	fmt.Printf("%s\n", vs)
}

Output:

Vault Selector (Filesystem Vault (/))

func SelectEnvBool 

func SelectEnvBool(envName string, v Vault) SelectOption

SelectEnvBool will use the provided vault if the given environment variable is true. Accepted truthy values are TRUE, T, YES, Y, and 1. Values are case-insensitive.

Example 
package main

import (
	"fmt"
	"os"

	"github.com/aphistic/goblin"
)

func main() {
	const envKey = "USE_SELECT_ENV_BOOL"
	os.Setenv(envKey, "true")

	memVault := goblin.NewMemoryVault()
	fsVault := goblin.NewFilesystemVault("/")

	vs := goblin.NewVaultSelector(
		goblin.SelectEnvBool(envKey, memVault),
		goblin.SelectDefault(fsVault),
	)

	fmt.Printf("%s\n", vs)
}

Output:

Vault Selector (Memory Vault)

func SelectEnvNotEmpty 

func SelectEnvNotEmpty(envName string, v Vault) SelectOption

SelectEnvNotEmpty will use the provided vault if the given environment variable is set with a non-empty value.

Example 
package main

import (
	"fmt"
	"os"

	"github.com/aphistic/goblin"
)

func main() {
	const envKey = "USE_SELECT_ENV_NON_EMPTY"
	os.Setenv(envKey, "any value here")

	memVault := goblin.NewMemoryVault()
	fsVault := goblin.NewFilesystemVault("/")

	vs := goblin.NewVaultSelector(
		goblin.SelectEnvNotEmpty(envKey, memVault),
		goblin.SelectDefault(fsVault),
	)

	fmt.Printf("%s\n", vs)
}

Output:

Vault Selector (Memory Vault)

func SelectPath 

func SelectPath(path string, v Vault) SelectOption

SelectPath will use the provided vault if the given path exists on disk.

Example 
package main

import (
	"fmt"

	"github.com/aphistic/goblin"
)

func main() {
	const vaultPath = "/usr"

	rootVault := goblin.NewFilesystemVault("/")
	usrVault := goblin.NewFilesystemVault(vaultPath)

	vs := goblin.NewVaultSelector(
		goblin.SelectPath(vaultPath, usrVault),
		goblin.SelectDefault(rootVault),
	)

	fmt.Printf("%s\n", vs)
}

Output:

Vault Selector (Filesystem Vault (/usr))

type SelectVaultMethod 

type SelectVaultMethod func() Vault

SelectVaultMethod is the method signature a vault selector option needs to implement.

type StatFS 

type StatFS interface {
	FS
	Stat(name string) (os.FileInfo, error)
}

StatFS mirrors the proposed io/fs.StatFS

https://go.googlesource.com/go/+/2ad964dc3731dac3ab561ab344042dbe316dbf28/src/io/fs/stat.go#8

type Vault 

type Vault interface {
	StatFS
	ReadDirFS
	ReadFileFS

	fmt.Stringer
}

Vault is the interface that provides all the interfaces a Goblin vault must implement.

func LoadMemoryVault 

func LoadMemoryVault(vaultData []byte, opts ...LoadMemoryOption) (Vault, error)

LoadMemoryVault takes a binary representation of a memory vault and unmarshales it into a vault.

type VaultSelector 

type VaultSelector struct {
	// contains filtered or unexported fields
}

VaultSelector is a vault that will use a vault selected by one of the added selectors.

Example 
package main

import (
	"fmt"
	"os"

	"github.com/aphistic/goblin"
)

func main() {
	// Use a FilesystemVault if a path is provided as an environment variable,
	// but fall back to using a MemoryVault by default. This, for example,
	// could be useful during development when you are iterating on embedded
	// files and don't want to have to rebuild for every change.
	const assetVaultPathKey = "ASSET_VAULT_PATH"

	_ = os.Setenv(assetVaultPathKey, "/")

	// Create a filesystem vault and a memory vault to use.
	fsysAssetVault := goblin.NewFilesystemVault(os.Getenv(assetVaultPathKey))
	memAssetVault := goblin.NewMemoryVault() // Loaded from an embedded vault

	// Create a vault selector that will use the filesystem asset vault
	// if the environment variable ASSET_VAULT_PATH has a non-empty vault,
	// use the embedded asset vault if not.
	assetVaultSelector := goblin.NewVaultSelector(
		goblin.SelectEnvNotEmpty(assetVaultPathKey, fsysAssetVault),
		goblin.SelectDefault(memAssetVault),
	)

	// Since ASSET_VAULT_PATH is not empty, we use the filesystem vault.
	v, _ := assetVaultSelector.GetVault()
	fmt.Printf("Vault: %s\n", v)

}

Output:

Vault: Filesystem Vault (/)

func NewVaultSelector 

func NewVaultSelector(opts ...SelectOption) *VaultSelector

NewVaultSelector creates a new vault selector.

func (*VaultSelector) AppendSelector 

func (vs *VaultSelector) AppendSelector(svm SelectVaultMethod)

AppendSelector adds an additional vault selector to the end of the vault selector list.

func (*VaultSelector) GetVault 

func (vs *VaultSelector) GetVault() (Vault, error)

GetVault returns the vault to be used by the vault selector using the currently provided selectors.

func (*VaultSelector) Glob 

func (vs *VaultSelector) Glob(pattern string) ([]string, error)

Glob returns names of files in the selected vault that match the given pattern.

func (*VaultSelector) Open 

func (vs *VaultSelector) Open(name string) (File, error)

Open will open the file at the provided path from the selected vault.

func (*VaultSelector) ReadDir 

func (vs *VaultSelector) ReadDir(dirName string) ([]os.FileInfo, error)

ReadDir returns a slice of file info for the provided directory from the selected vault.

func (*VaultSelector) ReadFile 

func (vs *VaultSelector) ReadFile(name string) ([]byte, error)

ReadFile returns the contents of the file at the given path from the selected vault.

func (*VaultSelector) Stat 

func (vs *VaultSelector) Stat(name string) (os.FileInfo, error)

Stat returns file info for the provided path from the selected vault.

func (*VaultSelector) String 

func (vs *VaultSelector) String() string

Source Files

View all Source files

Directories

Path Synopsis
cmd
goblin
Package main is the utility used to create and manage goblin vaults.
 Package main is the utility used to create and manage goblin vaults.
internal
logging
Package logging is the logger the Goblin binary uses internally.
 Package logging is the logger the Goblin binary uses internally.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.