wio

README

WIO - Whole-In-One to Go

WIO (Whole-In-One to Go) allows you to develop subcommands in Git way still keeping them in a single binary by embedding executables in a binary

Introduction

You know Git, right? Git implements variety of subcommends in a simple and extensible way, in that you can implement your own git foo subcommand only by placing an executable named git-foo in your $PATH. One of the benefit of this way is that you can use any language you like to implement your own Git subcommands. And actually, some of Git commands are implemented in Shellscript or Perl.

On the other hand, one of the beauty of Go is a single binary, which is portable and easy to distribute, deploy and execute.

I dreamed the combination of the flexibility of Git and the simplicity of Go. That's where WIO was born.

Installation

Clone Git repository and run the build script.

For Linux and macOS:

$ git clone github.com/Maki-Daisuke/go-whole-in-one
$ cd go-whole-in-one/cmd/wio
$ ./make.sh install

For Windows:

$ git clone github.com/Maki-Daisuke/go-whole-in-one
$ cd go-whole-in-one\cmd\wio
$ .\make.ps1 install

How to Use

First of all, make your project directory:

$ mkdir mycmd
$ cd mycmd

Then, init your project:

$ wio init

This command creates the following three files:

$ ls
main.go  pack.go  packing-list

Build and run it:

$ go build
$ ./mycmd
usage: mycmd [--version] [--help] <subcommand> <args>

It works! But, nothing interesting happens, yet.

Now, implement your first subcommand in Shellscript:

$ cat <<EOS > mycmd-hello
#!/bin/sh
echo 'Hello, world!'
EOS
$ chmod +x mycmd-hello

Then, build it using wio build, and here, do not forget to add the path to your subcommand, that is the current working directory in this case, into PATH environment variable:

$ PATH=`pwd`:$PATH wio build
$ ./mycmd hello
Hello, world!

You made your own command having a subcommand. Hooray!

You can copy it to another machine:

$ scp ./mycmd another-machine:~
$ ssh another-machine
another-machine> ./mycmd hello
Hello, world!

It works! You don't need to copy mycmd-hello, because it's just a single binary executable.

You can easily add more subcommands. Let's add one more:

$ cat <<EOS > mycmd-jsonpp
#!/bin/sh

jq . $*
EOS
$ chmod +x mycmd-jsonpp

You made it! But wait! This shellscript does not work on machines without jq since it depends on it. Don't worry. You can embed jq by adding it in your packing-list:

$ echo jq >> packing-list
$ cat packing-list
# This file was generated by wio-init.
mycmd-*
jq

Let's build it and use it on the other machine:

$ PATH=`pwd`:$PATH wio build
$ scp ./mycmd another-machine:~
$ ssh another-machine
another-machine> ./mycmd jsonpp
jq: error while loading shared libraries: libonig.so.2: cannot open shared object file: No such file or directory

Oops! It doesn't work... Ok, you need Oniguruma to run jq. Let's add it into the packing-list:

$ echo /usr/lib/x86_64-linux-gnu/libonig.so.2 >> packing-list
$ PATH=`pwd`:$PATH wio build
$ scp ./mycmd another-machine:~
$ ssh another-machine
another-machine> echo '{"my":"cmd","hello":"world"}'  |./mycmd jsonpp
{
  "my": "cmd",
  "hello": "world"
}

It works, hooray!

As you see, you can embed any kind of file you want, and make self-contained binary executables.

More Examples

In fact, wio command itself is a WIO-application.

See its source code for an example.

Command-Line Interface

wio command has three subcommands as follows. The all subcommands are intended to be called in the your project directory, that is, where main package resides.

wio init

Initializes a new WIO application project with boilerplates. As a result, it creates tree files: main.go, pack.go and packing-list.

wio generate

Generates and/or updates pack.go, which all embedded files reside in, in regarding to paking-list.

wio build

Shortcut for wio-generate && go build.

Packing-list

You can embed any file by adding in packing-list. wio generate reads packing-list, compresses listed files, and then embeds them into your application code.

Syntax of packing-list

• Each line shows one item
  • That is, a file to embed
• # begins comment
  • Text from # to end of line is ignored
• Wrapping white characters are ignored
  • Thus, the line foo bar # comment is equivalent to foo bar
• Look up commands by default
  • The line foo means, wio generate looks up a command named foo in your PATH
• If a line contains / or platform dependent path-separator, it is regarded as file path instead of command
  • You need to prepend ./ when you embed a file in the current directory
  • E.g. ./not-command
• You can use file glob like *, ? and [abc]
  • That indicates to embed all matched files
  • Wildcards can be used for commands. That means, to look up all commands matching glob from PATH, and embed all of them.

How it works

Essentially, it has three phases: Generate -> Deploy -> Execute

Generate

Generate phase is processed by wio generate command. It lists up files specified by packing-list and compresses them and generate Go source code which contains compressed data embeded. You need to run wio generate in your project directory when files you want to embed are changed.

You can run go generate instead of wio generate. It essentially does the same thing, but go generate && go build is more common idiom and handy when you use other generate-compatible packages.

Deploy

This phase is kicked when a WIO application is executed first time. It decompresses and unpack embedded files into a cache directory under system's temporary directory (which is usually /tmp on Linux).

This process is executed only once per user. Each user has her/his own cache directory, which can be read only by the user who executed the command. In other words, users who cannot run the command cannot read/write files embedded in the command. That keeps your confidentaial data secure.

Execute

Finally, this phase executes your command. But, before executing the command, it prepares execution environment. More specific, it sets the following three environment variables:

• WIOPATH
  • This environment variable holds path to cache firectory in which WIO unpacks embedded files.
  • This allows subcommands to access embedded files during execution.
• PATH
  • WIO prepends path to cache directory at the head of PATH environement variable.
  • In other words, when you invoke a command in subcommands, it will search command executable in the cache directory at first.
• LD_LIBRARY_PATH / DYLD_LIBRARY_PATH
  • This allows subcommands dynamically link shared libraries embedded.
  • In cotrast to PATH, WIO appends cache directory at the tail of LD_LIBRARY_PATH. That means, libraries you specify take a priority.
  • On macOS (Dawrin), DYLD_LIBRARY_PATH is set instead of LD_LIBRARY_PATH.

After setting those environment variables, it looks up approapriate subcommand and executes it.

CAVEAT

Despite WIO sets DYLD_LIBRARY_PATH on macOS (Darwin), it does not work as expected in most environments, unfortunately. Because of security concerns, DYLD_LIBRARY_PATH is ignored by SIP-protected binaries including /bin/sh. And, SIP (System Integrity Protection) is enebled by default on El Capitan and newer. Thus, the example above to use jq does not work on macOS.

However, you can explicitly set DYLD_LIBRARY_PATH by yourself like this:

#!/bin/sh

export DYLD_LIBRARY_PATH=$WIOPATH
jq . $*

This works as you expect.

Built-in Subcommands

Sometimes, you want to implement subcommands in Go and integrate them in your main command, because it's less overhead. You can do it by implementing wio.Command interface and registering it as built-in subcommand with wio.Register.

For your convenience, you can use FuncCommand type. For example:

func main(){
  wio.Register("mycmd", wio.FuncCommand(func(subname string, args []string){
    fmt.Println("This is my command!")
  }))
  wio.Exec(os.Args[1:])
}

See API reference for details.

WIO registers the following predefined built-in subcommands by default:

• help, --help, -h
  • Shows simple help message
• version, --version, -v
  • Shows version number

Yes, --help and --version look like options, but they are actually implemented as subcommands.

Of course, you can customize your subcommands as you like by editing your main.go.

API

See Godoc.

License

The Simplified BSD License (2-clause). See LICENSE file also.

Author

Daisuke (yet another) Maki

Documentation

Overview

Package wio provides API to customize your command-line application written using WIO (Whole-In-One to Go).

You don't need to care most of them, because wio command cares on befalf of you.

See https://github.com/Maki-Daisuke/go-whole-in-one for more information.

Index

• Variables
• func CompressWriter(w io.Writer, codec string) io.Writer
• func Exec(args []string)
• func IsExecutable(path string) (ok bool, err error)
• func ListSubcommands() (cmds []string, err error)
• func LookupExecutables(pattern string) (paths []string, err error)
• func Register(name string, cmd Command)
• func SetEnv(path string)
• func Unpack(dest string, data io.Reader, codec string)
• func WriteMainGo(name, version string) error
• func WritePackGo(data []byte, codec string) error
• func WritePackingList(name string) error
• type Command
• type FuncCommand
  • func (f FuncCommand) Exec(subname string, args []string)

Variables

var HelpCommand = FuncCommand(func(_ string, _ []string) {
	cmds, err := ListSubcommands()
	if err != nil {
		panic(err)
	}
	fmt.Printf("usage: %s SUBCOMMAND [ARGS...]\n\n", Name)
	fmt.Println("Available subcommands:")
	for _, c := range cmds {
		fmt.Printf("  %s\n", c)
	}
})

HelpCommand is a predefined built-in subcommand, which shows the default help message.

var Name = ""

Name holds the name of root command. This value is used for vriety of purposes. For example, it is used to make messages output by built-in help and version subcommand, to look up executables for subcommands, and also to determine the name of cache directory.

You can overwrite the value to customize the behavior of WIO. For example:

func init(){
    wio.Name = "newname"
}

Note that you must set Name in init() function, since wio has alreay created cache directory and done preparation stuffs before main() function is called.

var Version = "0"

Version holds version string of your command. Its default value is `"0"`. This is used in built-in `version` subcommand and also used to determine the name of cache directory.

As well as Name variable, you must set Version in init() function, before main() is called.

var VersionCommand = FuncCommand(func(_ string, _ []string) {
	fmt.Printf("%s version %s\n", Name, Version)
})

VersionCommand is a predefined built-in subcommand, which shows the name and the version number of the command.

Functions

func CompressWriter

func CompressWriter(w io.Writer, codec string) io.Writer

CompressWriter compresses w using codec.

func Exec

func Exec(args []string)

Exec searches a command implementation corresponding to the name of subcommand and executes it. Because it may call exec systemcall, lines following Exec will never be executed:

func main(){
    wio.Exec(os.Args[1:])  // This may call exec systemcall inside,
    fmt.Println("???")     // thus, execution never reaches here.
}

This rule is also the case for built-in commands. Exec calls os.Exit(0) when a built-in command is successfully finished. If you want to return status code from your built-in command, you need to call os.Exit with non-zero integer manually.

func IsExecutable

func IsExecutable(path string) (ok bool, err error)

IsExecutable reports whether the file indicated by path is an executable in a platform-dependent manner.

func ListSubcommands

func ListSubcommands() (cmds []string, err error)

ListSubcommands returns names of all available subcommands in sorted order. It is handy to implement your customized help message.

func LookupExecutables

func LookupExecutables(pattern string) (paths []string, err error)

LookupExecutables searches all executables matching pattern in your PATH environment variable, and returns absolute file paths of the executables. pattern is interpreted as file glob. See path/filepath#Glob for details of file glob syntax.

func Register

func Register(name string, cmd Command)

Register binds cmd to name as a built-in subcommand. For example, if you register as follows:

wio.Register("foo", wio.FuncCommand(func(argv0 string, argv []string){
    fmt.Printf("You called '%s' subcommand with: %v", argv0, argv)
}))

Then, you can invoke this function in your command line like this:

$ yourcmd foo bar baz
You called 'foo' subcommand with: [bar baz]

func SetEnv

func SetEnv(path string)

SetEnv is called in by pack.go during preparation of environment. You should not call this.

func Unpack

func Unpack(dest string, data io.Reader, codec string)

Unpack is called in Deployment phase by pack.go. You should not call this.

func WriteMainGo

func WriteMainGo(name, version string) error

WriteMainGo is called by wio command to generate main.go. You don't need to use this unless you want to implement your own generator.

func WritePackGo

func WritePackGo(data []byte, codec string) error

WritePackGo is called by wio command to generate pack.go. You don't need to use this unless you want to implement your own generator.

func WritePackingList

func WritePackingList(name string) error

WritePackingList is called by wio command to generate pack.go. You don't need to use this unless you want to implement your own generator.

Types

type Command

type Command interface {
	Exec(subname string, args []string)
}

The Command type represents a built-in subcommand, which can be registered with Register function.

type FuncCommand

type FuncCommand func(subname string, args []string)

The FuncCommand type is an adapter to allow the use of ordinary functions as built-in subcommand. If f is a function with the appropriate signature, FuncCommand(f) is a Command that calls f.

func (FuncCommand) Exec

func (f FuncCommand) Exec(subname string, args []string)

Exec calls f(subname, args).