ephemerald

README

ephemerald

Ephemerald manages pools of short-lived servers to be used for testing purposes. It was built to allow paralallel integration tests.

It has REST API for accessing server instances from any language and comes with a built-in go client. See the examples directory for example configurations and client usage.

The ephemerald server can run on a remote host; container connection parameters are rewritten so that the client connects to the right place. This way ephemerald can run on a large server and be used from less powerful development machines.

• Running
• Configuration
  • Params
  • Container
  • Lifecycle Actions
    • noop
    • exec
    • http.get
    • tcp.connect
    • postgres.exec
    • postgres.ping
    • postgres.truncate
    • redis.exec
    • redis.ping
    • redis.truncate
• API
  • Checkout
  • Return
  • Batch Checkout
  • Batch Return
• Building
• Installing
  • Homebrew
  • Binary
  • Source
• TODO

Running

To run the server, supply a configuration file:

$ ephemerald -c config.yaml

Press Q to quit the server.

Flags

• --help print help message.
• -p <port> changes the listen port. Defaults to 6000
• --ui stream will dump the event steam to the console in lieu of a curses-esque UI.
• --ui none will not print any UI information (useful with --log-file /dev/stdout)
• --log-file <path> write logs to file at path. Defaults to /dev/null
• --log-level <level> log level. defaults to info. Options are debug,info,warn,error

Note: use Ctrl-C to stop the server wen not in --ui tui mode (SIGINT,SIGQUIT always work too)

For example, to see only log messages (at debug level) use:

$ ephememerald --ui none --log-level debug --log-file /dev/stdout -c config.yaml

Configuration

Container pools are configured in a yaml (or json) file. Each pool has options for the container parameters and for lifecycle actions.

The following configuration creates a single pool called "pg" which maintains five containers from the "postgres" image and exposes port 5432 to clients. See the params and actions below for documentation on those fields.

pools:
  pg:
    image: postgres
    size: 5
    port: 5432
    params:
      username: postgres
      database: postgres
      url: postgres://{{.Username}}:@{{.Hostname}}:{{.Port}}/{{.Database}}?sslmode=disable
    actions:
      healthcheck:
        type: postgres.ping
        retries: 10
        delay:   50ms
      initialize:
        type:    exec
        path:    make
        args:    [ 'db:migrate' ]
        env:     [ 'DATABASE_URL={{.Url}}' ]
        timeout: 10s
      reset:
        type: postgres.truncate

See example/config.yaml for a full working configuration.

Params

The params entry allows for declaring parameters needed for connecting to the service. There are three fields with arbitrary values: username, password, database.

The url can be a golang template and will be executed with access to the following fields:

Name	Value
Hostname	The hostname that the container can be connected at
Port	The (automatically-generated) port number that is mapped to the exposed container port
Username	The username field declared in params
Password	The password field declared in params
Database	The database field declared in params

A params section for postgres may look like this:

username: postgres
database: postgres
url: postgres://{{.Username}}:{{.Password}}@{{.Hostname}}:{{.Port}}/{{.Database}}?sslmode=disable

Container

The container section is passed-through to docker when creating the container. The available options are:

• labels
• env
• cmd
• volumes
• entrypoint
• user
• capadd
• capdrop

Lifecycle Actions

There are three lifecycle actions: healthcheck, initialize, and reset.

• healthcheck is used to determine when the container is ready to be used.
• initialize is used to initialize the container (run migrations, etc...)
• reset may be used to revert the container to a state where it can be used again.

All of them are optional (though healthcheck should be used). If reset is not given, the container will be killed and a new one will be created to replace it.

Each action has, at a minimum, the following three parameters:

Name	Default	Description
retries	3	number of times to retry the action
timeout	1s	amount of time to allow for the action
delay	500ms	amount of time to delay before retrying

timeout and delay are durations; they must have a unit suffix as described here.

Note: actions may have different defaults for these fields.

noop

Does nothing. Useful as the reset action so that a container is always reused.

exec

Execute a command on the host operating system. Useful for running migrations to initialize a database.

Extra Parameters:

Name	Default	Description
command	""	command to execute
args	[]	command-line arguments
env	[]	environment variables
dir	""	directory to execute in.

The env entries may be templates with access to the same fields as the params url template. Additionally, the following environment variables are always set:

• EPHEMERALD_ID
• EPHEMERALD_HOSTNAME
• EPHEMERALD_PORT
• EPHEMERALD_USERNAME
• EPHEMERALD_PASSWORD
• EPHEMERALD_DATABASE
• EPHEMERALD_URL

If dir is not set, the working directory of the server isused.

http.get

Run a HTTP GET request.

Extra Parameters:

Name	Default	Description
url	""	url to request

If url is blank, the url from the params is used.

If url is not blank, it may be a template which has access to the same fields that params url template does.

tcp.connect

Connect to the exposed container port over TCP.

postgres.exec

Executes a query on the database.

Extra Parameters:

Name	Default	Description
command	"SELECT 1=1"	query to execute
args	[]	values to be escaped with positional arguments in command.

Example:

type:     postgres.exec
command: 'INSERT INTO users (name) VALUES ($1)'
args:    "Robert'); DROP TABLE STUDENTS;--"

postgres.ping

Pings the database. Useful for healthcheck.

postgres.truncate

Runs TRUNCATE TABLE x CASCADE for all tables x.

Extra Parameters:

Name	Default	Description
exclude	[]	an array of table names to not truncate (eg migration versions)

redis.exec

Execute a redis command.

Extra Parameters:

Name	Default	Description
command	"PING"	redis command to execute

redis.ping

This is an alias for redis.exec.

redis.truncate

This is an alias for redis.exec with a default command of "FLUSHALL".

API

There is a REST API for clients to checkout and return items from one or more pools.

Checkout

POST /checkout/{pool} checks out an instance from the given pool and returns that instance's parameters:

$ curl -s -XPOST localhost:6000/checkout/postgres | jq
{
  "id":"8482c266192f013346d03f71b2aa6d4b647909e3502ac525039bdd0fe9fcac30",
  "hostname":"localhost",
  "port":"34031",
  "username":"postgres",
  "database":"postgres",
  "url":"postgres://postgres:@localhost:34031/postgres?sslmode=disable"
}

Return

DELETE /return/{pool}/{id} returns the instance given by id to the pool pool:

$ curl -s -XDELETE localhost:6000/return/postgres/8482c266192f013346d03f71b2aa6d4b647909e3502ac525039bdd0fe9fcac30

Batch Checkout

POST /checkout checks out an instance from every configured pool.

$ curl -s -XPOST localhost:6000/checkout | tee checkout.json | jq
{
  "postgres": {
    "id": "2dedf5dbe9cc8d7a0cd71ed75455c7310db79aea44925562b82c01b959d85e7e",
    "hostname": "localhost",
    "port": "34023",
    "username": "postgres",
    "database": "postgres",
    "url": "postgres://postgres:@localhost:34023/postgres?sslmode=disable"
  },
  "redis": {
    "id": "a8dbf5043c7145510f48ccffa6f1e20b9f2c8140dda73d567a29dc2ec823ca46",
    "hostname": "localhost",
    "port": "34019",
    "database": "0",
    "url": "redis://localhost:34019/0"
  },
  "vault": {
    "id": "11f4752d5e0b762c65b05809c9500a6e0a20ee4a79b861638a084adf77dbfb78",
    "hostname": "localhost",
    "port": "34021",
    "url": "http://localhost:34021"
  }
}

Batch Return

DELETE /return returns multiple instances at once. Meant to be used with batch checkout.

$ cat checkout.json
{
  "postgres": {
    "id": "2dedf5dbe9cc8d7a0cd71ed75455c7310db79aea44925562b82c01b959d85e7e"
  },
  "redis": {
    "id": "a8dbf5043c7145510f48ccffa6f1e20b9f2c8140dda73d567a29dc2ec823ca46"
  },
  "vault": {
    "id": "11f4752d5e0b762c65b05809c9500a6e0a20ee4a79b861638a084adf77dbfb78"
  }
}
$ curl -XDELETE -H'Content-Type: application/json' -d @checkout.json localhost:6000/return

Note that the complete response from batch checkout may be sent. The only requirement is the id field for each pool instance.

Building

$ govendor get -d github.com/boz/ephemerald/...
$ cd $GOPATH/src/github.com/boz/ephemerald
$ make server example

Run the example server and client in separate terminals

$ ./ephemerald/ephemerald -c _example/config.yaml

$ ./_example/example

Installing

Source

Follow the building steps then run make install:

$ make install

Binary

Download the latest release for your system. Unpack the archive and put the binary in your path.

$ release="https://github.com/boz/ephemerald/releases/download/v0.3.1/ephemerald_Linux_x86_64.tar.gz"
$ curl -L "$release" | tar -C /tmp -zxv
$ /tmp/ephemerald -c config.yaml

Homebrew

$ brew install boz/repo/ephemerald

TODO

• Configuration
  • Current parsing is a disaster
  • Allow yaml
  • Allow built-in defaults (postgres, redis, etc...)
• Polish/Optimize/Cleanup/Refactor UI.
• Re-add remote actions (websockets API)
• Clients: nodejs, ruby, python, etc...
• Documentation
• Tests

Documentation

Index

• type Item
• type Pool
  • func NewPool(config *config.Config) (Pool, error)
  • func NewPoolWithContext(ctx context.Context, config *config.Config) (Pool, error)
• type PoolSet
  • func NewPoolSet(log logrus.FieldLogger, ctx context.Context, configs []*config.Config) (PoolSet, error)
• type StatusItem

Types

type Item

type Item interface {
	ID() string
}

type Pool

type Pool interface {
	Checkout() (params.Params, error)
	CheckoutWith(context.Context) (params.Params, error)
	Return(Item)
	Stop() error
	WaitReady() error
}

func NewPool

func NewPool(config *config.Config) (Pool, error)

func NewPoolWithContext

func NewPoolWithContext(ctx context.Context, config *config.Config) (Pool, error)

type PoolSet

type PoolSet interface {
	Checkout(name ...string) (params.Set, error)
	CheckoutWith(ctx context.Context, name ...string) (params.Set, error)
	ReturnAll(params.Set)
	Return(name string, item Item)
	WaitReady() error
	Stop() error
}

func NewPoolSet

func NewPoolSet(log logrus.FieldLogger, ctx context.Context, configs []*config.Config) (PoolSet, error)

type StatusItem

type StatusItem interface {
	Item
	Status() types.ContainerJSON
}