j2

command module

v0.0.0-...-9d0dce2 Latest Latest Go to latest Published: Mar 21, 2018 License: Apache-2.0 Imports: 18 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/pulcy/j2

Links

Open Source Insights

README ¶

J2

j2 is the pulcy service deployment tool. It takes a job description as input and generates (fleet) unit files for all tasks in the given job. The unit files will be pushed onto a CoreOS cluster.

Usage

To create/update a job on a cluster, run:

j2 run -j <jobpath> -c <clusterpath> [-o <optionspath]

To completely remove a job from a cluster, run:

j2 destroy -j <jobpath> -c <clusterpath>

Job specification

A job is a logical group of services.

A job contains one or more task groups. A task group is a group of tasks that are scheduled on the same machine. A task specifies a single container.

A very basic job looks like this:

job "basic" {
    task "someservice" {
        image = "myimage"
    }
}

Jobs are specified in HCL format. A job always has a name and only one job can be specified per file.

Objects in a job can be task, group and constraint. If you add a task directly to a job, it will be automatically wrapped in a task-group.

The following keys can be specified on a job.

id - id is used to give a job a unique identifier which is used for authentication when fetching secrets.
constraint - See Constraints

Tasks

A task is an object that specifies something that will be executed in a container.

The following keys can be specified on a task.

args - Contains the command line arguments passed during the start of the container.
environment - Contains zero of more environment variables passed during the start of the container.
image - Specifies the docker image that will be executed.
type - The type of a task can be "service" (default), "oneshot" or "proxy". Oneshot tasks are supposed to run for a while and then exit. Service tasks are supposed to run continuously and will be restarted in case of failure. Proxy tasks do not have a real service to run, instead they provide a virtual service that forwards all requests to another service (optionally with a rewrite rule).
after - Contains the name of zero or more other tasks in the same group. If set, this task will be started only after all listed tasks have been started.
volumes-from - Contains the name of zero or more other tasks. The volumes used by these other tasks will be mounted in the container of this task.
volumes - Contains a list of zero or more volume mounts for the container. Each volumes entry must be a valid docker volume description ("hostpath:containerpath")
ports - Contains a list of port specifications that specify which ports (exposed by the container) will be mapped into the port namespace of the machine on which the container is scheduled. Each port entry must be a valid docker port specification. Note that ports are not often used. In most cases you'll use a frontend or private-frontend.
links - Contains a list of task names that this task will be able to access through their private frontends. Each name must be a fully qualified task name (job.group.task).
capabilities - Contains a list of Linux capabilities to add to the container. (See docker run --cap-add)
constraint - See Constraints
http-check-path - Contains an HTTP path for the load-balancer to call when checking the status of this task.
frontend - Contains a public load-balancer registration. This configures the load-balancer to forward certain requests from the public network interface(s) of the cluster to this task. See Frontends.
private-frontend - Contains a private load-balancer registration. This configures the load-balancer to forward certain requests from the private network interface(s) of the cluster to this task. See Frontends.
secret - Contains a specification for a secret value to be fetched and mapped into the container. See Secret.
log-driver - Specifies the log-driver to use in docker. This can be "" (default) or "none". If not equal to "none", the log-args or the docker settings in the cluster are used.
target - The name of the task to forward requests to. Only used with type==proxy.

Frontends

Frontends are used to provide a configuration for the load-balancer.

The following keys can be specified on a public frontend.

domain - The load-balancer will forward requests that match this domain.
path-prefix - The load-balancer will forward requests where the path of the requests starts with this prefix.
ssl-cert - The load-balancer will use an SSL certificate with this filename for connections to this task. If you do not specify an SSL certificate and the load-balancer is configured to use Let's Encrypt a certificate will be automatically created for the specified domain.

The following keys can be specified on a all frontends.

port - The load-balancer will forward requests to this port of the task. If you do not specify a port, it will forward requests to any of the ports exposed by the container.
host-port - The load-balancer will listen on this port for requests intended for the task. If you do not specify a host-port, standard ports will be assumed.
mode - Specifies the mode the load-balancer will be configured for for this frontend. Mode can be http (default) or tcp. Frontends using tcp mode with a domain setting will offer TCP over TLS connections, using SNI to identify the correct task to forward the request to. The connection from the load-balancer to the task will use TCP only. Frontends using tcp mode without a domain setting request a host-port setting.
user - User objects specify password authentication to be used for requests forwarded for this task.
weight - Contains a value [0...100] used to order frontend specifications in the load-balancer. If 2 frontend specifications both match a specific request, the one with the highest weight will be used.

The following keys can be specified on a private frontend.

register-instance - If set, instances of this task will also be registered in the load-balancer under an instance specific name. This enables access to individual instances, in addition to load-balanced access to all instances of a task.

Secrets

Secrets are used to pass sensitive data to tasks in a secure manor. The sensitive data can be exposed as an environment variable (e.g. passwords) or as a file (e.g. certificates). Secrets are extracted from a Vault.

A secret looks like this:

secret "secret/mypassword" {
    environment = "MYPASSWORD"
}

The above secret results in the value of a secret under path "secret/mypassword" to be passed to the task in an environment variable named "MYPASSWORD".

secret "secret/mycertificate" {
    file = "/config/mycertificate.pem"
}

The above secret results in the value of a secret under path "secret/mycertificate" to be passed to the task in a file mounted on "/config/mycertificate.pem".

The following keys can be specified on a secret.

field - Contains the name of the field of the secret. If no field is specified, the value field is fetched.
environment - Contains the name of the environment variable that will be passed into the container.
file - Contains the full path of the file that will be mounted into the container.

You must specify an environment or a file, not both.

Task groups

A group is an object that groups one or more tasks such that they are always scheduled on the same machine. You use the count key to specify how many instances of a task-group you want to run. Each instance of a task-group contains a container for all tasks in that task-group. Multiple instances of a task-group are not guaranteed to run on the same machine. In fact you often want different instances to run on different machines for high availability.

The following keys can be specified on a group.

count - Specifies how many instances of the task-group that should be created.
global - If set to true, this task-group will create one instance for every machine in the cluster.
constraint - See Constraints
restart - If set to all, all tasks of this group will be restarted in case one of them restarts (or is updated).

Constraints

With constraints you can control on which machines tasks can be scheduled. Constraints can be specified on job, group and task level. Constrains on a deeper level overwrite constraints on high levels with the same attribute.

Here's an example of a constraint that forced a task to be scheduled on a machine that has region=eu-west in its metadata.

constraint {
    attribute = "meta.region"
    value = "eu-west"
}

The following keys can be specified on a constraint.

attribute - One of the attributes you can filter on. See Attributes below.
value - The value for this attribute.

Attributes

The following attributes can be used in constraints.

meta.<key> - Refers to a key used in the metadata of a machine.
node.id - Refers to the machine-id of a machine.

Cluster specification

A cluster file specifies those attributes of a cluster that are relevant for deploying jobs on it.

A typical cluster file looks like:

cluster "production" {
    domain = "example.com"
    stack = "production"
    instance-count = 3

    default-options {
        "force-ssl" = "true"
    }
}

A cluster file for a cluster provisioned by Gluon can look like this:

cluster "production" {
    domain = "example.com"
    stack = "production"
    instance-count = 3

    docker {
        log-args = ["--log-driver=fluentd", "--log-opt fluentd-address=127.0.0.1:24284"]
    }

    fleet {
        after = [
            "gluon.service",
            "loggly-fluentd-fluentd-mn.service"
        ]
        wants = "loggly-fluentd-fluentd-mn.service"
    }

    default-options {
        "force-ssl" = "true"
    }
}

The following keys can be specified on a cluster.

domain - The domain name of the cluster
stack - The name of the stack to deploy in. The combination of stack + domain forms the DNS name that is used to deploy units to.
instance-count - The number of machines in the cluster. If this number is higher than the count of a task-group, different instances of that task-group will be forced on different machines.
docker - A set of options applied to all docker commands generated for jobs on this cluster.
docker.log-args - Docker command line arguments related to logging. Uses when log-driver is not equal to none.
fleet - A set of options applied to all fleet units generated for jobs on this cluster.
fleet.after - A list of unit names to add to the After setting of each unit. This list is filtered such that units originating from the same job are excluded.
fleet.wants - A list of unit names to add to the Wants setting of each unit. This list is filtered such that units originating from the same job are excluded.
fleet.requires - A list of unit names to add to the Requires setting of each unit. This list is filtered such that units originating from the same job are excluded.
fleet.global-instance-constraints - A list of metadata constraints. One of them is added to an instance of a global unit with a count higher than 1. For example with global-instance-constraints = ["global=1", "global=2"] instance 1 will get global=1 and instance 2 will get global=2 as metadata. You can choose how to spread these metadata's across the machines of the cluster, but make sure that every machine has global=1 OR global=2 in its metadata and not both.

Why is it called J2?

This tool is named after the famous J-2 rocket engine that helped bring man to the moon. It was a predecessor for the RS-25 rocket engine that powered the Space Shuttle and even today it is an inspiration for the J-2X engine intended for NASA's Space Launch System.

Documentation ¶

There is no documentation for this package.

Source Files ¶

View all Source files

Directories ¶

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL

Path	Synopsis
cluster
deployment
deps
github.com/YakLabs/k8s-client Package client provides a simple Kubernetes client	Package client provides a simple Kubernetes client
github.com/YakLabs/k8s-client/http Package http provides an HTTP client for kubernetes	Package http provides an HTTP client for kubernetes
github.com/YakLabs/k8s-client/intstr
github.com/cenkalti/backoff Package backoff implements backoff algorithms for retrying operations.	Package backoff implements backoff algorithms for retrying operations.
github.com/coreos/etcd Package main is a simple wrapper of the real etcd entrypoint package (located at github.com/coreos/etcd/etcdmain) to ensure that etcd is still "go getable"; e.g.	Package main is a simple wrapper of the real etcd entrypoint package (located at github.com/coreos/etcd/etcdmain) to ensure that etcd is still "go getable"; e.g.
github.com/coreos/etcd/alarm Package alarm manages health status alarms in etcd.	Package alarm manages health status alarms in etcd.
github.com/coreos/etcd/auth Package auth provides client role authentication for accessing keys in etcd.	Package auth provides client role authentication for accessing keys in etcd.
github.com/coreos/etcd/auth/authpb Package authpb is a generated protocol buffer package.	Package authpb is a generated protocol buffer package.
github.com/coreos/etcd/client Package client provides bindings for the etcd APIs.	Package client provides bindings for the etcd APIs.
github.com/coreos/etcd/client/integration Package integration implements tests built upon embedded etcd, focusing on the correctness of the etcd v2 client.	Package integration implements tests built upon embedded etcd, focusing on the correctness of the etcd v2 client.
github.com/coreos/etcd/clientv3 Package clientv3 implements the official Go etcd client for v3.	Package clientv3 implements the official Go etcd client for v3.
github.com/coreos/etcd/clientv3/concurrency Package concurrency implements concurrency operations on top of etcd such as distributed locks, barriers, and elections.	Package concurrency implements concurrency operations on top of etcd such as distributed locks, barriers, and elections.
github.com/coreos/etcd/clientv3/integration Package integration implements tests built upon embedded etcd, and focuses on correctness of etcd client.	Package integration implements tests built upon embedded etcd, and focuses on correctness of etcd client.
github.com/coreos/etcd/clientv3/mirror Package mirror implements etcd mirroring operations.	Package mirror implements etcd mirroring operations.
github.com/coreos/etcd/clientv3/naming
github.com/coreos/etcd/compactor Package compactor implements automated policies for compacting etcd's mvcc storage.	Package compactor implements automated policies for compacting etcd's mvcc storage.
github.com/coreos/etcd/contrib/raftexample raftexample is a simple KV store using the raft and rafthttp libraries.	raftexample is a simple KV store using the raft and rafthttp libraries.
github.com/coreos/etcd/contrib/recipes
github.com/coreos/etcd/contrib/systemd/etcd2-backup-coreos
github.com/coreos/etcd/discovery Package discovery provides an implementation of the cluster discovery that is used by etcd.	Package discovery provides an implementation of the cluster discovery that is used by etcd.
github.com/coreos/etcd/e2e Package e2e implements tests built upon etcd binaries, and focus on end-to-end testing.	Package e2e implements tests built upon etcd binaries, and focus on end-to-end testing.
github.com/coreos/etcd/embed Package embed provides bindings for embedding an etcd server in a program.	Package embed provides bindings for embedding an etcd server in a program.
github.com/coreos/etcd/error Package error describes errors in etcd project.	Package error describes errors in etcd project.
github.com/coreos/etcd/etcdctl etcdctl is a command line application that controls etcd.	etcdctl is a command line application that controls etcd.
github.com/coreos/etcd/etcdctl/ctlv2 Package ctlv2 contains the main entry point for the etcdctl for v2 API.	Package ctlv2 contains the main entry point for the etcdctl for v2 API.
github.com/coreos/etcd/etcdctl/ctlv2/command Package command is a set of libraries for etcdctl commands.	Package command is a set of libraries for etcdctl commands.
github.com/coreos/etcd/etcdctl/ctlv3 Package ctlv3 contains the main entry point for the etcdctl for v3 API.	Package ctlv3 contains the main entry point for the etcdctl for v3 API.
github.com/coreos/etcd/etcdctl/ctlv3/command Package command is a set of libraries for etcd v3 commands.	Package command is a set of libraries for etcd v3 commands.
github.com/coreos/etcd/etcdmain Package etcdmain contains the main entry point for the etcd binary.	Package etcdmain contains the main entry point for the etcd binary.
github.com/coreos/etcd/etcdserver Package etcdserver defines how etcd servers interact and store their states.	Package etcdserver defines how etcd servers interact and store their states.
github.com/coreos/etcd/etcdserver/api Package api manages the capabilities and features that are exposed to clients by the etcd cluster.	Package api manages the capabilities and features that are exposed to clients by the etcd cluster.
github.com/coreos/etcd/etcdserver/api/v2http Package v2http provides etcd client and server implementations.	Package v2http provides etcd client and server implementations.
github.com/coreos/etcd/etcdserver/api/v2http/httptypes Package httptypes defines how etcd's HTTP API entities are serialized to and deserialized from JSON.	Package httptypes defines how etcd's HTTP API entities are serialized to and deserialized from JSON.
github.com/coreos/etcd/etcdserver/api/v3rpc Package v3rpc implements etcd v3 RPC system based on gRPC.	Package v3rpc implements etcd v3 RPC system based on gRPC.
github.com/coreos/etcd/etcdserver/api/v3rpc/rpctypes Package rpctypes has types and values shared by the etcd server and client for v3 RPC interaction.	Package rpctypes has types and values shared by the etcd server and client for v3 RPC interaction.
github.com/coreos/etcd/etcdserver/auth Package auth implements etcd authentication.	Package auth implements etcd authentication.
github.com/coreos/etcd/etcdserver/etcdserverpb Package etcdserverpb is a generated protocol buffer package.	Package etcdserverpb is a generated protocol buffer package.