raccoon

package module
v0.1.1-0...-7f03b72 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 26, 2016 License: GPL-3.0 Imports: 18 Imported by: 0

README

raccoon

WIP App orchestration, configuration and deployment

Raccoon logo

asciicast

Features

  • Pretty Output in real time
  • Dockerfile Syntax to ease learning path. WIP
    • RUN
    • ADD
    • ADD (folder contents)
    • MAINTAINER
    • ENV
  • Identity file auth.
  • Interactive user/password auth.
  • Cluster level authentication
  • 1 file JSON parsing of jobs (a file with the infrastructure connected with the task list)
  • 2 files JSON parsing of jobs (one file with infrastructure pointing to tasks in a second file of tasks list).
  • 3 files JSON parsing of jobs (a file for infrastructure, a file for tasks list and a file to connect them both).
  • Single task - single cluster 2 files automation in CLI. Like 3 files parsing but using a --task --cluster flags so it will only execute that task in the specified cluster. Of course, it also needs the infrastructure and task list.
  • Pure Dockerfile parsing. Reuse your Dockerfiles in normal hosts.
  • Templating
  • Array based tasks for cluster
  • API REST.
  • Support for JSON syntax parsing
  • CLI
  • Automation tests
  • Target information retrieval
  • Support for TOML syntax

Index

Raccoon CLI syntax

When you execute Raccoon from the command line without options, the following help will appear:

    NAME:
       raccoon job - Execute a job

    USAGE:
       raccoon job [command options] [arguments...]

    OPTIONS:
       --tasks value, -t value           Tasks file
       --infrastructure value, -i value  Infrastructure file
       --dispatcher value, -d value      Dispatching method between 3 options: sequential (no concurrent dispatch). simple (a Goroutine for each host) and worker_pool (a fixed number of workers) (default: "simple")
       --workersNumber value, -w value   In case of worker_pool dispath method, define the maximum number of workers here (default: 5)

How to use Raccoon as a standalone app

You use Raccoon similar of how you use tools like Ansible. You mainly work with two things:

  • An Infrastructure: An infrastructure is a list of clusters and a cluster is a list of hosts.
  • A Task list: A task is a group of commands to achieve some task. You can have tasks of only one command (for example, 'apt-get install htop' if you just want to install htop) or a tasks composed of many commands (like a list of installation commands to take some host to a desired state).

It's quite easy, you just need to define your infrastructure and the tasks you want to execute in every cluster, then you launch the application.

What's an Infrastructure:

An infrastructure definition looks like the following:

{
  "name":"Your infrastructure name (your project, for example)",
  "infrastructure":[
    {
      "name":"A name for a cluster (like Kafka hosts)",
      "tasks":["Install Kafka", "Open Kafka ports"],
      "hosts":[
        {
          "ip":"172.17.42.1",
          "sshPort":32768,
          "description":"A cluster authenticated at host level",
          "username":"root",
          "password":"root"
        }
      ]
    },
    {
      "name":"All hosts in this cluster share authentication",
      "tasks":["Install Kafka", "Open Kafka ports"],
      "username":"root",
      "password":"root",
      "hosts":[
        {
          "ip":"172.17.42.1",
          "sshPort":32768,
          "description":"hdfs02"
        },
        {
          "ip":"172.17.42.1",
          "sshPort":32769,
          "description":"hdfs03"
        }
      ]
    }
  ]
}
  • name: Is a name for your infrastructure, it is optional and its purpose is to identify the file between many infrastructure files. It could be your project's name or your company name.
  • infrastructure: List of clusters. Take a closer look. It is a JSON array.
    • name: Name of the cluster. They should describe the cluster grouping in some way like Cassandra machines or QA machines.
    • tasks: List of tasks that will be executed on this cluster
Host Level vs Cluster level authenticationnn

If all your hosts share the same authentication details, you can use the same syntax that you can use with the hosts in the cluster definition. It will apply to all hosts. So, for example, if we have two hosts that share the authentication details we can define our JSON this way:


    {
      "name":"All hosts in this cluster share authentication",
      "tasks":["Install Kafka", "Open Kafka ports"],
      "username":"root",
      "password":"root",
      "hosts":[
        {
          "ip":"172.17.42.1",
          "sshPort":32768,
          "description":"hdfs02"
        },
        {
          "ip":"172.17.42.1",
          "sshPort":32769,
          "description":"hdfs03",
          "password":"1234"
        }
      ]
    }

We have set the username and password at cluster level but we have set the SSH port at host level. The cluster authentication details will be used on each of its hosts. You can also override some detail at host level if neccesary.

Host

A host is a description of some host on your infrastructure. It contains authentication methods (defined by the structure of the JSON object, more on this later), an IP and a username.

  • hosts: List of hosts that compose this cluser.
    • ip: IP address to access the host.
    • username: Username to access the machine.
    • sshPort: If needed, SSH port to make the SSH connection to.
    • password: Password for the provided username.
    • identityFile: A file to authenticate the provided user
    • interactiveAuth: Prompt for the password
Authentication methods

Raccoon uses 3 possible authentication methods: user and password, identity file and interactive access.

User and password

When using user and password authentication, you have to provide in the infrastructure definition a username and password key in the host definition. For example:

  {
    "ip":"172.17.42.1",
    "sshPort":32768,
    "username":"root",
    "password":"root"
  }
Identity file

You can also use private/public key authentication to access some host. To use it, add the path to the identity file as a key called identityFile in host definition. don't forget to add the username key on host too. For example:

  {
    "ip":"172.17.42.1",
    "sshPort":32768,
    "username":"root",
    "identityFile":"/home/mcastro/.ssh/id_rsa"
  }
Interactive authentication

The interactive authentication will prompt the user for a password. Set interactiveAuth to true on host definition. As before, the user must be set in the host. Use sequential dispatching to avoid that the stdout gets filled before you can actually see the prompt for the password. For example:

  {
    "ip":"172.17.42.1",
    "sshPort":32768,
    "username":"root",
    "interactiveAuth": true
  }

What's a Task list

A task is a group of commands that will be executed on the targeted host. A command refers, for example, to a shell command. See the commands index below.

Each task can be paired to a cluster, and that task will be executed on the cluster. The syntax of the task list is the following:

[
  {
    "title": "task1",
    "maintainer": "Burkraith",
    "commands": [
      {
        "name": "ADD",
        "sourcePath": "doc.go",
        "destPath": "/tmp",
        "description": "Raccoon.go to /tmp"
      },
      {
        "name": "RUN",
        "description": "Removing htop",
        "instruction": "sudo yum remove -y htop"
      },
      {
        "name": "ADD",
        "sourcePath": "main.go",
        "destPath": "/tmp",
        "description": "copying raccoon.go to /tmp"
      }
    ]
  },
  {
    "title": "task2",
    "maintainer": "Mario",
    "commands": [
      {
        "name": "RUN",
        "description": "Removing htop",
        "instruction": "sudo apt-get remove -y htop"
      },
      {
        "name":"ADD",
        "description":"Adding the main.go file",
        "sourcePath":"main.go",
        "destPath":"/tmp"
      }
    ]
  }
]

First of all, take a close look because this JSON doesn't have root key. In this example we have two tasks and each task is composed of:

  • title: This is a very important piece. The title of this task is referred from the insfrastructure definition. So if we have in a cluster:
{
  "name":"myCluster",
  "tasks":["task1"]
  ...
}

The task with a title "task1" will be executed in cluster "myCluster". This is how Raccoon recognizes how to pair a task with a cluster.

  • maintainer: Optional description for the maintainer of the task.
  • commands: The list of commands that this task is composed of
    • name: Command name. This must be one of the commands described below. ADD, RUN, MAINTAINER are all valid commands.
    • description: An optional description to print to stdout when executing this command.
    • specific key-values for each command: All commands have a name and description in common. Then they have specific key-value pairs that contains the information to run the command. For example RUN command has the key instruction with the shell command you want to execute. Refer to the commands section for more information.

Commands:

Until now we have developed the following commands with their corresponding JSON formats (NOTE: bold fields are mandatory)

RUN
      {
        "name": "RUN",
        "description": "Removing htop",
        "instruction": "sudo yum remove -y htop"
      }

Like in Docker, RUN will execute the command on the target machine, the parameters are:

  • name: The name of the command for the parser, for a RUN instruction is always "RUN"
  • instruction: The command to execute in the target machine
  • description: A description of the instruction. This has no effect and it is only for logging purposes
ADD
      {
        "name": "ADD",
        "sourcePath": "raccoon.go",
        "destPath": "/tmp",
        "description": "copying raccoon.go to /tmp"
      }

ADD uses scp to send a file to the destination machine. It just supports single files yet until we work in a folder solution (to send an entire folder):

  • name: "ADD" must always be placed here so that the parser recognizes the instruction
  • sourcePath: The full source path and file name of the file to send. For example: /tmp/fileToSend.log. It can also be a folder. In this case, it will copy just the first level of files from the folder and ignore any nested folders.
  • destPath: The full path to leave the file into the target machine. The file will have the same name.
  • description: Optional description parameters for logging purposes.
ENV

Sets an environment variable on the target machine:

{
    "name": "ENV",
    "description": "Sets the variable GOROOT to /usr/local/go",
    "environment": "GOROOT=/usr/local/go"
}

The parser will look for a "=" in the "environment" value to split it into two pieces and set the environment variable.

  • name: "ENV" must always go here to use the ENV instruction
  • environment: A key-value separated by a "=" to set in the target machine. Left
  • description: Optional description parameters for logging purposes. side will be the environment name. Right side its value.
MAINTAINER

Prints the name of a maintainer as a specific command:

{
  "name":"MAINTAINER",
  "description":"Mario Castro"
}
  • name: Must always contain "MAINTAINER" if you want to use this feature.
  • description: the name of the maintainer.

Dispatching strategies

Raccoon can use 3 dispatching strategies. This strategies must be used in some specific scenarios:

Simple dispatching

This is the default strategy, Raccoon will launch a new goroutine for each host found on the infrastructure definition. This means that if you have 5 clusters of 5 hosts each, you have a total of 25 hosts and 25 goroutines will be launched concurrently to execute the task list.

As you can imagine, if your definition has a large number of hosts this could not be the best strategy. Depending on your server machine, you could be able to reach a higher or lower amount of hosts. Check the workers pool strategy if you have performance issues.

Simple strategy is used by default but you can force it by passing -d simple when launching Raccoon CLI.

Workers pool

Workers pool strategy limits the amount of goroutines that can be accessing hosts concurrently. This means that if you have 1000 hosts, you can set the workers pool to 10 and only 10 goroutines will be launched to work. Every time that a goroutine finishes with some host it starts with the next until they reach the end. Use this strategy if you are having performance issues.

To use the workers pool strategy pass -d workers_pool -w [n] when launching Raccoon through the CLI. [n] is the maximum number of workers you want.

Sequential dispatching

In some special situations, you could want to avoid concurrency when accessing hosts. The sequential strategy will go one host each time. This is useful, for example, if you use the interactive authentication.

REST Server

A REST server can be deployed by using the following command:

  raccoon server

This will deploy the server on port 8123, if you want to specify a different port you can use the flag --port

To use the server as a client, you must send a JSON with the same syntax that the Single file automation.

Documentation

Overview

Package raccoon is a utility for simple automation of hosts by using Dockerfile syntax. You use ADD, RUN, MAINTAINER, etc. commands as you would do in Docker but to provision hosts instead of containers using SSH. In this aspect, you could need to have root privileges in a machine to do some specific actions.

Raccoon uses JSON syntax to define its behaviour. This way you can generate Raccoon jobs by simply writing a JSON with whichever language you want and it provides a nice way to interact with the Raccoon server.

You can use Raccoon as a standalone app, running as server or as a library. Ideally the three modes should work nicely but some errors are expected as the heavy test of the package is done as a standalone app.

How to use Raccoon as a library

Raccoon can be easily used as a library. For each cluster and task, you need to create a hierarchy of Jobs. A Job is a type with two members: Cluster and Task. A Cluster is a list of Hosts and a

How to user Raccoon as server

Raccoon can also be used as server. Doing POST request you can launch some automation on some infrastructure.

Index

Constants

View Source 
const (
	//VERSION number will be printed when using CLI
	VERSION string = "0.3.3"

	//APP_DESCRIPTION will be printed when using CLI
	APP_DESCRIPTION string = "WIP Automation utility made easy with Dockerfile syntax"

	//APP_NAME is just in case I forget the name of the app :)
	APP_NAME string = "Raccoon"
)

Constants file define global constants around the app

Variables

This section is empty.

Functions

This section is empty.

Types

type Authentication 

type Authentication struct {
	//Username to access remote host
	Username string `json:"username,omitempty"`

	//Password to access remote host
	Password string `json:"password,omitempty"`

	//Identity file for auth
	IdentityFile string `json:"identityFile,omitempty"`

	//Choose to enter user and password during Raccoon execution
	InteractiveAuth bool `json:"interactiveAuth,omitempty"`

	//SSHPort is a optional non standard port for SSH. We consider 22 as the
	//standard port so set this field in case you are using a different one in
	//any host
	SSHPort int `json:"sshPort,omitempty"`
}

type Cluster 

type Cluster struct {
	Authentication

	//Name that identifies this cluster respect to others
	Name string `json:"name"`

	//Hosts are the array of hosts on this cluster
	Hosts []Host `json:"hosts"`

	//TasksToExecute is the name of the commands group that will be executed in
	// this cluster. This name must match the name written in the 'Title' member
	// of the Tasks file. In future version it will work with an array os tasks too.
	TasksToExecute []string `json:"tasks"`
}

Cluster is an array of Nodes (remotes machines) that compose our machine cluster

It must have a name to identify it from other clusters.

It should also have a linked task. A task is a group of commands that will be executed on this cluster

type Command 

type Command struct {

	//Name if a must in each command. It could be ADD or ENV but every command
	//must have one so that the parser can recognize the intents of the user
	Name string

	//Description is an optional description of a command that will be print on
	//stdout for logging purposes. It's recommended to use a description on each
	//command
	Description string
}

Command is the type that contains common command members. Every new command in the commands package must have a Command value inside so that logging can work properly.

type CommandsExecutor 

type CommandsExecutor interface {

	//Execute is the method that each strategy will execute on the provided
	//Host h
	Execute(h Host)

	//GetCommands returns the Command value that must be within every CommandExecutor implementor
	GetCommand() *Command
}

CommandsExecutor is an interface that every command must implement. A command is any strategy of the Dockerfile syntax that Raccoon offers.

type Dispatcher 

type Dispatcher interface {
	Dispatch(js []Job)
}

Dispatcher is the interface to implement new dispatching strategies in the app

type Host 

type Host struct {
	Authentication

	//IP of the remote host
	IP string `json:"ip"`

	//Description is optional information about the host
	Description string `json:"description,omitempty"`

	//Color that this host will output when printing in stdout
	Color color.Attribute

	//HostLogger is the logging mechanism for each host. Use this when logging
	//anything to stdout so it will be formatted and colored
	HostLogger *logrus.Logger
	// contains filtered or unexported fields
}

Host is a remote machine (virtual or physical) where we will execute our instructions on.

func (*Host) CloseNode 

func (h *Host) CloseNode() error

CloseNode closes stored ssh session in Host. Remember to call it explicitly after all instructions has finished

func (*Host) GetClient 

func (h *Host) GetClient() (*ssh.Client, error)

GetClient will create a random color for logging and a new connection to a Host on port 22.

func (*Host) GetSession 

func (h *Host) GetSession() (*ssh.Session, error)

GetSession returns a new session once a client is created. In case the connection was lost, it tries to connect again with the Host

func (*Host) InitializeNode 

func (h *Host) InitializeNode() error

InitializeNode must be called prior any execution on Hosts. It will try to get a ssh.Client object to open ssh session on host

type Infrastructure 

type Infrastructure struct {
	//Name for your infrastructure. It has no effect on the app but maintains
	//infrastructure json files ordered
	Name string `json:"name"`

	//Infrastructure is, in turn, the array of clusters that you want to execute
	//some automation on
	Infrastructure []Cluster `json:"infrastructure"`
}

Infrastructure represents a group of clusters. Each cluster represents a group of hosts.

type Job 

type Job struct {

	//Cluster is a group of hosts
	Cluster Cluster

	//Task is a single task that could be composed of one or more commands
	Task Task
}

Job is that relate a Cluster (group of hosts) with the Task that must be performed on it

type JobRequest 

type JobRequest struct {
	TaskList       *[]Task         `json:"tasks"`
	Infrastructure *Infrastructure `json:"infrastructure"`
}

JobRequest is a job defined by the user in a single JSON object

type Relation 

type Relation struct {
	ClusterName string   `json:"clusterName"`
	TaskList    []string `json:"tasks"`
}

type RelationList 

type RelationList []Relation

type SequentialDispatcher 

type SequentialDispatcher struct{}

SequentialDispatcher is a dispatching strategy that doesn't use any concurrent approach. It will execute each command on each host sequentially and waits for each to finish before starting the next.

func (*SequentialDispatcher) Dispatch 

func (s *SequentialDispatcher) Dispatch(jobs []Job)

Dispatch ranges over the hosts to execute the tasks on them sequentially without any kind of concurrency.

type SimpleDispatcher 

type SimpleDispatcher struct {
	sync.WaitGroup
}

SimpleDispatcher is the dispatching strategy to use when the number of hosts isn't very high. It will open a new SSH connection to each host at the same time since the beginning so, if you are having performance issues, try the workers pool or the sequential dispatcher to see if they get solved. SimpleDispatcher is the default dispatching strategy

func (*SimpleDispatcher) Dispatch 

func (s *SimpleDispatcher) Dispatch(jobs []Job)

Dispatch will create a new goroutine for each host in the job and launch the tasks that were specified on it.

type Task 

type Task struct {

	//Title of the task that will be referenced from clusters
	Title string `json:"title"`

	//Maintainer is an optional member
	Maintainer string `json:"maintainer,omitempty"`

	//Commands are the array of commands that will be executed on some host.
	//This member must never come from JSON that's why it has that key name
	Commands []CommandsExecutor `json:"noJson,omitempty"`

	//Command is the syntax definition of a command that Raccoon will interpret
	//into a CommandExecutor value
	Command []map[string]string `json:"commands"`
}

Task is a list of commands to execute on a host. A task must have a Title that must match with the title that was written in the task key of the infrastructure file

type WorkerPoolDispatcher 

type WorkerPoolDispatcher struct {
	Workers int
	sync.WaitGroup
}

WorkerPoolDispatcher is a dispatcher that acts as a worker pool. It won't maintain more connections to hosts atthe same time than the specified number in the Workers field

func (*WorkerPoolDispatcher) Dispatch 

func (w *WorkerPoolDispatcher) Dispatch(jobs []Job)

Dispatch will create the specified workers on the workers field, launching a goroutine with each. Then it will create a dispatcher routing in a different goroutine and finally it will send the jobs to the launched dispatcher for distribution.

Source Files

View all Source files

Directories

Path Synopsis
Package main contains the CLI application of Raccoon along with the options of the CLI interface
 Package main contains the CLI application of Raccoon along with the options of the CLI interface
Package instructions contains all implemented syntax in Raccoon.
 Package instructions contains all implemented syntax in Raccoon.
Package parser contains the interpreter that parses the JSON tasks and insfrastructure files into Jobs
 Package parser contains the interpreter that parses the JSON tasks and insfrastructure files into Jobs

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.