binp

README

binp

binp (pronounced "bin-pea") is a bin-packing tool to minimize costs when deploying architectures. It is similar to the functionality built into Kubernetes or Nomad, but this tool is designed under the UNIX philosophy of doing one thing well, favoring composability with other tools.

binp has one major function: given a description of a machine and services to run, how many machines are necessary, and which programs should run on each machine? In doing this, it also reports the excess resources, so you can decide if you want to decrease machine resources to save costs.

binp is both a library that can be used in other programs, as well as a CLI executable.

Install

$ go get -u egt.run/binp

How to use

binp accepts an inventory JSON file of servers and the services that can run on each of them and outputs a mapping of which services to run on numbered boxes.

For example, this could be our services.json file:

{
	"requirements": {
		"app_1": {
			"cpu": 2,
			"ram": "4 GB",
			"disk": "100 MB",
			"net": {"ports": [3000]}
		},
		"app_2": {
			"ram": "2 GB",
			"disk": "50 MB",
			"net": {"ports": [4000]}
		},
		"app_3": {
			"net": {
				"mbps": 1000,
				"ports": [3000, 4000]
			}
		},
		"postgres": {
			"cpu": 6,
			"ram": "32 GB",
			"disk": "1 TB",
			"net": {
				"mbps": 100,
				"ports": [5432]
			}
		},
		"redis": {
			"cpu": 1,
			"ram": "2 GB",
			"disk": "4 GB"
		}
	},
	"counts": {
		"openbsd": {
			"app_1": 4,
			"app_2": 2,
			"app_3": 2,
			"postgres": 2
		},
		"debian": {
			"redis": 1
		}
	},
	"providers": {
		"provider": {
			"openbsd": {
				"machineType": "type1",
				"disk": "1 TB",
				"image": "openbsd.tar.gz",
				"net": {"mbps": 1000}
			},
			"debian": {
				"machineType": "type2",
				"disk": "10 GB",
				"image": "debian.tar.gz",
				"net": {"mbps": 1000}
			}
		}
	},
	"machineTypes": {
		"type1": {"cpu": 12, "ram": "32 GB"},
		"type2": {"cpu": 1, "ram": "4 GB"}
	}
}

To run binp, we call this (output formatted slightly for legibility):

$ binp
{
	"provider": {
		"debian":  [
			["redis"]
		],
		"openbsd": [
			["postgres"],
			["postgres"],
			["app_1","app_2"],
			["app_1","app_2"],
			["app_1"],
			["app_1"]
		]
	}
}

binp's output details the services which should be run on each server, and the total number of servers needed.

To output just the number of machines needed, compose binp's output with other tools, like jq!

# Get the total number of servers to provision
$ binp | jq '.provider | flatten | length'

# Or get the number of servers for a single box type
$ binp | jq '.provider | .openbsd | length'

You could pass the length into something like Terraform to automatically provision the correct number of servers for your architecture, or the services themselves into Terrafirma.

To help save costs, binp can make recommendations as to the smallest CPU/RAM/Disk which meets the requirements in services.json file:

$ binp -min
openbsd:
	CPU:  6
	RAM:  32 GB
	Disk: 1 TB
	Net:  600 mbps
debian:
	CPU:  1
	RAM:  2 GB
	Disk: 1 TB
	Net:  0 mbps

Or the smallest CPU/RAM/Disk which will allow co-locating all services on a single box with -max.

$ binp -max
openbsd:
	CPU:  9
	RAM:  38 GB
	Disk: 1 TB
debian:
	CPU:  1
	RAM:  2 GB
	Disk: 4 GB

Algorithm

Bin-packing is an NP-hard problem. For that reason, binp does not try to find the optimal solution. Instead it tries to find a close-to-optimal solution that is good enough.

It uses a variation of the "first-fit descending algorithm," sorting services from largest to smallest with a scoring function, and then distributing services to each machine in rotation.

This approach finds solutions which distribute the same program on multiple machines whereever possible. This is especially useful for high availability environments where 1 (or several) VM failures should not result in an outage.

binp's solution is deterministic, so running multiple times given the same input will always yield the same output.

Documentation

Index

• type Box
  • func MaxBox(boxes []*Service) *Box
  • func MinBox(boxes []*Service) *Box
• type BoxName
• type BoxOpts
• type Config
  • func ParseConfig(filename string) (*Config, error)
  • func (c *Config) Pack() (map[ProviderName]map[BoxName][]*Box, error)
  • func (c *Config) ServicesForBox(boxName BoxName) (map[ServiceName]*Service, error)
• type MachineName
• type MachineType
• type ProviderName
• type Service
• type ServiceName

Types

type Box

type Box struct {
	CPU      int        `json:"cpu"`
	RAM      datasize   `json:"ram"`
	Disk     datasize   `json:"disk"`
	Net      network    `json:"net"`
	Services []*Service `json:"-"`
	// contains filtered or unexported fields
}

func MaxBox

func MaxBox(boxes []*Service) *Box

func MinBox

func MinBox(boxes []*Service) *Box

type BoxName

type BoxName string

type BoxOpts

type BoxOpts struct {
	MachineType MachineName `json:"machineType"`
	Disk        datasize    `json:"disk"`
	Net         network     `json:"net"`
}

type Config

type Config struct {
	// Requirements for each service that binp must satisfy.
	Requirements map[ServiceName]*Service `json:"requirements"`

	// Counts desired for each service. The number of boxes required to
	// satisfy the count is calculated by binp automatically.
	Counts map[BoxName]map[ServiceName]int `json:"counts"`

	// Providers describes the type of VM, the disk size, and the image to
	// use for each provider:region:zone combo.
	Providers map[ProviderName]map[BoxName]*BoxOpts `json:"providers"`

	// MachineTypes translates the provider-specific machine type names
	// (e.g. "n1-standard-1") into CPU and RAM values that binp can use.
	MachineTypes map[MachineName]*MachineType `json:"machineTypes"`
}

func ParseConfig

func ParseConfig(filename string) (*Config, error)

func (*Config) Pack

func (c *Config) Pack() (map[ProviderName]map[BoxName][]*Box, error)

Pack services into boxes.

func (*Config) ServicesForBox

func (c *Config) ServicesForBox(
	boxName BoxName,
) (map[ServiceName]*Service, error)

type MachineName

type MachineName string

type MachineType

type MachineType struct {
	CPU int      `json:"cpu"`
	RAM datasize `json:"ram"`
}

type ProviderName

type ProviderName string

type Service

type Service struct {
	CPU   int         `json:"cpu"`
	RAM   datasize    `json:"ram"`
	Disk  datasize    `json:"disk"`
	Net   network     `json:"net"`
	Count int         `json:"-"`
	Name  ServiceName `json:"-"`
}

type ServiceName

type ServiceName string