spawn

README

Openprovider Spawn Sync service

The Spawn service used as a HTTP REST sync service, that makes clustering mode simpler and easier for most of applications. What's the idea? There are several applications, which are developed to provide their service through HTTP Rest API. But you have no idea how to provide failover processing and clustering mode for these applications, because they are not compatible with, etc. And here we go. The Spawn service will make this job instead of the whole bunch of services that must be configured and communicating with each other. How it works? Let's see the scheme:

All GET requests from the Client will reproduce with a selected node. A node for GET requests will be selected by specified 'round-robin' and 'by priority' mode if they are active. All updates requests (PUT, POST, DELETE) will be repeated to every node or will be accumulated in the corresponded queue if the node is unreachable.

If temporarily switch off the node 3 from the process, due to maintenance or network loss, the updates worker of the node 3 will be stopped automatically and all updates of the node 3 would begin to accumulate in the queue. After the end of maintenance work all accumulated updates will posted in the node 3.

Embedded health checker makes possible to recover processing automatically.

The service has API to control of the nodes. So, if you setup the Spawn API on localhost:7118, you can use query below to show what nodes are configured:

curl -XGET http://localhost:7118/nodes?pretty=true

{
  "duration": 25982,
  "took": "25.982µs",
  "data": {
    "results": [
      {
        "host": "node1.myapp.com",
        "port": 7017,
        "priority": 1,
        "active": true,
        "maintenance": false
      },
      {
        "host": "node2.myapp.com",
        "port": 7017,
        "priority": 2,
        "active": true,
        "maintenance": false
      }
    ],
    "success": true,
    "total": 2
  }
}

Of course you could modify/delete these setting with PUT/DELETE request. Use API helper '/list' to see all supported methods.

Currently this is not production version, follow the updates.

Restrictions

• HTTPS is not supported currently

Config

Example of a config file

{
  "host": "spawn.myapp.com",
  "port": 7117,
  "api": {
    "host": "spawn.myapp.com",
    "port": 7118
  },
  "query-mode": {
    "round-robin": true,
    "by-priority": true
  },
  "health-check": {
    "seconds": 10,
    "url": "/info",
    "regexp": "^version:[0-9]+$"
  },
  "nodes": [
    {
      "host": "node1.myapp.com",
      "port": 7017,
      "priority": 1,
      "active": true,
      "maintenance": false
    },
    {
      "host": "node2.myapp.com",
      "port": 7017,
      "priority": 2,
      "active": true,
      "maintenance": false
    },
    {
      "host": "node3.myapp.com",
      "port": 7017,
      "priority": 3,
      "active": false,
      "maintenance": false
    }
  ]
}

Install from source

You need have installed golang 1.4.2 (not 1.4.2+)

This service is "go-gettable", just do:

go get github.com/openprovider/spawn/spawnctl

Usage

spawnctl - Spawn Sync Service Control

Usage:
  spawnctl install | remove | start | stop | status
  spawnctl [ -t | --test ] [ --option | --option ... ]
  spawnctl -h | --help
  spawnctl -v | --version

Commands:
  install           Install as service
  remove            Remove service
  start             Start service
  stop              Stop service
  status            Check service status

  -h --help         Show this screen
  -v --version      Show version
  -t --test         Test mode

Options:
  --config=PATH          Path to the config file
  --host=HOST            Host name or IP address
  --port=PORT            Port number
  --api-host=HOST        API host name or IP address
  --api-port=PORT        API port number
  --round-robin          Use round-robin mode for querying of nodes
  --by-priority          Nodes will queried according to priority
  --check-sec=SECONDS    Check nodes every number of seconds
  --check-url=URL        URL to check nodes (/info, etc)
  --check-regexp=REGEXP  Regexp pattern to check nodes

Todo

• More of Tests coverage and benchmarks
• Synchronization between the nodes
• Exponential Back-off
• support of HTTPS
• Monitoring system inside
• Admin panel for monitoring and configuration of the nodes
• Extended logging system

Authors

Igor Dolzhikov

Contributors

All the contributors are welcome. If you would like to be the contributor please accept some rules.

• The pull requests will be accepted only in "develop" branch
• All modifications or additions should be tested

Thank you for your understanding!

License

MIT License

Documentation

Overview

Package spawn used as a HTTP REST sync service, that makes clustering mode simpler and easier for most of applications.

What's the idea? There are several applications, which are developed to provide their service through HTTP Rest API. But you have no idea how to provide failover processing and clustering mode for these applications, because they are not compatible with, etc. And here we go. The Spawn service will make this job instead of the whole bunch of services that must be configured and communicating each other.

Spawn Sync Service

Index

• Constants
• type HealthCheck
• type Metrics
• type MetricsBandle
  • func (bundle *MetricsBandle) SetMetrics(id, metricType, method string)
• type Node
• type NodeBundle
  • func (bundle *NodeBundle) CurrentFromRing() (Node, bool)
  • func (bundle *NodeBundle) Delete(host string, port uint64) bool
  • func (bundle *NodeBundle) DeleteAll()
  • func (bundle *NodeBundle) DeleteAllByHost(host string) bool
  • func (bundle *NodeBundle) Get(host string, port uint64) (node Node, ok bool)
  • func (bundle *NodeBundle) GetAll() (nodes []Node, total int)
  • func (bundle *NodeBundle) GetAllByHost(host string) (nodes []Node, total int)
  • func (bundle *NodeBundle) InitRing()
  • func (bundle *NodeBundle) Set(node *Node) bool
  • func (bundle *NodeBundle) SetAll(nodes []Node) bool
  • func (bundle *NodeBundle) TwistRing()
• type Server
  • func NewServer(name string) (*Server, error)
  • func (server *Server) RoundTrip(request *http.Request) (*http.Response, error)
  • func (server *Server) Run(hostPort, apiHostPort string, transport http.RoundTripper, nodes []Node, ...) (status string, err error)
  • func (server *Server) Shutdown() (status string, err error)

Constants

const (

	// VERSION - current version of the service
	VERSION = "0.3.5"

	// DATE - revision date of the service
	DATE = "2016-04-24T01:27:17Z"

	// MaxSignals - maximum count of update signals
	MaxSignals = 1000

	// MaxJobs - maximum count of update jobs for every bundle
	MaxJobs = 100000

	// DefaultTimeout is a timeout for the worker's response
	DefaultTimeout time.Duration = 10
)

Types

type HealthCheck

type HealthCheck struct {

	// health check time of the node in seconds
	Seconds time.Duration `json:"seconds"`

	// url which will be checked
	URL string `json:"url"`

	// regexp pattern for extended check analyze
	Pattern string `json:"regexp"`
}

HealthCheck contains parameters which used for checking node

type Metrics

type Metrics struct {
	Success struct {
		Get    uint64 `json:"get"`
		Set    uint64 `json:"set"`
		Delete uint64 `json:"delete"`
	} `json:"success"`
	Failure struct {
		Get    uint64 `json:"get"`
		Set    uint64 `json:"set"`
		Delete uint64 `json:"delete"`
	} `json:"failure"`
	Queued struct {
		Get    uint64 `json:"get"`
		Set    uint64 `json:"set"`
		Delete uint64 `json:"delete"`
	} `json:"queued"`
}

type MetricsBandle

type MetricsBandle struct {
	*Server
	// contains filtered or unexported fields
}

MetricsBandle contains an embedded server link and Node records

func (*MetricsBandle) SetMetrics

func (bundle *MetricsBandle) SetMetrics(id, metricType, method string)

type Node

type Node struct {
	Host        string `json:"host"`
	Port        uint64 `json:"port"`
	Priority    int    `json:"priority"`
	Active      bool   `json:"active"`
	Maintenance bool   `json:"maintenance"`
}

Node contains the node parameters:

- Host is the host name or IP,

- Port is the port number,

• Priority defines a sequence, which will be operating according to attribute of the priority. A example of sorted values by priority - from highest to lowest (1,2,3,0,0,0,-1,-2,-3) the priority '0' has neutral priority value between high and low,

• Active defines status of the node. If it will be set to false, the queue which related with the node will be deleted or it will be created otherwise

• Maintenance mode is using to stop the worker and accumulate updates in the queue. If maintenance mode set to false all updates will posted in the node.

type NodeBundle

type NodeBundle struct {
	*Server
	// contains filtered or unexported fields
}

NodeBundle contains an embedded server link and Node records

func (*NodeBundle) CurrentFromRing

func (bundle *NodeBundle) CurrentFromRing() (Node, bool)

CurrentFromRing gets a current Node from the the ring ('round-robin')

func (*NodeBundle) Delete

func (bundle *NodeBundle) Delete(host string, port uint64) bool

Delete one of the node record specified by host and port

func (*NodeBundle) DeleteAll

func (bundle *NodeBundle) DeleteAll()

DeleteAll - deletes all the nodes records

func (*NodeBundle) DeleteAllByHost

func (bundle *NodeBundle) DeleteAllByHost(host string) bool

DeleteAllByHost - deletes all the nodes records specified by host

func (*NodeBundle) Get

func (bundle *NodeBundle) Get(host string, port uint64) (node Node, ok bool)

Get - gets one of the node record specified by host and port

func (*NodeBundle) GetAll

func (bundle *NodeBundle) GetAll() (nodes []Node, total int)

GetAll - gets all the nodes records sorted according to priority

func (*NodeBundle) GetAllByHost

func (bundle *NodeBundle) GetAllByHost(host string) (nodes []Node, total int)

GetAllByHost - gets all the nodes records specified by host and sorted according to priority

func (*NodeBundle) InitRing

func (bundle *NodeBundle) InitRing()

InitRing - inits the nodes in the ring ('round-robin') and resets a pointer to the node

func (*NodeBundle) Set

func (bundle *NodeBundle) Set(node *Node) bool

Set - updates the node record or create one if it does not exist

func (*NodeBundle) SetAll

func (bundle *NodeBundle) SetAll(nodes []Node) bool

SetAll - updates all the nodes records or create them if records do not exist

func (*NodeBundle) TwistRing

func (bundle *NodeBundle) TwistRing()

TwistRing - sets a pointer to the next node from the ring

type Server

type Server struct {

	// Server name/description
	Name string

	// Embeded router
	*router.Router

	// Node Bundle contains the Node records
	Nodes *NodeBundle

	// Metrics Bundle contains the Metrics records
	Metrics *MetricsBandle
	// contains filtered or unexported fields
}

Server Record

func NewServer

func NewServer(name string) (*Server, error)

NewServer creates a new server which contains the nodes/queues

func (*Server) RoundTrip

func (server *Server) RoundTrip(request *http.Request) (*http.Response, error)

RoundTrip manages all requests/responses

func (*Server) Run

func (server *Server) Run(
	hostPort, apiHostPort string,
	transport http.RoundTripper,
	nodes []Node,
	roundRobin, byPriority bool,
	check HealthCheck,
	authService auth.Auth,
) (status string, err error)

Run the server, init the handlers, init the specified modes. If transport http.RoundTripper is not defined will be used default transport. http.RoundTripper contains callback function which handle all incoming requests and get responses/errors

func (*Server) Shutdown

func (server *Server) Shutdown() (status string, err error)

Shutdown closes the server graceful