gossip

Documentation

Overview

Package gossip implements a protocol for sharing information between Cockroach nodes using an ad-hoc, peer-to-peer network. The self-assembled network aims to minimize time for new information to reach each node, and minimize network traffic required.

Gossiped information is identified by key. Gossip information is captured by info objects. Info objects may be stored individually (e.g. the number of nodes in the system), or may be organized into groups (e.g. multiple values of the same type from different originators).

Groups organize multiple instance of info for the same key prefix. Groups come in two types: MinGroup groups keep only the minimum values seen; MaxGroup groups keep only the maximum values seen. An example is load or disk capacity values for nodes. In a cluster with thousands of nodes, groups force the gossip network to limit itself to only a portion of total data volume (e.g. the 100 least loaded nodes or the 100 disks with most unused capacity).

Single-valued info values can have any type. Values to be used with groups must either be of type int64, float64, string or implement the util.Ordered interface.

A map of info objects and a map of Group objects are kept by a Gossip instance. Single-valued info objects can be added via Gossip.AddInfo(). Groups must be registered via Gossip.RegisterGroup(). Info objects are added to groups if their key matches. Info can be queried for single-valued keys via Gossip.GetInfo. Sorted values for groups are queried via Gossip.GetGroupInfos().

Index

• Constants
• Variables
• func MakeNodeIDGossipKey(nodeID int32) string
• func SimulateNetwork(nodeCount int, network string, gossipInterval time.Duration, ...)
• type Gossip
  • func New() *Gossip
  • func (g *Gossip) AddInfo(key string, val interface{}, ttl time.Duration) error
  • func (g *Gossip) GetGroupInfos(prefix string) ([]interface{}, error)
  • func (g *Gossip) GetInfo(key string) (interface{}, error)
  • func (s Gossip) Gossip(args *GossipRequest, reply *GossipResponse) error
  • func (g *Gossip) Incoming() []net.Addr
  • func (g *Gossip) MaxHops() uint32
  • func (g *Gossip) Outgoing() []net.Addr
  • func (g *Gossip) RegisterGroup(prefix string, limit int, typeOf GroupType) error
  • func (g *Gossip) SetBootstrap(bootstraps []net.Addr)
  • func (g *Gossip) SetInterval(interval time.Duration)
  • func (g *Gossip) Start(rpcServer *rpc.Server)
  • func (g *Gossip) Stop() <-chan error
• type GossipRequest
• type GossipResponse
• type GroupType

Constants

const (
	// KeyClusterID is the unique UUID for this Cockroach cluster.
	// The value is a string UUID for the cluster.
	KeyClusterID = "cluster-id"

	// KeyConfigAccounting is the accounting configuration map.
	KeyConfigAccounting = "accounting"

	// KeyConfigPermission is the permission configuration map.
	KeyConfigPermission = "permissions"

	// KeyConfigZone is the zone configuration map.
	KeyConfigZone = "zones"

	// KeyMaxAvailCapacityPrefix is the key prefix for gossiping available
	// store capacity. The suffix is composed of:
	// <datacenter>-<hex node ID>-<hex store ID>. The value is a
	// storage.StoreDescriptor struct.
	KeyMaxAvailCapacityPrefix = "max-avail-capacity-"

	// KeyNodeCount is the count of gossip nodes in the network.  The
	// value is an int64 containing the count of nodes in the cluster.
	// TODO(spencer): should remove this and instead just count the
	//   number of node ids being gossiped.
	KeyNodeCount = "node-count"

	// KeyNodeIDPrefix is the key prefix for gossiping node id
	// addresses. The actual key is suffixed with the hexadecimal
	// representation of the node id and the value is the host:port
	// string address of the node. E.g. node-1bfa: fwd56.sjcb1:24001
	KeyNodeIDPrefix = "node-"

	// KeySentinel is a key for gossip which must not expire or else the
	// node considers itself partitioned and will retry with bootstrap hosts.
	KeySentinel = KeyClusterID

	// KeyFirstRangeMetadata is the metadata for the "first" range. The
	// "first" range contains the meta1 key range, the first level of
	// the bi-level key addressing scheme. The value is a slice of
	// storage.Replica structs.
	KeyFirstRangeMetadata = "first-range"
)

Constants for gossip keys.

const (
	// MaxPeers is the maximum number of connected gossip peers.
	MaxPeers = 10
)

Variables

var (
	// GossipBootstrap is a comma-separated list of node addresses which
	// active as bootstrap hosts for connecting to the gossip network.
	GossipBootstrap = flag.String(
		"gossip", "",
		"addresses (comma-separated host:port pairs) of node addresses for gossip bootstrap")
	// GossipInterval is a time interval specifying how often gossip is
	// communicated between hosts on the gossip network.
	GossipInterval = flag.Duration(
		"gossip_interval", 2*time.Second,
		"approximate interval (time.Duration) for gossiping new information to peers")
)

Functions

func MakeNodeIDGossipKey

func MakeNodeIDGossipKey(nodeID int32) string

MakeNodeIDGossipKey returns the gossip key for node ID info.

func SimulateNetwork

func SimulateNetwork(nodeCount int, network string, gossipInterval time.Duration,
	simCallback func(cycle int, nodes map[string]*Gossip) bool)

SimulateNetwork creates nodeCount gossip nodes. The network should be set to either "tcp" or "unix". The gossipInterval should be set to a compressed simulation timescale, though large enough to give the concurrent goroutines enough time to pass data back and forth in order to yield accurate estimates of how old data actually ends up being at the various nodes. After each gossipInterval period, simCallback is invoked; when it returns false, the simulation ends. If it returns true, the simulation continues another cycle.

Node0 gossips the node count as well as the gossip sentinel. The gossip bootstrap hosts are set to the first three nodes (or fewer if less than three are available).

At each cycle of the simulation, node 0 gossips the sentinel. If the simulation requires other nodes to gossip, this should be done via simCallback.

The simulation callback receives a map of nodes, keyed by node address.

Types

type Gossip

type Gossip struct {
	Name      string        // Optional node name
	Connected chan struct{} // Closed upon initial connection
	// contains filtered or unexported fields
}

Gossip is an instance of a gossip node. It embeds a gossip server. During bootstrapping, the bootstrap list contains candidates for entry to the gossip network.

func New

func New() *Gossip

New creates an instance of a gossip node.

func (*Gossip) AddInfo

func (g *Gossip) AddInfo(key string, val interface{}, ttl time.Duration) error

AddInfo adds or updates an info object. Returns an error if info couldn't be added.

func (*Gossip) GetGroupInfos

func (g *Gossip) GetGroupInfos(prefix string) ([]interface{}, error)

GetGroupInfos returns a slice of info values from specified group, or an error if group is not registered.

func (*Gossip) GetInfo

func (g *Gossip) GetInfo(key string) (interface{}, error)

GetInfo returns an info value by key or an error if specified key does not exist or has expired.

func (Gossip) Gossip

func (s Gossip) Gossip(args *GossipRequest, reply *GossipResponse) error

Gossip receives gossipped information from a peer node. The received delta is combined with the infostore, and this node's own gossip is returned to requesting client.

func (*Gossip) Incoming

func (g *Gossip) Incoming() []net.Addr

Incoming returns a slice of incoming gossip client connection addresses.

func (*Gossip) MaxHops

func (g *Gossip) MaxHops() uint32

MaxHops returns the maximum number of hops to reach the furthest gossiped information currently in the network.

func (*Gossip) Outgoing

func (g *Gossip) Outgoing() []net.Addr

Outgoing returns a slice of outgoing gossip client connection addresses. Note that these outgoing client connections may not actually be legitimately connected. They may be in the process of trying, or may already have failed, but haven't yet been processed by the gossip instance.

func (*Gossip) RegisterGroup

func (g *Gossip) RegisterGroup(prefix string, limit int, typeOf GroupType) error

RegisterGroup registers a new group with info store. Returns an error if the group was already registered.

func (*Gossip) SetBootstrap

func (g *Gossip) SetBootstrap(bootstraps []net.Addr)

SetBootstrap initializes the set of gossip node addresses used to bootstrap the gossip network.

func (*Gossip) SetInterval

func (g *Gossip) SetInterval(interval time.Duration)

SetInterval sets the interval at which fresh info is gossiped to incoming gossip clients.

func (*Gossip) Start

func (g *Gossip) Start(rpcServer *rpc.Server)

Start launches the gossip instance, which commences joining the gossip network using the supplied rpc server and the gossip bootstrap addresses specified via command-line flag: -gossip.

This method starts bootstrap loop, gossip server, and client management in separate goroutines and returns.

func (*Gossip) Stop

func (g *Gossip) Stop() <-chan error

Stop shuts down the gossip server. Returns a channel which signals exit once all outgoing clients are closed and the management loop for the gossip instance is finished.

type GossipRequest

type GossipRequest struct {
	Addr   net.Addr // Address of requesting node's server
	LAddr  net.Addr // Local address of client on requesting node
	MaxSeq int64    // Maximum sequence number of gossip from this peer

	Delta *infoStore // Reciprocal delta of new info since last gossip
}

GossipRequest is passed to the Gossip.Gossip RPC.

type GossipResponse

type GossipResponse struct {
	Delta     *infoStore // Requested delta of server's infostore
	Alternate net.Addr   // Non-nil means client should retry with this address
}

GossipResponse is returned from the Gossip.Gossip RPC. Delta will be nil in the event that Alternate is set.

type GroupType

type GroupType int

GroupType indicates the bounds of the values encountered within the group.

const (
	// MinGroup maintains minimum values for keys matching group prefix.
	MinGroup GroupType = iota
	// MaxGroup maintains maximum values for keys matching group prefix.
	MaxGroup
)