dragonboat

package module
v2.1.0+incompatible Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 20, 2019 License: Apache-2.0 Imports: 39 Imported by: 0

README

dragonboat

Dragonboat - A Multi-Group Raft library in Go / 中文版

license Build status travis-ci Go Report Card Godoc

About

Dragonboat is a high performance multi-group Raft consensus library in Go with C++11 binding support.

Consensus algorithms such as Raft provides fault-tolerance by alllowing a system continue to operate as long as the majority member servers are available. For example, a Raft cluster of 5 servers can make progress even if 2 servers fail. It also appears to clients as a single node with strong data consistency always provided. All running servers can be used to initiate read requests for aggregated read throughput.

Dragonboat handles all technical difficulties associated with Raft to allow users to just focus on their application domains. It is also very easy to use, our step-by-step examples can help new users to master it in half an hour.

Features

  • Easy to use API for building Raft based applications in Go or C++
  • Feature complete and scalable multi-group Raft implementation
  • Fully pipelined and TLS mutual authentication support, ready for high latency open environment
  • Custom Raft log storage and Raft RPC support, easy to integrate with latest I/O techs
  • Optional Drummer server component for managing large number of Raft groups with high availability
  • Extensively tested including using Jepsen's Knossos linearizability checker, some results are here

Performance

Dragonboat is the fastest open source multi-group Raft implementation on Github.

For 3-nodes system using mid-range hardware, e.g. 22 cores Intel Xeon at 2.8Ghz and enterprise NVME SSD (details here), Dragonboat can sustain at 9 million writes per second when the payload is 16bytes each or 11 million mixed I/O per second at 9:1 read:write ratio. High throughput is maintained in geographically distributed environment. When the RTT between nodes is 30ms, 2 million I/O per second can still be achieved using a much larger number of clients. throughput

The number of concurrent active Raft groups affects the overall throughput as requests become harder to be batched. On the other hand, having thousands of idle Raft groups has a much smaller impact on throughput. nodes

Table below shows write latencies in millisecond, Dragonboat has <5ms P99 write latency when handling 8 million writes per second at 16 bytes each. Read latency is lower than writes as the ReadIndex protocol employed for linearizable reads doesn't require fsync-ed disk I/O.

Ops Payload Size 99.9% percentile 99% percentile AVG
1m 16 2.24 1.19 0.79
1m 128 11.11 1.37 0.92
1m 1024 71.61 25.91 3.75
5m 16 4.64 1.95 1.16
5m 128 36.61 6.55 1.96
8m 16 12.01 4.65 2.13

When tested on a single Raft group, Dragonboat can sustain writes at 1.25 million per second when payload is 16 bytes each, average latency is 1.3ms and the P99 latency is 2.6ms. This is achieved when using an average of 3 cores (2.8GHz) on each server.

As visualized below, Stop-the-World pauses caused by Golang's GC are sub-millisecond on highly loaded systems. Such very short Stop-the-World pause time is set to be further significantly reduced in the coming Go 1.12 release. Golang's runtime.ReadMemStats reports that less than 1% of the available CPU time is used by GC on highly loaded system. stw

Requirements

  • x86_64 Linux or MacOS, Go 1.10 and 1.11, GCC or Clang with C++11 support
  • RocksDB 5.13.4 or above when using RocksDB for storing Raft logs

Getting Started

Note that, steps below use code from the Master branch. Master is our unstable branch for development. Please use released versions for any production purposes.

To download Dragonboat to your Go workspace:

$ go get -u -d github.com/lni/dragonboat

You need to decide whether to use RocksDB or LevelDB to store Raft logs. RocksDB is recommended.

RocksDB

If RocksDB 5.13.4 or above has not been installed, use the following commands to install RocksDB 5.13.4 to /usr/local/lib and /usr/local/include.

$ cd $GOPATH/src/github.com/lni/dragonboat
$ make install-rocksdb-ull

Run built-in tests to check the installation:

$ cd $GOPATH/src/github.com/lni/dragonboat
$ make dragonboat-test

To build your application

CGO_CFLAGS="-I/path/to/rocksdb/include" CGO_LDFLAGS="-L/path/to/rocksdb/lib -lrocksdb" go build -v pkgname
LevelDB

Nothing need to be installed when using LevelDB based Raft Log storage.

To run built-in tests using LevelDB based storage:

$ cd $GOPATH/src/github.com/lni/dragonboat
$ DRAGONBOAT_LOGDB=leveldb make dragonboat-test

To build the your application using LevelDB based Raft log storage

go build -v -tags="dragonboat_leveldb" pkgname
Optional - C++ binding

The C++ binding is only required when you want to use Dragonboat in your C++ project. To install the C++ binding:

$ cd $GOPATH/src/github.com/lni/dragonboat
$ make binding
$ sudo make install-binding

Run C++ binding tests (gtest is required):

$ cd $GOPATH/src/github.com/lni/dragonboat
$ make clean
$ make test-cppwrapper

FAQ, docs, step-by-step examples and online chat are available.

Examples

Dragonboat examples are here.

Status

Dragonboat is production ready.

Contributing

For reporting bugs, please open an issue. For contributing improvements or new features, please send in the pull request.

License

Dragonboat is licensed under the Apache License Version 2.0. See LICENSE for details.

Third party code used in Dragonboat and their licenses is summarized here.

Documentation

Overview

Package dragonboat is a multi-group Raft implementation.

The NodeHost struct is the facade interface for all features provided by the dragonboat package. Each NodeHost instance, identified by its RaftAddress property, usually runs on a separate host managing its CPU, storage and network resources. Each NodeHost can manage Raft nodes from many different Raft groups known as Raft clusters. Each Raft cluster is identified by its ClusterID Each Raft cluster usually consists of multiple nodes, identified by their NodeID values. Nodes from the same Raft cluster are suppose to be distributed on different NodeHost instances across the network, this brings fault tolerance to node failures as application data stored in such a Raft cluster can be available as long as the majority of its managing NodeHost instances (i.e. its underlying hosts) are available.

User applications can leverage the power of the Raft protocol implemented in dragonboat by implementing its IStateMachine component. IStateMachine is defined in github.com/lni/dragonboat/statemachine. Each cluster node is associated with an IStateMachine instance, it is in charge of updating, querying and snapshotting application data, with minimum exposure to the complexity of the Raft protocol implementation.

User applications can use NodeHost's APIs to update the state of their IStateMachine instances, this is called making proposals. Once accepted by the majority nodes of a Raft cluster, the proposal is considered as committed and it will be applied on all member nodes of the Raft cluster. Applications can also make linearizable reads to query the state of their IStateMachine instances. Dragonboat employs the ReadIndex protocol invented by Diego Ongaro to implement linearizable reads. Both read and write operations can be initiated on any member nodes, although initiating from the leader nodes incurs the lowest overhead.

Dragonboat guarantees the linearizability of your I/O when interacting with the IStateMachine. In plain English, writes (via making proposal) to your Raft cluster appears to be instantaneous, once a write is completed, all later reads (linearizable read using the ReadIndex protocol as implemented and provided in dragonboat) should return the value of that write or a later write. Once a value is returned by a linearizable read, all later reads should return the same value or the result of a later write.

To strictly provide such guarantee, we need to implement the at-most-once semantic required by linearizability. For a client, when it retries the proposal that failed to complete before its deadline during the previous attempt, it has the risk to have the same proposal committed and applied twice into the IStateMachine. Dragonboat prevents this by implementing the client session concept described in Diego Ongaro's PhD thesis.

Dragonboat is a feature complete Multi-Group Raft implementation - snapshotting, membership change, leadership transfer and non-voting members are also provided.

Dragonboat is also extensively optimized. The Raft protocol implementation is fully pipelined, meaning proposals can start before the completion of previous proposals. This is critical for system throughput in high latency environment. Dragonboat is also fully batched, it batches internal operations whenever possible to maximize system throughput.

Index

Examples

Constants

View Source 
const (

	// DragonboatMajor is the major version number
	DragonboatMajor = 2
	// DragonboatMinor is the minor version number
	DragonboatMinor = 1
	// DragonboatPatch is the patch version number
	DragonboatPatch = 0
	// DEVVersion is a boolean flag indicating whether this is a dev version
	DEVVersion = false
)

Variables

View Source 
var (
	// ErrClusterNotFound indicates that the specified cluster is not found.
	ErrClusterNotFound = errors.New("cluster not found")
	// ErrClusterAlreadyExist indicates that the specified cluster already exist.
	ErrClusterAlreadyExist = errors.New("cluster already exist")
	// ErrInvalidClusterSettings indicates that cluster settings specified for
	// the StartCluster method are invalid.
	ErrInvalidClusterSettings = errors.New("cluster settings are invalid")
	// ErrDeadlineNotSet indicates that the context parameter provided does not
	// carry a deadline.
	ErrDeadlineNotSet = errors.New("deadline not set")
	// ErrInvalidDeadline indicates that the specified deadline is invalid, e.g.
	// time in the past.
	ErrInvalidDeadline = errors.New("invalid deadline")
)
View Source 
var (
	// ErrInvalidSession indicates that the specified client session is invalid.
	ErrInvalidSession = errors.New("invalid session")
	// ErrTimeoutTooSmall indicates that the specified timeout value is too small.
	ErrTimeoutTooSmall = errors.New("specified timeout value is too small")
	// ErrPayloadTooBig indicates that the payload is too big.
	ErrPayloadTooBig = errors.New("payload is too big")
	// ErrSystemBusy indicates that the system is too busy to handle the request.
	ErrSystemBusy = errors.New("system is too busy try again later")
	// ErrClusterClosed indicates that the requested cluster is being shut down.
	ErrClusterClosed = errors.New("raft cluster already closed")
	// ErrBadKey indicates that the key is bad, retry the request is recommended.
	ErrBadKey = errors.New("bad key try again later")
	// ErrPendingConfigChangeExist indicates that there is already a pending
	// membership change exist in the system.
	ErrPendingConfigChangeExist = errors.New("pending config change request exist")
	// ErrTimeout indicates that the operation timed out.
	ErrTimeout = errors.New("timeout")
	// ErrSystemStopped indicates that the system is being shut down.
	ErrSystemStopped = errors.New("system stopped")
	// ErrCanceled indicates that the request has been canceled.
	ErrCanceled = errors.New("request canceled")
	// ErrRejected indicates that the request has been rejected.
	ErrRejected = errors.New("request rejected")
)
View Source 
var (
	// ErrNoSnapshot is the error used to indicate that there is no snapshot
	// available.
	ErrNoSnapshot = errors.New("no snapshot available")
)
View Source 
var (
	// NodeHostInfoReportSecond defines how often NodeHost reports its info to
	// optional Master servers. It is defined in the number of seconds.
	NodeHostInfoReportSecond uint64 = settings.Soft.NodeHostInfoReportSecond
)

Functions

func IsTempError 

func IsTempError(err error) bool

IsTempError returns a boolean value indicating whether the specified error is a temporary error that worth to be retried later with the exact same input, potentially on a more suitable NodeHost instance.

Types

type ClusterInfo 

type ClusterInfo struct {
	// ClusterID is the cluster ID of the Raft cluster node
	ClusterID uint64
	// NodeID is the node ID of the Raft cluster node
	NodeID uint64
	// IsLeader indicates whether this is a leader node
	IsLeader bool
	// Nodes is a map of member node IDs to Raft addresses
	Nodes map[uint64]string
	// ConfigChangeIndex is the current config change index of the Raft node.
	// ConfigChangeIndex is increased each time when a new membership change
	// is applied
	ConfigChangeIndex uint64
	// Pending is a boolean flag indicating whether details of the cluster
	// node is still not available. The Pending flag is set to true usually
	// because the node has not had anything applied
	Pending bool
	// Incomplete is a boolean flag indicating whether the ClusterInfo record
	// has the Nodes map intentionally omitted to save bandwidth
	Incomplete bool
}

ClusterInfo provides Raft cluster details. The optional Master client periodically sends a list of ClusterInfo to master servers to notify master the details of all Raft clusters available on each NodeHost instance. ClusterInfo is an optional struct used by Master servers.

type ICompleteHandler 

type ICompleteHandler interface {
	Notify(RequestResult)
	Release()
}

ICompleteHandler is a handler interface that will be invoked when the request in completed. This interface is used by the language bindings, applications are not expected to directly use this interface.

type IMasterClient 

type IMasterClient interface {
	// Name returns the unique type name of the IMasterClient.
	Name() string
	// Stop stops the MasterClient instance.
	Stop()
	// GetDeploymentID returns the deployment ID from master servers.
	// Each unique multi-raft setup managed by Master servers is identified by a
	// uint64 deployment ID, Raft nodes with different deployment ID won't be able
	// to communicate with each other. It a fool-proof feature helping to prevent
	// corruption to your Raft log data by accidentally mis-configtured NodeHosts.
	// GetDeploymentID is called shortly after the launch of the NodeHost instance,
	// all Raft nodes managed by that NodeHost will be associated with the
	// deployment ID returned by GetDeploymentID. Master server is required to
	// always return the same the Deployment ID value during its entire life cycle.
	// Note that the deployment ID concept is optional, you can return a constant
	// uint64 value if you don't want to actually use this fool-proof this
	// feature.
	GetDeploymentID(ctx context.Context, url string) (uint64, error)
	// SendNodeHostInfo is called by NodeHost periodically to notify Master
	// servers the details of those Raft clusters managed by the NodeHost. This
	// also gives the IMasterClient an opportunity to receive requests from Master
	// servers to restore, repair or re-balance Raft clusters. Such received
	// requests should be internally queued to be handled periodically when
	// HandleMasterRequests is called.
	SendNodeHostInfo(ctx context.Context, url string, nhi NodeHostInfo) error
	// HandleMasterRequests is called by NodeHost periodically to handle pending
	// master requests received and queued when the SendNodeHostInfo method is
	// called.
	HandleMasterRequests(ctx context.Context) error
}

IMasterClient is the interface to be implemented for connecting NodeHosts with Master servers. Note both Master server and IMasterClient are optional.

Consider a typical deployment in which you have say dozens of NodeHost instances spawned across many servers, each hosting hundreds of Raft nodes. To ensure availability, you have to assume that some NodeHost instances can fail at certain time. Your system need to be able to automatically react to such failures so affected Raft groups can continue to have the quorums available. Master is a simple concept that can be used to help providing such capabilities.

In a Master server implementation, NodeHosts periodically contact the Master servers via their IMasterClient clients to report their own availability and the details of those managed Raft nodes. The NodeHost itself and all Raft nodes managed by that NodeHost are considered as dead once the NodeHost instance fails to contact the Master for a defined period of time. The Master servers then request other NodeHosts to spawn new Raft nodes to replace the unavailable ones. Master servers are expected to have its own mechanisms to ensure its own high availability.

In a more advanced design, you can also implement features such as moving certain Raft nodes to selected NodeHost instances to re-balance load.

Our Drummer server, available at github.com/lni/dragonboat/drummer, is a Master server reference implementation extensively used in our tests. Its IMasterClient client, available at github.com/lni/dragonboat/drummer/client/nodehost.go is a reference IMasterClient implementation that interacts with the Drummer server.

type INodeUser 

type INodeUser interface {
	// Propose starts an asynchronous proposal on the Raft cluster represented by
	// the INodeUser instance. Its semantics is the same as the Propose() method
	// in NodeHost.
	Propose(s *client.Session,
		cmd []byte, timeout time.Duration) (*RequestState, error)
	// ReadIndex starts the asynchronous ReadIndex protocol used for linearizable
	// reads on the Raft cluster represented by the INodeUser instance. Its
	// semantics is the same as the ReadIndex() method in NodeHost.
	ReadIndex(timeout time.Duration) (*RequestState, error)
}

INodeUser is the interface implemented by a Raft node user type. A Raft node user can be used to directly initiate proposals or read index operations without locating the Raft node in NodeHost's node list first. It is useful when doing bulk load operations on selected clusters.

type MasterClientFactoryFunc 

type MasterClientFactoryFunc func(*NodeHost) IMasterClient

MasterClientFactoryFunc is the factory function for creating a new IMasterClient instance.

type Membership 

type Membership struct {
	// ConfigChangeID is the Raft entry index of the last applied membership
	// change entry.
	ConfigChangeID uint64
	// Nodes is a map of NodeID values to NodeHost Raft addresses for all regular
	// Raft nodes.
	Nodes map[uint64]string
	// Observers is a map of NodeID values to NodeHost Raft addresses for all
	// observers.
	Observers map[uint64]string
	// Removed is a set of NodeID values that have been removed from the Raft
	// cluster. They are not allowed to be added back to the cluster.
	Removed map[uint64]struct{}
}

Membership is the struct used to describe Raft cluster membership query results.

type NodeHost 

type NodeHost struct {
	// contains filtered or unexported fields
}

NodeHost manages Raft clusters and enables them to share resources such as transport and persistent storage etc. NodeHost is also the central access point for Dragonboat functionalities provided to applications.

func NewNodeHost 

func NewNodeHost(nhConfig config.NodeHostConfig) *NodeHost

NewNodeHost creates a new NodeHost instance. The returned NodeHost instance is configured using the specified NodeHostConfig instance. In a typical application, it is expected to have one NodeHost on each server.

Example 
// Let's say we want to put all LogDB's WAL data in a directory named wal,
// everything else is stored in a directory named dragonboat. Assume the
// RTT between nodes is 200 milliseconds, and the nodehost address is
// myhostname:5012
nhc := config.NodeHostConfig{
	WALDir:         "wal",
	NodeHostDir:    "dragonboat",
	RTTMillisecond: 200,
	RaftAddress:    "myhostname:5012",
}
// Creates a nodehost instance using the above NodeHostConfig instnace.
nh := NewNodeHost(nhc)
log.Printf("nodehost created, running on %s", nh.RaftAddress())

Output:

func NewNodeHostWithMasterClientFactory 

func NewNodeHostWithMasterClientFactory(nhConfig config.NodeHostConfig,
	factory MasterClientFactoryFunc) *NodeHost

NewNodeHostWithMasterClientFactory creates a new NodeHost instance and uses the specified MasterClientFactoryFunc function to create the optional master client. Master server details should be provided in the MasterServers field in nhConfig. Note that Master Client and Master Servers are both optional.

func (*NodeHost) CloseSession 

func (nh *NodeHost) CloseSession(ctx context.Context,
	session *client.Session) error

CloseSession closes the specified client session by unregistering it from the system. This is a synchronous method meaning it will only return after its confirmed completion, failure or timeout.

Closed client session should no longer be used in future proposals.

func (*NodeHost) GetClusterMembership 

func (nh *NodeHost) GetClusterMembership(ctx context.Context,
	clusterID uint64) (*Membership, error)

GetClusterMembership returns the membership information from the specified Raft cluster. This method guarantees that the returned membership information is linearizable. This is a synchronous method meaning it will only return after its confirmed completion, failure or timeout.

func (*NodeHost) GetLeaderID 

func (nh *NodeHost) GetLeaderID(clusterID uint64) (uint64, bool, error)

GetLeaderID returns the leader node ID of the specified Raft cluster based on local node's knowledge. The returned boolean value indicates whether the leader information is available.

func (*NodeHost) GetNewSession 

func (nh *NodeHost) GetNewSession(ctx context.Context,
	clusterID uint64) (*client.Session, error)

GetNewSession starts an synchronous proposal to create, register and return a new client session object. A client session object is used to ensure that a retried proposal, e.g. proposal retried after timeout, will not be applied more than once into the IStateMachine.

Returned client session instance should not be used concurrently. Use multiple client sessions when you need to concurrently start multiple proposals.

func (*NodeHost) GetNoOPSession 

func (nh *NodeHost) GetNoOPSession(clusterID uint64) *client.Session

GetNoOPSession returns a NO-OP client session ready to be used for making proposals. The NO-OP client session is a dummy client session that will not be checked or enforced. Use this No-OP client session when you want to ignore features provided by client sessions. A NO-OP client session is not registered on the server side and thus not required to be closed at the end of its life cycle.

Returned NO-OP client session instance can be concurrently used in multiple goroutines.

Use this NO-OP client session when your IStateMachine provides idempotence in its own implementation.

func (*NodeHost) GetNodeUser 

func (nh *NodeHost) GetNodeUser(clusterID uint64) (INodeUser, error)

GetNodeUser returns an INodeUser instance ready to be used to directly make proposals or read index operations without locating the node repeatedly in the NodeHost. A possible use case is when loading a large data set say with billions of proposals into the dragonboat based system.

func (*NodeHost) HasNodeInfo 

func (nh *NodeHost) HasNodeInfo(clusterID uint64, nodeID uint64) bool

HasNodeInfo returns a boolean value indicating whether the specified node has been bootstrapped on the current NodeHost instance.

func (*NodeHost) NodeHostConfig 

func (nh *NodeHost) NodeHostConfig() config.NodeHostConfig

NodeHostConfig returns the NodeHostConfig instance used for configuring this NodeHost instance.

func (*NodeHost) Propose 

func (nh *NodeHost) Propose(session *client.Session, cmd []byte,
	timeout time.Duration) (*RequestState, error)

Propose starts an asynchronous proposal on the Raft cluster specified in the Session object. The input byte slice can be reused for other purposes immediate after the return of this method.

This method returns a RequestState instance or an error immediately. Application can wait on the CompleteC member channel of the returned RequestState instance to get notified for the outcome of the proposal and access to the result of the proposal.

After the proposal is completed, i.e. RequestResult is received from the CompletedC channel of the returned RequestState, unless NO-OP client session is used, it is caller's responsibility to update the Session instance accordingly based on the RequestResult.Code value. Basically, when RequestTimeout is returned, you can retry the same proposal without updating your client session instance, when a RequestRejected value is returned, it usually means the session instance has been evicted from the server side, the Raft paper recommends you to crash your client in this highly unlikely event. When the proposal completed successfully with a RequestCompleted value, application must call client.ProposalCompleted() to get the client session ready to be used in future proposals.

func (*NodeHost) ProposeSession 

func (nh *NodeHost) ProposeSession(session *client.Session,
	timeout time.Duration) (*RequestState, error)

ProposeSession starts an asynchronous proposal on the specified cluster for client session related operations. Depending on the state of the client session object, the supported operations are for registering or unregistering a client session. Application can select on the CompleteC member channel of the returned RequestState instance to get notified for the completion and result of the proposal.

func (*NodeHost) RaftAddress 

func (nh *NodeHost) RaftAddress() string

RaftAddress returns the Raft address of the NodeHost instance. The returned RaftAddress value is used to identify this NodeHost instance. It is also the address used for exchanging Raft messages and snapshots between distributed NodeHost instances.

func (*NodeHost) ReadIndex 

func (nh *NodeHost) ReadIndex(clusterID uint64,
	timeout time.Duration) (*RequestState, error)

ReadIndex starts the asynchronous ReadIndex protocol used for linearizable read on the specified cluster. This method returns a RequestState instance or an error immediately. Application should wait on the CompleteC channel of the returned RequestState object to get notified on the outcome of the ReadIndex operation. On a successful completion, the ReadLocal method can then be invoked to query the state of the IStateMachine to complete the read operation with linearizability guarantee.

func (*NodeHost) ReadLocal deprecated 

func (nh *NodeHost) ReadLocal(clusterID uint64,
	query []byte) ([]byte, error)

ReadLocal queries the specified Raft node. To ensure the linearizability of the I/O, ReadLocal should only be called after receiving a RequestCompleted notification from the ReadIndex method.

Deprecated: Applications should use ReadLocalNode instead.

func (*NodeHost) ReadLocalNode 

func (nh *NodeHost) ReadLocalNode(rs *RequestState,
	query []byte) ([]byte, error)

ReadLocalNode queries the Raft node identified by the input RequestState instance. To ensure the IO linearizability, ReadLocalNode should only be called after receiving a RequestCompleted notification from the ReadIndex method. See ReadIndex's example for more details.

func (*NodeHost) RequestAddNode 

func (nh *NodeHost) RequestAddNode(clusterID uint64,
	nodeID uint64, address string, configChangeIndex uint64,
	timeout time.Duration) (*RequestState, error)

RequestAddNode is a Raft cluster membership change method for requesting the specified node to be added to the specified Raft cluster. It starts an asynchronous request to add the node to the Raft cluster membership list. Application can wait on the CompleteC member of the returned RequestState instance to get notified for the outcome.

If there is already an observer with the same nodeID in the cluster, it will be promoted to a regular node with voting power. The address parameter of the RequestAddNode call is ignored when promoting an observer to a regular node.

After the node is successfully added to the Raft cluster, it is application's responsibility to call StartCluster on the right NodeHost instance to actually start the Raft cluster node.

When the raft cluster is created with the OrderedConfigChange config flag set as false, the configChangeIndex parameter is ignored. Otherwise, it should be set to the most recent Config Change Index value returned by the GetClusterMembership method. The requested add node operation will be rejected if other membership change has happened since the last call to GetClusterMembership.

func (*NodeHost) RequestAddObserver 

func (nh *NodeHost) RequestAddObserver(clusterID uint64,
	nodeID uint64, address string, configChangeIndex uint64,
	timeout time.Duration) (*RequestState, error)

RequestAddObserver is a Raft cluster membership change method for requesting the specified node to be added to the specified Raft cluster as an observer without voting power. It starts an asynchronous request to add the specified node as an observer.

Such observer is able to receive replicated states from the leader node, but it is not allowed to vote for leader, it is not considered as a part of the quorum when replicating state. An observer can be promoted to a regular node with voting power by making a RequestAddNode call using its clusterID and nodeID values. An observer can be removed from the cluster by calling RequestDeleteNode with its clusterID and nodeID values.

Application should later call StartCluster with config.Config.IsObserver set to true on the right NodeHost to actually start the observer instance.

When the raft cluster is created with the OrderedConfigChange config flag set as false, the configChangeIndex parameter is ignored. Otherwise, it should be set to the most recent Config Change Index value returned by the GetClusterMembership method. The requested add observer operation will be rejected if other membership change has happened since the last call to GetClusterMembership.

func (*NodeHost) RequestDeleteNode 

func (nh *NodeHost) RequestDeleteNode(clusterID uint64,
	nodeID uint64,
	configChangeIndex uint64, timeout time.Duration) (*RequestState, error)

RequestDeleteNode is a Raft cluster membership change method for requesting the specified node to be removed from the specified Raft cluster. It starts an asynchronous request to remove the node from the Raft cluster membership list. Application can wait on the CompleteC member of the returned RequestState instance to get notified for the outcome.

It is not guaranteed that deleted node will automatically close itself and be removed from its managing NodeHost instance. It is application's responsibility to call RemoveCluster on the right NodeHost instance to actually have the cluster node removed from its managing NodeHost instance.

When the raft cluster is created with the OrderedConfigChange config flag set as false, the configChangeIndex parameter is ignored. Otherwise, it should be set to the most recent Config Change Index value returned by the GetClusterMembership method. The requested delete node operation will be rejected if other membership change has happened since the last call to GetClusterMembership.

func (*NodeHost) RequestLeaderTransfer 

func (nh *NodeHost) RequestLeaderTransfer(clusterID uint64,
	targetNodeID uint64) error

RequestLeaderTransfer makes a request to transfer the leadership of the specified Raft cluster to the target node identified by targetNodeID. It returns an error if the request fails to be started. There is no guarantee that such request can be fulfilled, i.e. the leadership transfer can still fail after a successful return of the RequestLeaderTransfer method.

func (*NodeHost) StartCluster 

func (nh *NodeHost) StartCluster(nodes map[uint64]string,
	join bool, createStateMachine func(uint64, uint64) statemachine.IStateMachine,
	config config.Config) error

StartCluster adds the specified Raft cluster node to the NodeHost and starts the node to make it ready for accepting incoming requests. The input parameter nodes is a map of node ID to RaftAddress for indicating what are initial nodes when the Raft cluster is first created. The join flag indicates whether the node is a new node joining an existing cluster. createStateMachine is a factory function for creating the IStateMachine instance, config is the configuration instance that will be passed to the underlying Raft node object, the cluster ID and node ID of the involved node is given in the ClusterID and NodeID fields of the config object.

Note that this method is not for changing the membership of the specified Raft cluster, it launches a node that is already a member of the Raft cluster.

As a summary, when -

  • starting a brand new Raft cluster with initial member nodes, set join to false and specify all initial member node details in the nodes map.
  • restarting an crashed or stopped node, set join to false. the content of the nodes map is ignored.
  • joining a new node to an existing Raft cluster, set join to true and leave the nodes map empty. This requires the joining node to have already been added as a member of the Raft cluster.
Example 
nhc := config.NodeHostConfig{
	WALDir:         "wal",
	NodeHostDir:    "dragonboat",
	RTTMillisecond: 200,
	RaftAddress:    "myhostname:5012",
}
// Creates a nodehost instance using the above NodeHostConfig instnace.
nh := NewNodeHost(nhc)
// config for raft
rc := config.Config{
	NodeID:             1,
	ClusterID:          100,
	ElectionRTT:        5,
	HeartbeatRTT:       1,
	CheckQuorum:        true,
	SnapshotEntries:    10000,
	CompactionOverhead: 5000,
}
peers := make(map[uint64]string)
peers[100] = "myhostname1:5012"
peers[200] = "myhostname2:5012"
peers[300] = "myhostname3:5012"
// Use this NO-OP data store in this example
NewStateMachine := func(clusterID uint64, nodeID uint64) statemachine.IStateMachine {
	return &tests.NoOP{}
}
if err := nh.StartCluster(peers, false, NewStateMachine, rc); err != nil {
	log.Fatalf("failed to add cluster, %v\n", err)
}

Output:

func (*NodeHost) StartClusterUsingPlugin 

func (nh *NodeHost) StartClusterUsingPlugin(nodes map[uint64]string,
	join bool, pluginFilename string, config config.Config) error

StartClusterUsingPlugin adds a new cluster node to the NodeHost and start running the new node. Different from the StartCluster method in which you specify the factory function used for creating the IStateMachine instance, StartClusterUsingPlugin requires the full path of the CPP plugin you want the Raft cluster to use.

func (*NodeHost) Stop 

func (nh *NodeHost) Stop()

Stop stops all Raft nodes managed by the NodeHost instance, closes the transport and persistent storage modules.

func (*NodeHost) StopCluster 

func (nh *NodeHost) StopCluster(clusterID uint64) error

StopCluster removes and stops the Raft node associated with the specified Raft cluster from the NodeHost. The node to be removed and stopped is identified by the clusterID value.

Note that this is not the membership change operation to remove the node from the Raft cluster.

func (*NodeHost) StopNode 

func (nh *NodeHost) StopNode(clusterID uint64, nodeID uint64) error

StopNode removes the specified Raft cluster node from the NodeHost and stops that running Raft node.

Note that this is not the membership change operation to remove the node from the Raft cluster.

func (*NodeHost) SyncPropose 

func (nh *NodeHost) SyncPropose(ctx context.Context,
	session *client.Session, cmd []byte) (uint64, error)

SyncPropose makes a synchronous proposal on the Raft cluster specified by the input client session object. It returns the result code returned by IStateMachine's Update method, or the error encountered. The input byte slice can be reused for other purposes immediate after the return of this method.

After calling SyncPropose, unless NO-OP client session is used, it is caller's responsibility to update the client session instance accordingly based on SyncPropose's outcome. Basically, when a ErrTimeout error is returned, application can retry the same proposal without updating the client session instance. When ErrInvalidSession error is returned, it usually means the session instance has been evicted from the server side, the Raft paper recommends to crash the client in this highly unlikely event. When the proposal completed successfully, caller must call client.ProposalCompleted() to get it ready to be used in future proposals.

func (*NodeHost) SyncRead 

func (nh *NodeHost) SyncRead(ctx context.Context, clusterID uint64,
	query []byte) ([]byte, error)

SyncRead performs a synchronous linearizable read on the specified Raft cluster. The query byte slice specifies what to query, it will be passed to the Lookup method of the IStateMachine after the system determines that it is safe to perform the local read on IStateMachine. It returns the query result from IStateMachine's Lookup method or the error encountered.

type NodeHostInfo 

type NodeHostInfo struct {
	// NodeHostAddress is the address of the NodeHost. This is the address for
	// exchanging Raft messages between Nodehosts
	NodeHostAddress string
	// NodeHostAPIAddress is the API address for making RPC calls to NodeHost.
	// Such RPC calls are usually used to make Raft proposals and linearizable
	// reads.
	NodeHostAPIAddress string
	// Region is the region of the NodeHost.
	Region string
	// ClusterInfo is a list of all Raft clusters managed by the NodeHost
	ClusterInfoList []ClusterInfo
	// ClusterIDList is a list of cluster IDs for all Raft clusters managed by
	// the NodeHost
	ClusterIDList []uint64
	// LogInfoIncluded is a boolean flag indicating whether the LogInfo contains
	// a list of raftio.NodeInfo values representing all Raft logs stored on
	// the NodeHost.
	LogInfoIncluded bool
	// LogInfo is a list of raftio.NodeInfo values representing all Raft logs
	// stored on the NodeHost. This list will be empty when LogInfoIncluded is
	// set to false.
	LogInfo []raftio.NodeInfo
}

NodeHostInfo provides info about the NodeHost, including its managed Raft clusters and Raft logs saved in its local persistent storage.

type RequestResult 

type RequestResult struct {
	// contains filtered or unexported fields
}

RequestResult is the result struct returned for the request.

func (*RequestResult) Completed 

func (rr *RequestResult) Completed() bool

Completed returns a boolean value indicating the request request completed successfully. For proposals, it means the proposal has been committed by the Raft cluster and applied on the local node. For ReadIndex operation, it means the cluster is now ready for a local read.

func (*RequestResult) GetResult 

func (rr *RequestResult) GetResult() uint64

GetResult returns the result value of the request. When making a proposal, the returned result is the value returned by the Update method of the IStateMachine instance.

func (*RequestResult) Rejected 

func (rr *RequestResult) Rejected() bool

Rejected returns a boolean value indicating the request is rejected. For a proposal, it means that the used client session instance is not registered or it has been evicted on the server side. When requesting a client session to be registered, Rejected means the another client session with the same client ID has already been registered. When requesting a client session to be unregistered, Rejected means the specified client session is not found on the server side. For a membership change request, it means the request is out of order and thus ignored. Note that the out-of-order check when making membership changes is only imposed when IMasterClient is used in NodeHost.

func (*RequestResult) Terminated 

func (rr *RequestResult) Terminated() bool

Terminated returns a boolean value indicating the request terminated due to the requested Raft cluster is being shut down.

func (*RequestResult) Timeout 

func (rr *RequestResult) Timeout() bool

Timeout returns a boolean value indicating whether the Request timed out.

type RequestResultCode 

type RequestResultCode int

RequestResultCode is the result code returned to the client to indicate the outcome of the request.

func (RequestResultCode) String 

func (c RequestResultCode) String() string

type RequestState 

type RequestState struct {

	// CompleteC is a channel for delivering request result to users.
	CompletedC chan RequestResult
	// contains filtered or unexported fields
}

RequestState is the object used to provide request result to users.

func (*RequestState) Release 

func (r *RequestState) Release()

Release puts the RequestState object back to the sync.Pool pool.

Source Files

View all Source files

Directories

Path Synopsis
Package binding contains utility functions and structs for making language bindings.
 Package binding contains utility functions and structs for making language bindings.
Package client implements the client Session struct for identifying proposal clients and their progress.
 Package client implements the client Session struct for identifying proposal clients and their progress.
Package config contains functions and types used for managing dragonboat's configurations.
 Package config contains functions and types used for managing dragonboat's configurations.
Package drummer implements a reference Master server and client.
 Package drummer implements a reference Master server and client.
client
Package client contains RPC client functions and structs.
 Package client contains RPC client functions and structs.
drummerpb
Package drummerpb is a generated protocol buffer package.
 Package drummerpb is a generated protocol buffer package.
multiraftpb
Package multiraftpb is a generated protocol buffer package.
 Package multiraftpb is a generated protocol buffer package.
server
Package server contains server programs used by the drummer package.
 Package server contains server programs used by the drummer package.
server/drummer
Drummer is dragonboat's drummer server program.
 Drummer is dragonboat's drummer server program.
server/drummercmd
Drummercmd is a drummer client program.
 Drummercmd is a drummer client program.
server/nodehost
Nodehost is dragonboat's nodehost server program.
 Nodehost is dragonboat's nodehost server program.
internal
cpp
Package cpp implements a C++11 wrapper to allow IStateMachine to be implemented in C++11.
 Package cpp implements a C++11 wrapper to allow IStateMachine to be implemented in C++11.
logdb
Package logdb implements the persistent log storage used by Dragonboat.
 Package logdb implements the persistent log storage used by Dragonboat.
logdb/gorocksdb
Package gorocksdb provides the ability to create and access RocksDB databases.
 Package gorocksdb provides the ability to create and access RocksDB databases.
logdb/levigo
Package levigo provides the ability to create and access LevelDB databases.
 Package levigo provides the ability to create and access LevelDB databases.
raft
Package raft is a distributed consensus package that implements the Raft protocol.
 Package raft is a distributed consensus package that implements the Raft protocol.
rsm
Package rsm implements Replicated State Machines used in Dragonboat.
 Package rsm implements Replicated State Machines used in Dragonboat.
server
settings
Package settings is used for managing internal parameters that can be set at compile time by expert level users.
 Package settings is used for managing internal parameters that can be set at compile time by expert level users.
tests
Package tests contains various helper functions and modules used in tests.
 Package tests contains various helper functions and modules used in tests.
tests/charybdefs/server
tests/kvpb
tests/kvtest
kvtest is IStateMachine plugin used in various tests.
 kvtest is IStateMachine plugin used in various tests.
tests/lcm
Package lcm is a linearizable checker manager.
 Package lcm is a linearizable checker manager.
tests/lcm/checker
tests/lcm/porcupine
tests/snapshotbench
transport
Package transport implements the transport component used for exchanging Raft messages between NodeHosts.
 Package transport implements the transport component used for exchanging Raft messages between NodeHosts.
utils/cache
utils/cache/biogo/store/interval
Package interval implements an interval tree based on an augmented Left-Leaning Red Black tree.
 Package interval implements an interval tree based on an augmented Left-Leaning Red Black tree.
utils/cache/biogo/store/interval/landscape
Package landscape implements persistence landscape calculation for a set of intervals.
 Package landscape implements persistence landscape calculation for a set of intervals.
utils/cache/biogo/store/kdtree
Package kdtree implements a k-d tree.
 Package kdtree implements a k-d tree.
utils/cache/biogo/store/llrb
Package llrb implements Left-Leaning Red Black trees as described by Robert Sedgewick.
 Package llrb implements Left-Leaning Red Black trees as described by Robert Sedgewick.
utils/cache/biogo/store/step
Package step implements a step vector type.
 Package step implements a step vector type.
utils/compression
utils/compression/snappy
Package snappy implements the snappy block-based compression format.
 Package snappy implements the snappy block-based compression format.
utils/envutil
utils/fileutil
utils/lang
utils/leaktest
utils/logutil
utils/logutil/capnslog
utils/netutil
utils/netutil/cenk/backoff
Package backoff implements backoff algorithms for retrying operations.
 Package backoff implements backoff algorithms for retrying operations.
utils/netutil/facebookgo/clock
utils/netutil/rubyist/circuitbreaker
Package circuit implements the Circuit Breaker pattern.
 Package circuit implements the Circuit Breaker pattern.
utils/random
utils/stringutil
utils/syncutil
Package logger manages loggers used in dragonboat.
 Package logger manages loggers used in dragonboat.
plugin
rpc
rpc/transportpb
Package raftio contains structs, interfaces and function definitions required to build customized persistent Raft log storage and Raft RPC modules.
 Package raftio contains structs, interfaces and function definitions required to build customized persistent Raft log storage and Raft RPC modules.
Package raftpb contains definitions of Raft internal structs, states and messages that are not visible to typical dragonboat applications.
 Package raftpb contains definitions of Raft internal structs, states and messages that are not visible to typical dragonboat applications.
Package statemachine contains the definition of the IStateMachine interface required to be implemented by dragonboat based applications.
 Package statemachine contains the definition of the IStateMachine interface required to be implemented by dragonboat based applications.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.