gorums

package module
v0.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 10, 2021 License: MIT Imports: 28 Imported by: 18

README

Gorums

license go reference build golangci-lint

Gorums [1] is a novel framework for building fault tolerant distributed systems. Gorums offers a flexible and simple quorum call abstraction, used to communicate with a set of processes, and to collect and process their responses. Gorums provides separate abstractions for (a) selecting processes for a quorum call and (b) processing replies. These abstractions simplify the main control flow of protocol implementations, especially for quorum-based systems, where only a subset of the replies to a quorum call need to be processed.

Gorums uses code generation to produce an RPC library that clients can use to invoke quorum calls. Gorums is a wrapper around the gRPC library. Services are defined using the protocol buffers interface definition language.

System Requirements

To build and deploy Gorums, you need the following software installed:

  • Protobuf compiler (protoc)
  • Make
  • Ansible (used by benchmark script)

Contributors Guide

We value your contributions. Before starting a contribution, please reach out to us by posting on an existing issue or creating a new one. Students and other contributors are encouraged to follow these guidelines:

  • We recommend using VSCode with the following plugins
    • Go plugin with the
      • gopls language server enabled
      • golangci-lint enabled
    • Code Spell Checker
    • markdownlint
    • vscode-proto3
  • Code should regularly be merged into master through pull requests.

Examples

The original EPaxos implementation modified to use Gorums can be found here.

A collection of different algorithms for reconfigurable atomic storage implemented using Gorums can be found here.

Documentation

Publications

[1] Tormod Erevik Lea, Leander Jehl, and Hein Meling. Towards New Abstractions for Implementing Quorum-based Systems. In 37th International Conference on Distributed Computing Systems (ICDCS), Jun 2017.

[2] Sebastian Pedersen, Hein Meling, and Leander Jehl. An Analysis of Quorum-based Abstractions: A Case Study using Gorums to Implement Raft. In Proceedings of the 2018 Workshop on Advanced Tools, Programming Languages, and PLatforms for Implementing and Evaluating Algorithms for Distributed systems.

Authors

  • Hein Meling
  • John Ingve Olsen
  • Tormod Erevik Lea
  • Leander Jehl

Documentation

Overview

Package gorums provide protobuf options for gRPC-based quorum calls.

Index

Constants

View Source 
const (
	// MaxVersion is the maximum supported version for generated .pb.go files.
	// It is always the current version of the module.
	MaxVersion = version.Minor

	// GenVersion is the runtime version required by generated .pb.go files.
	// This is incremented when generated code relies on new functionality
	// in the runtime.
	GenVersion = 5

	// MinVersion is the minimum supported version for generated .pb.go files.
	// This is incremented when the runtime drops support for old code.
	MinVersion = 0
)
View Source 
const ContentSubtype = "gorums"
View Source 
const LevelNotSet = -1

LevelNotSet is the zero value level used to indicate that no level (and thereby no reply) has been set for a correctable quorum call.

Variables

View Source 
var (
	// call types
	//
	// optional bool rpc = 50001;
	E_Rpc = &file_gorums_proto_extTypes[0]
	// optional bool unicast = 50002;
	E_Unicast = &file_gorums_proto_extTypes[1]
	// optional bool multicast = 50003;
	E_Multicast = &file_gorums_proto_extTypes[2]
	// optional bool quorumcall = 50004;
	E_Quorumcall = &file_gorums_proto_extTypes[3]
	// optional bool correctable = 50005;
	E_Correctable = &file_gorums_proto_extTypes[4]
	// options for call types
	//
	// optional bool async = 50010;
	E_Async = &file_gorums_proto_extTypes[5]
	// optional bool per_node_arg = 50020;
	E_PerNodeArg = &file_gorums_proto_extTypes[6]
	// optional string custom_return_type = 50030;
	E_CustomReturnType = &file_gorums_proto_extTypes[7]
)

Extension fields to descriptorpb.MethodOptions.

View Source 
var File_gorums_proto protoreflect.FileDescriptor
View Source 
var ID = func(n1, n2 *Node) bool {
	return n1.id < n2.id
}

ID sorts nodes by their identifier in increasing order.

View Source 
var LastNodeError = func(n1, n2 *Node) bool {
	if n1.channel.lastErr() != nil && n2.channel.lastErr() == nil {
		return false
	}
	return true
}

LastNodeError sorts nodes by their LastErr() status in increasing order. A node with LastErr() != nil is larger than a node with LastErr() == nil.

View Source 
var Port = func(n1, n2 *Node) bool {
	p1, _ := strconv.Atoi(n1.Port())
	p2, _ := strconv.Atoi(n2.Port())
	return p1 < p2
}

Port sorts nodes by their port number in increasing order. Warning: This function may be removed in the future.

Functions

func ConfigCreationError added in v0.3.0 

func ConfigCreationError(err error) error

ConfigCreationError returns an error reporting that a Configuration could not be created due to err.

func TestSetup 

func TestSetup(t testing.TB, numServers int, srvFn func(i int) ServerIface) ([]string, func())

TestSetup starts numServers gRPC servers using the given registration function, and returns the server addresses along with a stop function that should be called to shut down the test. This function can be used by other packages for testing purposes.

Types

type Async added in v0.3.0 

type Async struct {
	// contains filtered or unexported fields
}

func (*Async) Done added in v0.3.0 

func (f *Async) Done() bool

Done reports if a reply and/or error is available for the called method.

func (*Async) Get added in v0.3.0 

func (f *Async) Get() (protoreflect.ProtoMessage, error)

Get returns the reply and any error associated with the called method. The method blocks until a reply or error is available.

type CallData added in v0.3.0 

type CallData struct {
	Message protoreflect.ProtoMessage
	Method  string
}

type CallOption added in v0.3.0 

type CallOption func(*callOptions)

CallOption is a function that sets a value in the given callOptions struct

func WithNoSendWaiting added in v0.3.0 

func WithNoSendWaiting() CallOption

WithNoSendWaiting is a CallOption that makes Unicast or Multicast methods return immediately instead of blocking until the message has been sent.

type Codec added in v0.3.0 

type Codec struct {
	// contains filtered or unexported fields
}

func NewCodec added in v0.3.0 

func NewCodec() *Codec

func (Codec) Marshal added in v0.3.0 

func (c Codec) Marshal(m interface{}) (b []byte, err error)

func (Codec) Name added in v0.3.0 

func (c Codec) Name() string

func (Codec) String added in v0.3.0 

func (c Codec) String() string

func (Codec) Unmarshal added in v0.3.0 

func (c Codec) Unmarshal(b []byte, m interface{}) (err error)

type ConfigOption added in v0.3.0 

type ConfigOption interface{}

ConfigOption is a marker interface for options to NewConfiguration.

type Configuration added in v0.3.0 

type Configuration []*Node

Configuration represents a static set of nodes on which quorum calls may be invoked.

func NewConfiguration added in v0.3.0 

func NewConfiguration(mgr *Manager, opt NodeListOption) (nodes Configuration, err error)

NewConfiguration returns a configuration based on the provided list of nodes. Nodes can be supplied using WithNodeMap or WithNodeList, or WithNodeIDs. A new configuration can also be created from an existing configuration, using the And, WithNewNodes, Except, and WithoutNodes methods.

func (Configuration) And added in v0.4.0 

func (c Configuration) And(d Configuration) NodeListOption

And returns a NodeListOption that can be used to create a new configuration combining c and d.

func (Configuration) AsyncCall added in v0.3.0 

func (c Configuration) AsyncCall(ctx context.Context, d QuorumCallData) *Async

func (Configuration) CorrectableCall added in v0.3.0 

func (c Configuration) CorrectableCall(ctx context.Context, d CorrectableCallData) *Correctable

func (Configuration) Equal added in v0.3.0 

func (c Configuration) Equal(b Configuration) bool

Equal returns true if configurations b and c have the same set of nodes.

func (Configuration) Except added in v0.4.0 

func (c Configuration) Except(rm Configuration) NodeListOption

Except returns a NodeListOption that can be used to create a new configuration from c without the nodes in rm.

func (Configuration) Multicast added in v0.3.0 

func (c Configuration) Multicast(ctx context.Context, d QuorumCallData, opts ...CallOption)

Multicast is a one-way call; no replies are processed. By default this function returns once the message has been sent to all nodes. Providing the call option WithNoSendWaiting, the function may return before the message has been sent.

func (Configuration) NodeIDs added in v0.3.0 

func (c Configuration) NodeIDs() []uint32

NodeIDs returns a slice of this configuration's Node IDs.

func (Configuration) Nodes added in v0.3.0 

func (c Configuration) Nodes() []*Node

Nodes returns the nodes in this configuration.

func (Configuration) QuorumCall added in v0.3.0 

func (c Configuration) QuorumCall(ctx context.Context, d QuorumCallData) (resp protoreflect.ProtoMessage, err error)

func (Configuration) Size added in v0.3.0 

func (c Configuration) Size() int

Size returns the number of nodes in this configuration.

func (Configuration) WithNewNodes added in v0.4.0 

func (c Configuration) WithNewNodes(new NodeListOption) NodeListOption

WithNewNodes returns a NodeListOption that can be used to create a new configuration combining c and the new nodes.

func (Configuration) WithoutNodes added in v0.4.0 

func (c Configuration) WithoutNodes(ids ...uint32) NodeListOption

WithoutNodes returns a NodeListOption that can be used to create a new configuration from c without the given node IDs.

type Correctable added in v0.3.0 

type Correctable struct {
	// contains filtered or unexported fields
}

func (*Correctable) Done added in v0.3.0 

func (c *Correctable) Done() <-chan struct{}

func (*Correctable) Get added in v0.3.0 

func (c *Correctable) Get() (protoreflect.ProtoMessage, int, error)

func (*Correctable) Watch added in v0.3.0 

func (c *Correctable) Watch(level int) <-chan struct{}

type CorrectableCallData added in v0.3.0 

type CorrectableCallData struct {
	Message        protoreflect.ProtoMessage
	Method         string
	PerNodeArgFn   func(protoreflect.ProtoMessage, uint32) protoreflect.ProtoMessage
	QuorumFunction func(protoreflect.ProtoMessage, map[uint32]protoreflect.ProtoMessage) (protoreflect.ProtoMessage, int, bool)
	ServerStream   bool
}

type EnforceVersion added in v0.4.0 

type EnforceVersion uint

EnforceVersion is used by code generated by protoc-gen-gorums to statically enforce minimum and maximum versions of this package. A compilation failure implies either that:

  • the runtime package is too old and needs to be updated OR
  • the generated code is too old and needs to be regenerated.

The runtime package can be upgraded by running: 

go get github.com/relab/gorums

The generated code can be regenerated by running: 

protoc --gorums_out=${PROTOC_GEN_GORUMS_ARGS} ${PROTO_FILES}

Example usage by generated code: 

const (
	// Verify that this generated code is sufficiently up-to-date.
	_ = gorums.EnforceVersion(genVersion - gorums.MinVersion)
	// Verify that runtime/protoimpl is sufficiently up-to-date.
	_ = gorums.EnforceVersion(gorums.MaxVersion - genVersion)
)

The genVersion is the current minor version used to generated the code. This compile-time check relies on negative integer overflow of a uint being a compilation failure (guaranteed by the Go specification).

type Error added in v0.3.0 

type Error struct {
	NodeID uint32
	Cause  error
}

Error is used to report that a single gRPC call failed.

func (Error) Error added in v0.3.0 

func (e Error) Error() string

type Manager added in v0.3.0 

type Manager struct {
	// contains filtered or unexported fields
}

Manager maintains a connection pool of nodes on which quorum calls can be performed.

func NewManager added in v0.3.0 

func NewManager(opts ...ManagerOption) *Manager

NewManager returns a new Manager for managing connection to nodes added to the manager. This function accepts manager options used to configure various aspects of the manager. This function is meant for internal use. You should use the `NewManager` function in the generated code instead.

func (*Manager) AddNode added in v0.3.0 

func (m *Manager) AddNode(node *Node) error

AddNode adds the node to the manager's node pool and establishes a connection to the node.

func (*Manager) Close added in v0.3.0 

func (m *Manager) Close()

Close closes all node connections and any client streams.

func (*Manager) Node added in v0.3.0 

func (m *Manager) Node(id uint32) (node *Node, found bool)

Node returns the node with the given identifier if present.

func (*Manager) NodeIDs added in v0.3.0 

func (m *Manager) NodeIDs() []uint32

NodeIDs returns the identifier of each available node. IDs are returned in the same order as they were provided in the creation of the Manager.

func (*Manager) Nodes added in v0.3.0 

func (m *Manager) Nodes() []*Node

Nodes returns a slice of each available node. IDs are returned in the same order as they were provided in the creation of the Manager.

func (*Manager) Size added in v0.3.0 

func (m *Manager) Size() (nodes int)

Size returns the number of nodes in the Manager.

type ManagerOption added in v0.3.0 

type ManagerOption func(*managerOptions)

ManagerOption provides a way to set different options on a new Manager.

func WithBackoff added in v0.3.0 

func WithBackoff(backoff backoff.Config) ManagerOption

WithBackoff allows for changing the backoff delays used by Gorums.

func WithDialTimeout added in v0.3.0 

func WithDialTimeout(timeout time.Duration) ManagerOption

WithDialTimeout returns a ManagerOption which is used to set the dial context timeout to be used when initially connecting to each node in its pool.

func WithGrpcDialOptions added in v0.3.0 

func WithGrpcDialOptions(opts ...grpc.DialOption) ManagerOption

WithGrpcDialOptions returns a ManagerOption which sets any gRPC dial options the Manager should use when initially connecting to each node in its pool.

func WithLogger added in v0.3.0 

func WithLogger(logger *log.Logger) ManagerOption

WithLogger returns a ManagerOption which sets an optional error logger for the Manager.

func WithMetadata added in v0.3.0 

func WithMetadata(md metadata.MD) ManagerOption

WithMetadata returns a ManagerOption that sets the metadata that is sent to each node when the connection is initially established. This metadata can be retrieved from the server-side method handlers.

func WithNoConnect added in v0.3.0 

func WithNoConnect() ManagerOption

WithNoConnect returns a ManagerOption which instructs the Manager not to connect to any of its nodes. Mainly used for testing purposes.

func WithPerNodeMetadata added in v0.3.0 

func WithPerNodeMetadata(f func(uint32) metadata.MD) ManagerOption

WithPerNodeMetadata returns a ManagerOption that allows you to set metadata for each node individually.

func WithSendBufferSize added in v0.3.0 

func WithSendBufferSize(size uint) ManagerOption

WithSendBufferSize allows for changing the size of the send buffer used by Gorums. A larger buffer might achieve higher throughput for asynchronous calltypes, but at the cost of latency.

type Message added in v0.3.0 

type Message struct {
	Metadata *ordering.Metadata
	Message  protoreflect.ProtoMessage
	// contains filtered or unexported fields
}

func WrapMessage added in v0.3.0 

func WrapMessage(md *ordering.Metadata, resp protoreflect.ProtoMessage, err error) *Message

WrapMessage wraps the metadata, response and error status in a gorumsMessage

type MultiSorter added in v0.3.0 

type MultiSorter struct {
	// contains filtered or unexported fields
}

MultiSorter implements the Sort interface, sorting the nodes within.

func OrderedBy added in v0.3.0 

func OrderedBy(less ...lessFunc) *MultiSorter

OrderedBy returns a Sorter that sorts using the less functions, in order. Call its Sort method to sort the data.

func (*MultiSorter) Len added in v0.3.0 

func (ms *MultiSorter) Len() int

Len is part of sort.Interface.

func (*MultiSorter) Less added in v0.3.0 

func (ms *MultiSorter) Less(i, j int) bool

Less is part of sort.Interface. It is implemented by looping along the less functions until it finds a comparison that is either Less or not Less. Note that it can call the less functions twice per call. We could change the functions to return -1, 0, 1 and reduce the number of calls for greater efficiency: an exercise for the reader.

func (*MultiSorter) Sort added in v0.3.0 

func (ms *MultiSorter) Sort(nodes []*Node)

Sort sorts the argument slice according to the less functions passed to OrderedBy.

func (*MultiSorter) Swap added in v0.3.0 

func (ms *MultiSorter) Swap(i, j int)

Swap is part of sort.Interface.

type Node added in v0.3.0 

type Node struct {
	// contains filtered or unexported fields
}

Node encapsulates the state of a node on which a remote procedure call can be performed.

func NewNode added in v0.3.0 

func NewNode(addr string) (*Node, error)

NewNode returns a new node for the provided address.

func NewNodeWithID added in v0.3.0 

func NewNodeWithID(addr string, id uint32) (*Node, error)

NewNodeWithID returns a new node for the provided address and id.

func (*Node) Address added in v0.3.0 

func (n *Node) Address() string

Address returns network address of n.

func (*Node) FullString added in v0.3.0 

func (n *Node) FullString() string

FullString returns a more descriptive string representation of n that includes id, network address and latency information.

func (*Node) ID added in v0.3.0 

func (n *Node) ID() uint32

ID returns the ID of n.

func (*Node) LastErr added in v0.3.0 

func (n *Node) LastErr() error

LastErr returns the last error encountered (if any) for this node.

func (*Node) Latency added in v0.5.0 

func (n *Node) Latency() time.Duration

Latency returns the latency between the client and this node.

func (*Node) Port added in v0.3.0 

func (n *Node) Port() string

Port returns network port of n.

func (*Node) RPCCall added in v0.3.0 

func (n *Node) RPCCall(ctx context.Context, d CallData) (resp protoreflect.ProtoMessage, err error)

func (*Node) String added in v0.3.0 

func (n *Node) String() string

func (*Node) Unicast added in v0.3.0 

func (n *Node) Unicast(ctx context.Context, d CallData, opts ...CallOption)

Unicast is a one-way call; no replies are processed. By default this function returns once the message has been sent. Providing the call option WithNoSendWaiting, the function may return before the message has been sent.

type NodeListOption added in v0.4.0 

type NodeListOption interface {
	ConfigOption
	// contains filtered or unexported methods
}

NodeListOption must be implemented by node providers.

func WithNodeIDs added in v0.3.0 

func WithNodeIDs(ids []uint32) NodeListOption

WithNodeIDs returns a NodeListOption containing a list of node IDs. This assumes that the provided node IDs have already been registered with the manager.

func WithNodeList added in v0.3.0 

func WithNodeList(addrsList []string) NodeListOption

WithNodeList returns a NodeListOption containing the provided list of node addresses. With this option, node IDs are generated by the Manager.

func WithNodeMap added in v0.3.0 

func WithNodeMap(idMap map[string]uint32) NodeListOption

WithNodeMap returns a NodeListOption containing the provided mapping from node addresses to application-specific IDs.

type QuorumCallData added in v0.3.0 

type QuorumCallData struct {
	Message        protoreflect.ProtoMessage
	Method         string
	PerNodeArgFn   func(protoreflect.ProtoMessage, uint32) protoreflect.ProtoMessage
	QuorumFunction func(protoreflect.ProtoMessage, map[uint32]protoreflect.ProtoMessage) (protoreflect.ProtoMessage, bool)
}

QuorumCallData holds the message, destination nodes, method identifier, and other information necessary to perform the various quorum call types supported by Gorums.

type QuorumCallError added in v0.3.0 

type QuorumCallError struct {
	Reason     string
	ReplyCount int
	Errors     []Error
}

A QuorumCallError is used to report that a quorum call failed.

func (QuorumCallError) Error added in v0.3.0 

func (e QuorumCallError) Error() string

type Server added in v0.3.0 

type Server struct {
	// contains filtered or unexported fields
}

Server serves all ordering based RPCs using registered handlers.

func NewServer added in v0.3.0 

func NewServer(opts ...ServerOption) *Server

NewServer returns a new instance of GorumsServer. This function is intended for internal Gorums use. You should call `NewServer` in the generated code instead.

func (*Server) GracefulStop added in v0.3.0 

func (s *Server) GracefulStop()

GracefulStop waits for all RPCs to finish before stopping.

func (*Server) RegisterHandler added in v0.3.0 

func (s *Server) RegisterHandler(method string, handler requestHandler)

func (*Server) Serve added in v0.3.0 

func (s *Server) Serve(listener net.Listener) error

Serve starts serving on the listener.

func (*Server) Stop added in v0.3.0 

func (s *Server) Stop()

Stop stops the server immediately.

type ServerIface added in v0.3.0 

type ServerIface interface {
	Serve(net.Listener) error
	Stop()
}

type ServerOption added in v0.3.0 

type ServerOption func(*serverOptions)

ServerOption is used to change settings for the GorumsServer

func WithGRPCServerOptions added in v0.3.0 

func WithGRPCServerOptions(opts ...grpc.ServerOption) ServerOption

WithGRPCServerOptions allows to set gRPC options for the server.

func WithReceiveBufferSize added in v0.3.0 

func WithReceiveBufferSize(size uint) ServerOption

WithReceiveBufferSize sets the buffer size for the server. A larger buffer may result in higher throughput at the cost of higher latency.

Source Files

View all Source files

Directories

Path Synopsis
cmd
benchmark
protoc-gen-gorums
protoc-gen-gorums/dev
protoc-gen-gorums/gengorums
Package gengorums is internal to the gorums protobuf module.
 Package gengorums is internal to the gorums protobuf module.
examples module
internal
correctable
leakcheck
Package leakcheck contains functions to check leaked goroutines.
 Package leakcheck contains functions to check leaked goroutines.
protoc
version
tests
config
dummy
metadata
oneway
ordering
qf
tls
unresponsive

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.