fidias

package module
v0.1.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 29, 2017 License: MPL-2.0 Imports: 18 Imported by: 0

README

fidias Build Status

Table of Contents

Installation

  1. Download a pre-compiled binary from the releases page.

  2. Extract the binary using unzip or tar.

  3. Move the binary into your $PATH.

Start a cluster

The default cluster requires a minimum of 3 nodes to function though this can be changed in the configuration. Below are the steps to spin up a local test cluster.

1. Start the first node in a terminal:

$ fidiasd -debug

2. Start 2 or more nodes - each in separate terminals. Change the addresses to appropriately match your configuration.

$ fidiasd -cluster-addr 127.0.0.1:54322 -http-addr 127.0.0.1:9091 -join 127.0.0.1:54321 -debug
$ fidiasd -cluster-addr 127.0.0.1:54323 -http-addr 127.0.0.1:9092 -join 127.0.0.1:54321 -debug
...

3. You should start seeing peers joining the cluster. To confirm the cluster is functional, perform a locate call and ensure it responds with locations. Here's a sample of what the request and response would look like:

$ curl -XGET http://127.0.0.1:9090/v1/locate/testkey
[
  {
    "ID": "913a73b565c8e2c8ed94497580f619397709b8b6",
    "Priority": 0,
    "Vnode": {
      "Host": "127.0.0.1:54322",
      "Id": "93bb6a44e8e8922afe0dbcbba3a1ed3eaa859850",
      "Meta": "http=127.0.0.1:9091"
    }
  },
  {
    "ID": "e68fc90abb1e381e42e99ecad64b6e8ecc5f0e0b",
    "Priority": 1,
    "Vnode": {
      "Host": "127.0.0.1:54323",
      "Id": "2209bd42504b643b3acfd7e6599699c7a53d962d",
      "Meta": "http=127.0.0.1:9092"
    }
  },
  {
    "ID": "3be51e6010738d73983ef4202ba0c3e421b46360",
    "Priority": 2,
    "Vnode": {
      "Host": "127.0.0.1:54321",
      "Id": "591c0a39be4478ce77124239189bab225a73de75",
      "Meta": "http=127.0.0.1:9090"
    }
  }
]

The cluster is now running and can be used. Details on the HTTP API can be found in the API docs.

Development
  • When using debug mode a significant performance degrade may be seen.
Roadmap

Coming Soon!!

  • Persistence
  • Locking mechanisms
  • Authentication, Authorization and Access

Documentation

Overview

Package fidias is a generated protocol buffer package.

It is generated from these files: 

rpc.proto

It has these top-level messages: 

KeyLocation

Index

Constants

View Source 
const (
	// OpSet is the op to set a ke-value pair
	OpSet byte = iota + 1
	// OpDel is the op to delete a key-value pair
	OpDel
)

Variables

This section is empty.

Functions

func RegisterFidiasRPCServer 

func RegisterFidiasRPCServer(s *grpc.Server, srv FidiasRPCServer)

Types

type Config 

type Config struct {
	Ring            *hexaring.Config
	Hexalog         *hexalog.Config
	RelocateBufSize int           // Relocate request buffer size
	Replicas        int           // Number of replicas for a key
	RetryInterval   time.Duration // interval to wait before retrying a proposal
	StableThreshold time.Duration // Threshold after ring event to consider we are stable
}

Config hold the guac config along with the underlying log and ring config

func DefaultConfig 

func DefaultConfig(hostname string) *Config

DefaultConfig returns a default sane config setting the hostname on the log and ring configs

func (*Config) Hasher added in v0.1.2 

func (conf *Config) Hasher() hexatype.Hasher

Hasher returns the log hasher. This is a helper function

func (*Config) Hostname 

func (conf *Config) Hostname() string

Hostname returns the configured hostname. The assumption here is the log and ring hostnames are the same as they should be checked and set prior to using this call

type DummyFSM 

type DummyFSM struct{}

DummyFSM is a placeholder FSM that does nothing

func (*DummyFSM) Apply 

func (fsm *DummyFSM) Apply(entryID []byte, entry *hexatype.Entry) interface{}

Apply gets called by hexalog each time a new entry has been commit and accepted by the cluster

func (*DummyFSM) Get 

func (fsm *DummyFSM) Get(key []byte) (*hexatype.KeyValuePair, error)

Get is a noop to satisfy the KeyValueFSM interface

type Fidias 

type Fidias struct {
	// contains filtered or unexported fields
}

Fidias is the core that manages all operations for a node. It primary manages rebalancing, replication, and appropriately deals with cluster churn.

func New 

func New(conf *Config, appFSM KeyValueFSM, idx store.IndexStore, entries store.EntryStore, logStore *hexalog.LogStore, stableStore hexalog.StableStore, server *grpc.Server) (g *Fidias, err error)

New instantiates a new instance of Fidias based on the given config and stores along with a grpc server instance to register the network transports

func (*Fidias) GetEntry 

func (fidias *Fidias) GetEntry(key, id []byte) (entry *hexatype.Entry, meta *ReMeta, err error)

GetEntry tries to get an entry from the ring. It gets the replica locations and queries upto the max allowed successors for each location.

func (*Fidias) GetKey 

func (fidias *Fidias) GetKey(key []byte) (kvp *hexatype.KeyValuePair, meta *ReMeta, err error)

GetKey tries to get a key-value pair from a given replica set on the ring. This is not be confused with the log key. It scours the first replica only.

func (*Fidias) Leader added in v0.1.2 

func (fidias *Fidias) Leader(key []byte, locs hexaring.LocationSet) (*hexalog.KeyLeader, error)

Leader returns the leader of the given location set from the underlying log.

func (*Fidias) Leaving 

func (fidias *Fidias) Leaving(local, pred, succ *chord.Vnode)

Leaving is called by the Ring when this node willingly leaves. This is only triggered if an explicit leave is issued

func (*Fidias) NewEntry 

func (fidias *Fidias) NewEntry(key []byte) (*hexatype.Entry, *ReMeta, error)

NewEntry returns a new Entry for the given key from Hexalog. It returns an error if the node is not part of the location set or a lookup error occurs

func (*Fidias) NewPredecessor 

func (fidias *Fidias) NewPredecessor(local, newPred, oldPred *chord.Vnode)

NewPredecessor is called when a local vnode finds a new predecessor. This causes a rebalance of keys. All key hashes less than the new predecessor are transferred to the new predecessor.

func (*Fidias) PredecessorLeaving 

func (fidias *Fidias) PredecessorLeaving(local, remote *chord.Vnode)

PredecessorLeaving is only triggered if an explicit leave is issued

func (*Fidias) ProposeEntry 

func (fidias *Fidias) ProposeEntry(entry *hexatype.Entry, opts *hexatype.RequestOptions) (ballot *hexalog.Ballot, err error)

ProposeEntry finds locations for the entry and submits a new proposal to those locations.

func (*Fidias) Register 

func (fidias *Fidias) Register(ring *hexaring.Ring)

Register registers the chord ring to fidias. This is due to the fact that guac and the ring depend on each other and the ring may not be intialized yet. Only upon ring registration, the rebalancing is started.

func (*Fidias) Shutdown 

func (fidias *Fidias) Shutdown()

Shutdown is called but a chord node is shutdown

func (*Fidias) Status 

func (fidias *Fidias) Status() *Status

Status returns the status of this node

func (*Fidias) SuccessorLeaving 

func (fidias *Fidias) SuccessorLeaving(local, remote *chord.Vnode)

SuccessorLeaving is only triggered if an explicit leave is issued

type FidiasRPCClient 

type FidiasRPCClient interface {
	GetKeyRPC(ctx context.Context, in *hexatype.KeyValuePair, opts ...grpc.CallOption) (*hexatype.KeyValuePair, error)
	// Send key-location id pairs
	RelocateRPC(ctx context.Context, opts ...grpc.CallOption) (FidiasRPC_RelocateRPCClient, error)
}

func NewFidiasRPCClient 

func NewFidiasRPCClient(cc *grpc.ClientConn) FidiasRPCClient

type FidiasRPCServer 

type FidiasRPCServer interface {
	GetKeyRPC(context.Context, *hexatype.KeyValuePair) (*hexatype.KeyValuePair, error)
	// Send key-location id pairs
	RelocateRPC(FidiasRPC_RelocateRPCServer) error
}

type FidiasRPC_RelocateRPCClient added in v0.1.2 

type FidiasRPC_RelocateRPCClient interface {
	Send(*KeyLocation) error
	Recv() (*KeyLocation, error)
	grpc.ClientStream
}

type FidiasRPC_RelocateRPCServer added in v0.1.2 

type FidiasRPC_RelocateRPCServer interface {
	Send(*KeyLocation) error
	Recv() (*KeyLocation, error)
	grpc.ServerStream
}

type InMemKeyValueFSM 

type InMemKeyValueFSM struct {
	// contains filtered or unexported fields
}

InMemKeyValueFSM is a hexalog FSM for an in-memory key-value store. It implements the FSM interface and provides a get function to retrieve keys as all write are handled by the FSM

func NewInMemKeyValueFSM 

func NewInMemKeyValueFSM() *InMemKeyValueFSM

NewInMemKeyValueFSM inits a new InMemKeyValueFSM

func (*InMemKeyValueFSM) Apply 

func (fsm *InMemKeyValueFSM) Apply(entryID []byte, entry *hexatype.Entry) interface{}

Apply applies the given entry to the InMemKeyValueFSM. entryID is the hash id of the entry. The first byte in entry.Data contains the operation to be performed followed by the actual value.

func (*InMemKeyValueFSM) Get 

func (fsm *InMemKeyValueFSM) Get(key []byte) (*hexatype.KeyValuePair, error)

Get gets a value for the key. It reads it directly from the stored log entry

type KVNetTransport 

type KVNetTransport interface {
	GetKey(host string, key []byte) (*hexatype.KeyValuePair, error)
}

KVNetTransport implements a network transport for key-value operations

type KeyLocation added in v0.1.2 

type KeyLocation struct {
	// Actual key
	Key []byte `protobuf:"bytes,1,opt,name=Key,json=key,proto3" json:"Key,omitempty"`
	// Location id
	Marker []byte `protobuf:"bytes,2,opt,name=Marker,json=marker,proto3" json:"Marker,omitempty"`
	// Total known height of the key
	Height uint32 `protobuf:"varint,3,opt,name=Height,json=height" json:"Height,omitempty"`
}

func (*KeyLocation) Descriptor added in v0.1.2 

func (*KeyLocation) Descriptor() ([]byte, []int)

func (*KeyLocation) GetHeight added in v0.1.2 

func (m *KeyLocation) GetHeight() uint32

func (*KeyLocation) GetKey added in v0.1.2 

func (m *KeyLocation) GetKey() []byte

func (*KeyLocation) GetMarker added in v0.1.2 

func (m *KeyLocation) GetMarker() []byte

func (*KeyLocation) ProtoMessage added in v0.1.2 

func (*KeyLocation) ProtoMessage()

func (*KeyLocation) Reset added in v0.1.2 

func (m *KeyLocation) Reset()

func (*KeyLocation) String added in v0.1.2 

func (m *KeyLocation) String() string

type KeyValueFSM 

type KeyValueFSM interface {
	hexalog.FSM
	Get(key []byte) (*hexatype.KeyValuePair, error)
}

KeyValueFSM is an FSM for a key value store. Aside from fsm functions, it also contains read-only key-value functions needed.

type KeyValueStore 

type KeyValueStore interface {
	Get(key []byte) (*hexatype.KeyValuePair, error)
}

KeyValueStore implements a key value store interface

type NetTransport 

type NetTransport struct {
	// contains filtered or unexported fields
}

NetTransport implements a network transport needed for fidias

func NewNetTransport added in v0.1.2 

func NewNetTransport(kvs KeyValueStore, idx store.IndexStore, reapInterval, maxIdle time.Duration, replicas int, hasher hexatype.Hasher) *NetTransport

NewNetTransport instantiates a new network transport using the given key-value store.

func (*NetTransport) GetKey 

func (trans *NetTransport) GetKey(host string, key []byte) (*hexatype.KeyValuePair, error)

GetKey retrieves a key from a remote host

func (*NetTransport) GetKeyRPC 

func (trans *NetTransport) GetKeyRPC(ctx context.Context, in *hexatype.KeyValuePair) (*hexatype.KeyValuePair, error)

GetKeyRPC serves a GetKey request

func (*NetTransport) GetRelocateStream added in v0.1.2 

func (trans *NetTransport) GetRelocateStream(local, remote *chord.Vnode) (*RelocateStream, error)

GetRelocateStream gets a stream to send rebalance data across

func (*NetTransport) Register added in v0.1.2 

func (trans *NetTransport) Register(ch chan<- *relocateReq)

Register registers a write channel used for submitting reloc. requests

func (*NetTransport) RelocateRPC added in v0.1.2 

func (trans *NetTransport) RelocateRPC(stream FidiasRPC_RelocateRPCServer) error

RelocateRPC serves a rebalance request for the ring

func (*NetTransport) Shutdown 

func (trans *NetTransport) Shutdown()

Shutdown signals the transport to be shutdown. After shutdown no new connections can be

type ReMeta 

type ReMeta struct {
	Vnode   *chord.Vnode         // vnode processing the request or response
	PeerSet hexaring.LocationSet // set of peers involved
}

ReMeta contains metadata associated to a request or response

type RelocateStream added in v0.1.2 

type RelocateStream struct {
	FidiasRPC_RelocateRPCClient // grp stream client
	// contains filtered or unexported fields
}

RelocateStream is a stream to handle relocating of keys between nodes.

func (*RelocateStream) Recycle added in v0.1.2 

func (rs *RelocateStream) Recycle()

Recycle recycles the stream returning the conn back to the pool

type Relocator added in v0.1.2 

type Relocator struct {
	// contains filtered or unexported fields
}

Relocator is responsible for moving data as needed when the underlying cluster topology changes

func NewRelocator added in v0.1.2 

func NewRelocator(conf *Config, idx store.IndexStore, trans RelocatorTransport) *Relocator

NewRelocator instantiates a new Relocator

type RelocatorTransport added in v0.1.2 

type RelocatorTransport interface {
	GetRelocateStream(local, remote *chord.Vnode) (*RelocateStream, error)
}

RelocatorTransport implements a transport needed by the key rebalancing engine

type Status added in v0.1.2 

type Status struct {
	Ring *chord.Status
}

Status contains status information of a node.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.