hexaring

package module
v0.0.0-...-7d3a640 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 15, 2017 License: MPL-2.0 Imports: 18 Imported by: 3

README

hexaring Build Status

Hexaring implements a replicated lookup mechanism on top of chord in order to disassociate the ring key mapping from the application keys. It provides various helper functions to perform ring based operations to allow this.

Location Based Lookups

Lookups return the specified number of locations for a given key around the ring. The algorithm used is as follows:

  • Hash a key to compute the natural key
  • Get requested number of unique replicas around the ring using the natural key as the offset

Documentation

Overview

Package hexaring is a generated protocol buffer package.

It is generated from these files: 

structs.proto

It has these top-level messages: 

Location
LookupRequest
LookupResponse

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func BuildReplicaHashes 

func BuildReplicaHashes(key []byte, count int64, h hash.Hash) [][]byte

BuildReplicaHashes hashes the given key and build the required additional hashes returning the requested count of hashes.

func CalculateRingVertexBytes 

func CalculateRingVertexBytes(hash []byte, count int64) [][]byte

CalculateRingVertexBytes returns the a slice of bytes one for each vertex

func CalculateRingVertexes 

func CalculateRingVertexes(hash []byte, count int64) []*big.Int

CalculateRingVertexes returns the requested number of vertexes around the ring equi-distant from each other except for potentially the last one which may be larger

func DefaultConfig 

func DefaultConfig(hostname string) *chord.Config

DefaultConfig returns a sane config

func RegisterLookupRPCServer 

func RegisterLookupRPCServer(s *grpc.Server, srv LookupRPCServer)

Types

type InMemPeerStore 

type InMemPeerStore struct {
	// contains filtered or unexported fields
}

InMemPeerStore implements an in-memory PeerStore interface

func NewInMemPeerStore 

func NewInMemPeerStore() *InMemPeerStore

NewInMemPeerStore instantiates a new in-memory peer store

func (*InMemPeerStore) AddPeer 

func (ps *InMemPeerStore) AddPeer(peer string) bool

AddPeer adds the given peer to the store. If it exists then the last seen time is updated and false is returned

func (*InMemPeerStore) Peers 

func (ps *InMemPeerStore) Peers() []string

Peers returns a slice of all known peers

func (*InMemPeerStore) RemovePeer 

func (ps *InMemPeerStore) RemovePeer(peer string)

RemovePeer removes a peer from the store.

type Location 

type Location struct {
	ID []byte `protobuf:"bytes,1,opt,name=ID,json=iD,proto3" json:"ID,omitempty"`
	// Priority among locations in a set
	Priority int32 `protobuf:"varint,2,opt,name=Priority,json=priority" json:"Priority,omitempty"`
	// Index within location group
	Index int32 `protobuf:"varint,3,opt,name=Index,json=index" json:"Index,omitempty"`
	// Vnode for location id
	Vnode *chord.Vnode `protobuf:"bytes,4,opt,name=Vnode,json=vnode" json:"Vnode,omitempty"`
}

func (*Location) Descriptor 

func (*Location) Descriptor() ([]byte, []int)

func (*Location) GetID 

func (m *Location) GetID() []byte

func (*Location) GetIndex 

func (m *Location) GetIndex() int32

func (*Location) GetPriority 

func (m *Location) GetPriority() int32

func (*Location) GetVnode 

func (m *Location) GetVnode() *chord.Vnode

func (*Location) Host 

func (loc *Location) Host() string

Host returns the host of the location

func (Location) MarshalJSON 

func (loc Location) MarshalJSON() ([]byte, error)

MarshalJSON is a custom Location json marshaller

func (*Location) ProtoMessage 

func (*Location) ProtoMessage()

func (*Location) Reset 

func (m *Location) Reset()

func (*Location) String 

func (m *Location) String() string

type LocationSet 

type LocationSet []*Location

LocationSet is a set of locations responsible for a key.

func (LocationSet) EndRange 

func (locs LocationSet) EndRange(locID []byte) (end []byte, err error)

EndRange returns the ending range for the given starting location id. It returns an error if the location is not in the set

func (LocationSet) GetByHost 

func (locs LocationSet) GetByHost(host string) (*Location, error)

GetByHost returns a location by the given host or a not found error otherwise

func (LocationSet) GetNext 

func (locs LocationSet) GetNext(host string) (*Location, error)

GetNext returns the next location after the given host

func (LocationSet) NaturalRange 

func (locs LocationSet) NaturalRange() (start []byte, end []byte)

NaturalRange returns the hash range for the natural key hash

func (LocationSet) String 

func (locs LocationSet) String() string

type LookupRPCClient 

type LookupRPCClient interface {
	LookupRPC(ctx context.Context, in *LookupRequest, opts ...grpc.CallOption) (*LookupResponse, error)
	LookupHashRPC(ctx context.Context, in *LookupRequest, opts ...grpc.CallOption) (*LookupResponse, error)
	LookupReplicatedRPC(ctx context.Context, in *LookupRequest, opts ...grpc.CallOption) (*LookupResponse, error)
	LookupReplicatedHashRPC(ctx context.Context, in *LookupRequest, opts ...grpc.CallOption) (*LookupResponse, error)
}

func NewLookupRPCClient 

func NewLookupRPCClient(cc *grpc.ClientConn) LookupRPCClient

type LookupRPCServer 

type LookupRPCServer interface {
	LookupRPC(context.Context, *LookupRequest) (*LookupResponse, error)
	LookupHashRPC(context.Context, *LookupRequest) (*LookupResponse, error)
	LookupReplicatedRPC(context.Context, *LookupRequest) (*LookupResponse, error)
	LookupReplicatedHashRPC(context.Context, *LookupRequest) (*LookupResponse, error)
}

type LookupRequest 

type LookupRequest struct {
	Key []byte `protobuf:"bytes,1,opt,name=Key,json=key,proto3" json:"Key,omitempty"`
	N   int32  `protobuf:"varint,2,opt,name=N,json=n" json:"N,omitempty"`
}

func (*LookupRequest) Descriptor 

func (*LookupRequest) Descriptor() ([]byte, []int)

func (*LookupRequest) GetKey 

func (m *LookupRequest) GetKey() []byte

func (*LookupRequest) GetN 

func (m *LookupRequest) GetN() int32

func (*LookupRequest) ProtoMessage 

func (*LookupRequest) ProtoMessage()

func (*LookupRequest) Reset 

func (m *LookupRequest) Reset()

func (*LookupRequest) String 

func (m *LookupRequest) String() string

type LookupResponse 

type LookupResponse struct {
	Locations []*Location    `protobuf:"bytes,1,rep,name=Locations,json=locations" json:"Locations,omitempty"`
	Vnodes    []*chord.Vnode `protobuf:"bytes,2,rep,name=Vnodes,json=vnodes" json:"Vnodes,omitempty"`
}

func (*LookupResponse) Descriptor 

func (*LookupResponse) Descriptor() ([]byte, []int)

func (*LookupResponse) GetLocations 

func (m *LookupResponse) GetLocations() []*Location

func (*LookupResponse) GetVnodes 

func (m *LookupResponse) GetVnodes() []*chord.Vnode

func (*LookupResponse) ProtoMessage 

func (*LookupResponse) ProtoMessage()

func (*LookupResponse) Reset 

func (m *LookupResponse) Reset()

func (*LookupResponse) String 

func (m *LookupResponse) String() string

type NetClient 

type NetClient struct {
	// contains filtered or unexported fields
}

NetClient provides RPC calls to the ring

func NewNetClient 

func NewNetClient(reapInterval, maxIdle time.Duration) *NetClient

NewNetClient instantiates a new NetClient. It takes the max connection idle time as an argument

func (*NetClient) Lookup 

func (client *NetClient) Lookup(host string, n int32, key []byte) ([]*chord.Vnode, error)

Lookup performs a key lookup on a host

func (*NetClient) LookupHash 

func (client *NetClient) LookupHash(host string, n int32, hash []byte) ([]*chord.Vnode, error)

LookupHash performs a LookupHash on a host

func (*NetClient) LookupReplicated 

func (client *NetClient) LookupReplicated(host string, key []byte, n int32) ([]*Location, error)

LookupReplicated performs LookupReplicated request on a host

func (*NetClient) LookupReplicatedHash 

func (client *NetClient) LookupReplicatedHash(host string, hash []byte, n int32) ([]*Location, error)

LookupReplicatedHash lookups a hash and its replicas against the given host

func (*NetClient) Shutdown 

func (client *NetClient) Shutdown()

Shutdown stops reaping connections and disabled getting any new connections

type NetTransport 

type NetTransport struct {
	// contains filtered or unexported fields
}

NetTransport implements the server side lookup interface

func NewNetTransport 

func NewNetTransport(r *Ring) *NetTransport

NewNetTransport instantiates a new network transport to serve client ring requests

func (*NetTransport) LookupHashRPC 

func (trans *NetTransport) LookupHashRPC(ctx context.Context, req *LookupRequest) (*LookupResponse, error)

LookupHashRPC serves a LookupHash request

func (*NetTransport) LookupRPC 

func (trans *NetTransport) LookupRPC(ctx context.Context, req *LookupRequest) (*LookupResponse, error)

LookupRPC serves a Lookup request

func (*NetTransport) LookupReplicatedHashRPC 

func (trans *NetTransport) LookupReplicatedHashRPC(ctx context.Context, req *LookupRequest) (*LookupResponse, error)

LookupReplicatedHashRPC serves a LookupReplicatedHash request

func (*NetTransport) LookupReplicatedRPC 

func (trans *NetTransport) LookupReplicatedRPC(ctx context.Context, req *LookupRequest) (*LookupResponse, error)

LookupReplicatedRPC serves a LookupReplicated request

func (*NetTransport) RegisterServer 

func (trans *NetTransport) RegisterServer(server *grpc.Server)

type Peer 

type Peer struct {
	Address  string
	LastSeen uint64
}

Peer contains peer contact information

type PeerJSONStore 

type PeerJSONStore struct {
	*InMemPeerStore
	// contains filtered or unexported fields
}

PeerJSONStore implements a json file based PeerStore interface it inherits the in-memory interface for caching

func NewPeerJSONStore 

func NewPeerJSONStore(filename string) (*PeerJSONStore, error)

NewPeerJSONStore implements a JSON PeerStore with an in-memory store for caching

func (*PeerJSONStore) AddPeer 

func (ps *PeerJSONStore) AddPeer(peer string) bool

AddPeer adds a peer to the json store

func (*PeerJSONStore) Commit 

func (ps *PeerJSONStore) Commit() error

Commit writes the in-memory peer list to the stable store.

type PeerStore 

type PeerStore interface {
	Peers() []string
	AddPeer(string) bool
	RemovePeer(string)
}

PeerStore implements a peer store interface

type Ring 

type Ring struct {
	*chord.Ring // Underlying chord ring
	// contains filtered or unexported fields
}

Ring is a node part of the chord ring allowing to perform ring operations. This is used on peers participating in the ring.

func New 

func New(conf *chord.Config, peers PeerStore, trans *chord.GRPCTransport) *Ring

New instantiates a new ring func New(conf *Config, peers PeerStore, rpcTimeout, maxConnIdle time.Duration) *Ring {

func (*Ring) Create 

func (r *Ring) Create() error

Create creates a new ring. This is only to be called once.

func (*Ring) Hostname 

func (r *Ring) Hostname() string

Hostname returns the hostname of the node per the config.

func (*Ring) Join 

func (r *Ring) Join() error

Join tries to join an existing ring using any of the peers from the PeerStore.

func (*Ring) LookupReplicated 

func (r *Ring) LookupReplicated(key []byte, n int) (LocationSet, error)

LookupReplicated returns vnodes where a key and n replicas are located.

func (*Ring) LookupReplicatedHash 

func (r *Ring) LookupReplicatedHash(hash []byte, n int) (LocationSet, error)

LookupReplicatedHash returns vnodes where a key and n replicas are located. Each replica call is performed in its own go-routine. Each replica returned is a unique node. It returns a n error if the lookup fails or enough unique nodes are not found.

func (*Ring) LookupReplicatedHashSerial 

func (r *Ring) LookupReplicatedHashSerial(hash []byte, n int) (LocationSet, error)

LookupReplicatedHashSerial returns vnodes where a key and n replicas are located. Each replica returned is a unique node. It returns a n error if the lookup fails or enough unique nodes are not found.

func (*Ring) NumSuccessors 

func (r *Ring) NumSuccessors() int

NumSuccessors returns the num of succesors per the config.

func (*Ring) RegisterServer 

func (r *Ring) RegisterServer(server *grpc.Server)

RegisterServer registers the underlying transport to the grpc server

func (*Ring) RetryJoin 

func (r *Ring) RetryJoin() error

RetryJoin keeps looping through the available peers to join. It implements a backoff for each retry

func (*Ring) Scour 

func (r *Ring) Scour(locs LocationSet, cb func(*chord.Vnode) error) (int, error)

Scour traverses each location up to the allowed number of succesors, issueing the callback for each node. It skips nodes that have already been visited. If the callback returns an error, it is immediately exits. It returns the number of nodes visited and/or an error either from the lookup or callback.

func (*Ring) ScourReplica 

func (r *Ring) ScourReplica(locID []byte, cb func(*chord.Vnode) error) (int, error)

ScourReplica scours a replica location id upto the allowable number of vnodes.

func (*Ring) ScourReplicatedKey 

func (r *Ring) ScourReplicatedKey(key []byte, replicas int, cb func(*chord.Vnode) error) (int, error)

ScourReplicatedKey finds the replca hash locations for the given key and calls Scour on each location. This is a helper function to Scour.

func (*Ring) ScourSector 

func (r *Ring) ScourSector(start, end []byte, cb func(*chord.Vnode) error) (int, error)

ScourSector scours all nodes between start and end hashes issueing the callback for the chosen vnodes. It skips nodes that have already been visited.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.