fidias

package module
v0.1.9 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 7, 2017 License: MPL-2.0 Imports: 27 Imported by: 0

README

fidias Build Status

Table of Contents

Installation

  1. Download a pre-compiled binary from the releases page.

  2. Extract the binary using unzip or tar depending on the one downloaded.

  3. Move the binary into your $PATH.

Start a cluster

The default cluster requires a minimum of 3 nodes to function though this can be changed in the configuration. Below are the steps to spin up a local test cluster.

1. Start the first node in a terminal:

$ fidiasd -debug -bind-addr 127.0.0.1:54321 -http-addr 127.0.0.1:9090

2. Start 2 or more nodes - each in separate terminals. Change the addresses to appropriately match your configuration.

$ fidiasd -bind-addr 127.0.0.1:54322 -http-addr 127.0.0.1:9091 -join 127.0.0.1:54321 -debug
$ fidiasd -bind-addr 127.0.0.1:54323 -http-addr 127.0.0.1:9092 -join 127.0.0.1:54321 -debug
...

3. You should start seeing peers joining the cluster. To confirm the cluster is functional, perform a locate call and ensure it responds with locations. Here's a sample of what the request and response would look like:

$ curl -XGET http://127.0.0.1:9090/v1/locate/testkey
[
  {
    "ID": "7af630cfe6d6c180dc56caabc76c36965185eabbd71c0a0b4ef800298d147816",
    "Priority": 0,
    "Index": 0,
    "Vnode": {
      "ID": "7d9b8af0cdd93e56e846a13435453525e6a88568474c53675a6084a3d0ad1886",
      "Host": "172.19.0.7:32100",
      "Meta": {
        "http": "127.0.0.1:7700"
      }
    }
  },
  {
    "ID": "d04b86253c2c16d631ac20011cc18beba6db40112c715f60a44d557ee269cd6b",
    "Priority": 1,
    "Index": 0,
    "Vnode": {
      "ID": "d09c14c4e817d5584cda42d1d60fb389c87067ab712f1a91d9e243ff6696c393",
      "Host": "172.19.0.8:32100",
      "Meta": {
        "http": "127.0.0.1:7703"
      }
    }
  },
  {
    "ID": "25a0db7a91816c2b870175567216e140fc30956681c6b4b5f9a2aad437bf22c0",
    "Priority": 2,
    "Index": 2,
    "Vnode": {
      "ID": "4aef6d0fe81410d02859717b124e52971d28eb3f4b3eba3cf1c4c5f579a3594e",
      "Host": "172.19.0.2:32100",
      "Meta": {
        "http": "127.0.0.1:7705"
      }
    }
  }
]

The cluster is now running and can be used. Details on the HTTP API can be found in the API docs.

Development
  • When using debug mode a significant performance degrade may be seen.
Known Issues
  • When using fidias in docker on a Mac with persistent storage, a massive performance hit is incurred due to the way docker volumes and persistence are managed by docker on a Mac. This is only pertinent for Macs

Documentation

Overview

Package fidias is a generated protocol buffer package.

It is generated from these files: 

rpc.proto

It has these top-level messages: 

KeyLocation
KeyValuePair
FileVersion
PathRPC

Index

Constants

View Source 
const (
	// OpSet is the op to set a ke-value pair
	OpSet byte = iota + 1
	// OpDel is the op to delete a key-value pair
	OpDel
)
View Source 
const (
	// OpFsSet is used to set a path in the fsm
	OpFsSet byte = iota + 10
	// OpFsDel is used to delete a path from the fsm
	OpFsDel
)

Variables

View Source 
var (
	// ErrVersionNotFound is used when a file version is not found
	ErrVersionNotFound = errors.New("version not found")
	// ErrVersionExists is used when a new version being created has
	// the same name as an already existing one
	ErrVersionExists = errors.New("version exists")
)

Functions

func RegisterFidiasRPCServer 

func RegisterFidiasRPCServer(s *grpc.Server, srv FidiasRPCServer)

Types

type Config 

type Config struct {
	Version string // This is internally set

	Ring            *chord.Config
	Hexalog         *hexalog.Config
	RelocateBufSize int           // Relocate request buffer size
	RetryInterval   time.Duration // interval to wait before retrying a proposal

	//StableThreshold time.Duration // Threshold after ring event to consider we are stable
	KeyValueNamespace   string
	FileSystemNamespace string
	// Web UI directory
	UIDir string
}

Config hold the guac config along with the underlying log and ring config

func DefaultConfig 

func DefaultConfig(hostname string) *Config

DefaultConfig returns a default sane config setting the hostname on the log and ring configs

func (*Config) Hasher added in v0.1.2 

func (conf *Config) Hasher() hexatype.Hasher

Hasher returns the log hasher. This is a helper function

func (*Config) Hostname 

func (conf *Config) Hostname() string

Hostname returns the configured hostname. The assumption here is the log and ring hostnames are the same as they should be checked and set prior to using this call

type DHT added in v0.1.4 

type DHT interface {
	LookupReplicated(key []byte, replicas int) (hexaring.LocationSet, error)
	LookupReplicatedHash(hash []byte, replicas int) (hexaring.LocationSet, error)
	ScourReplicatedKey(key []byte, replicas int, cb func(*chord.Vnode) error) (int, error)
}

type FSM added in v0.1.4 

type FSM struct {
	// contains filtered or unexported fields
}

FSM is a hexalog FSM for an in-memory key-value store. It implements the FSM interface and provides a get function to retrieve keys as all write are handled by the FSM

func NewFSM added in v0.1.4 

func NewFSM(kvprefix, fsprefix string) *FSM

NewFSM inits a new FSM

func (*FSM) Apply added in v0.1.4 

func (fsm *FSM) Apply(entryID []byte, entry *hexatype.Entry) interface{}

Apply applies the given entry to the FSM. entryID is the hash id of the entry. The first byte in entry.Data contains the operation to be performed followed by the actual value.

func (*FSM) Close added in v0.1.4 

func (fsm *FSM) Close() error

Close is a no-op to satisfy the KeyValueFSM interface

func (*FSM) GetKey added in v0.1.4 

func (fsm *FSM) GetKey(key []byte) (*KeyValuePair, error)

GetKey gets a value for the key. It reads it directly from the stored log entry

func (*FSM) GetPath added in v0.1.4 

func (fsm *FSM) GetPath(name string) (*VersionedFile, error)

GetPath returns a path with pointers to all of its versions

func (*FSM) Open added in v0.1.4 

func (fsm *FSM) Open() error

Open initialized the internal data structures. It always returns nil

type Fetcher added in v0.1.4 

type Fetcher struct {
	// contains filtered or unexported fields
}

Fetcher manages fetching block and log entries from the network

func NewFetcher added in v0.1.4 

func NewFetcher(idx hexalog.IndexStore, ent hexalog.EntryStore, replicas, bufSize int, hasher hexatype.Hasher) *Fetcher

NewFetcher inits a Fetcher with the given options.

func (*Fetcher) RegisterBlockTransport added in v0.1.4 

func (fet *Fetcher) RegisterBlockTransport(blks blox.Transport)

RegisterBlockTransport registers a transport for block fetching

func (*Fetcher) RegisterDHT added in v0.1.4 

func (fet *Fetcher) RegisterDHT(dht DHT)

RegisterDHT registers the DHT to the fetcher and starts the fetch loop. This must be called after the transport and healer interfaces have been registered.

func (*Fetcher) RegisterHealer added in v0.1.4 

func (fet *Fetcher) RegisterHealer(healer Healer)

RegisterHealer registers the log healer to the fetcher to submit heal requests

func (*Fetcher) RegisterTransport added in v0.1.4 

func (fet *Fetcher) RegisterTransport(trans FetcherTransport)

RegisterTransport registers a transport for log fetching

type FetcherTransport added in v0.1.4 

type FetcherTransport interface {
	FetchKeylog(host string, entry *hexatype.Entry, opts *hexatype.RequestOptions) (*hexalog.FutureEntry, error)
}

FetcherTransport implements a tranport interface to fetch all entries from a given entry down

type Fidias 

type Fidias struct {
	// contains filtered or unexported fields
}

Fidias is the core that manages all operations for a node. It primary manages rebalancing, replication, and appropriately deals with cluster churn.

func New 

func New(conf *Config, hexlog *Hexalog, fsm *FSM, relocator *Relocator, fetcher *Fetcher,
	keyvs *Keyvs, dev *RingDevice, trans *NetTransport) *Fidias

New instantiates a new instance of Fidias based on the given config and stores along with a grpc server instance to register the network transports

func (*Fidias) FileSystem added in v0.1.4 

func (fids *Fidias) FileSystem() *FileSystem

FileSystem returns the fidias file-system

func (*Fidias) Leaving 

func (fidias *Fidias) Leaving(local, pred, succ *chord.Vnode)

Leaving is called by the Ring when this node willingly leaves. This is only triggered if an explicit leave is issued

func (*Fidias) NewPredecessor 

func (fidias *Fidias) NewPredecessor(local, newPred, oldPred *chord.Vnode)

NewPredecessor is called when a local vnode finds a new predecessor. This causes a rebalance of keys. All key hashes less than the new predecessor are transferred to the new predecessor.

func (*Fidias) PredecessorLeaving 

func (fidias *Fidias) PredecessorLeaving(local, remote *chord.Vnode)

PredecessorLeaving is only triggered if an explicit leave is issued

func (*Fidias) Register 

func (fids *Fidias) Register(ring *hexaring.Ring)

Register registers the chord ring to fidias. This is due to the fact that guac and the ring depend on each other and the ring may not be intialized yet. Only upon ring registration, the rebalancing is started.

func (*Fidias) Shutdown 

func (fidias *Fidias) Shutdown()

Shutdown is called but a chord node is shutdown

func (*Fidias) Status 

func (fidias *Fidias) Status() *Status

Status returns the status of this node

func (*Fidias) SuccessorLeaving 

func (fidias *Fidias) SuccessorLeaving(local, remote *chord.Vnode)

SuccessorLeaving is only triggered if an explicit leave is issued

type FidiasRPCClient 

type FidiasRPCClient interface {
	// Get key-value pair from remote
	GetKeyRPC(ctx context.Context, in *KeyValuePair, opts ...grpc.CallOption) (*KeyValuePair, error)
	// Get versioned path from remote
	GetPathRPC(ctx context.Context, in *PathRPC, opts ...grpc.CallOption) (*PathRPC, error)
	// Send key-location id pairs
	RelocateRPC(ctx context.Context, opts ...grpc.CallOption) (FidiasRPC_RelocateRPCClient, error)
	// Send block id and journal value
	RelocateBlocksRPC(ctx context.Context, opts ...grpc.CallOption) (FidiasRPC_RelocateBlocksRPCClient, error)
}

func NewFidiasRPCClient 

func NewFidiasRPCClient(cc *grpc.ClientConn) FidiasRPCClient

type FidiasRPCServer 

type FidiasRPCServer interface {
	// Get key-value pair from remote
	GetKeyRPC(context.Context, *KeyValuePair) (*KeyValuePair, error)
	// Get versioned path from remote
	GetPathRPC(context.Context, *PathRPC) (*PathRPC, error)
	// Send key-location id pairs
	RelocateRPC(FidiasRPC_RelocateRPCServer) error
	// Send block id and journal value
	RelocateBlocksRPC(FidiasRPC_RelocateBlocksRPCServer) error
}

type FidiasRPC_RelocateBlocksRPCClient added in v0.1.4 

type FidiasRPC_RelocateBlocksRPCClient interface {
	Send(*KeyLocation) error
	Recv() (*KeyLocation, error)
	grpc.ClientStream
}

type FidiasRPC_RelocateBlocksRPCServer added in v0.1.4 

type FidiasRPC_RelocateBlocksRPCServer interface {
	Send(*KeyLocation) error
	Recv() (*KeyLocation, error)
	grpc.ServerStream
}

type FidiasRPC_RelocateRPCClient added in v0.1.2 

type FidiasRPC_RelocateRPCClient interface {
	Send(*KeyLocation) error
	Recv() (*KeyLocation, error)
	grpc.ClientStream
}

type FidiasRPC_RelocateRPCServer added in v0.1.2 

type FidiasRPC_RelocateRPCServer interface {
	Send(*KeyLocation) error
	Recv() (*KeyLocation, error)
	grpc.ServerStream
}

type File added in v0.1.4 

type File struct {
	*filesystem.BloxFile
	// contains filtered or unexported fields
}

File is a fidias file representing a standard OS file type interface. It contains methods to perform native go file type operations

func (*File) Close added in v0.1.4 

func (file *File) Close() error

Close closes the underlying BloxFile and updates hexalog with the new hash entries

func (*File) Name added in v0.1.4 

func (file *File) Name() string

Name returns the absolute path name of the file

func (*File) Versions added in v0.1.5 

func (file *File) Versions() *VersionedFile

Versions returns the underlying VersionedFile instance

type FileSystem added in v0.1.4 

type FileSystem struct {
	// contains filtered or unexported fields
}

FileSystem represents a fidias filesystem

func NewFileSystem added in v0.1.4 

func NewFileSystem(host, namespace string, dev *RingDevice, hexlog *Hexalog, verfs VersionedFileStore) *FileSystem

NewFileSystem inits a new FileSystem instance. There can be as many instances needed. namespace is used to prefix all keys.

func (*FileSystem) Create added in v0.1.4 

func (fs *FileSystem) Create(name string) (*File, error)

Create creates a new file

func (*FileSystem) GetVersions added in v0.1.5 

func (fs *FileSystem) GetVersions(name string) (*VersionedFile, error)

GetVersions gets the VersionedFile associated to the provided file name.

func (*FileSystem) Mkdir added in v0.1.5 

func (fs *FileSystem) Mkdir(name string) error

Mkdir creates a new directory by submitting a log entry

func (*FileSystem) Open added in v0.1.4 

func (fs *FileSystem) Open(name string) (*File, error)

Open opens the active version of the named file for reading. If successful, methods on the returned file can be used for reading; the associated file descriptor has mode O_RDONLY.

func (*FileSystem) RegisterDHT added in v0.1.4 

func (fs *FileSystem) RegisterDHT(dht DHT)

RegisterDHT registers the DHT for lookups

func (*FileSystem) RegisterTransport added in v0.1.4 

func (fs *FileSystem) RegisterTransport(remote FileSystemTransport)

RegisterTransport registers a network transport for the filesystem used to get remote paths

func (*FileSystem) Stat added in v0.1.4 

func (fs *FileSystem) Stat(name string) (os.FileInfo, error)

Stat performs a stat call on the file returning a standard os.FileInfo object

type FileSystemFSM added in v0.1.4 

type FileSystemFSM interface {
	// Get a VersionedFile by name
	Get(name string) (*VersionedFile, error)
	// ApplySet is called when an entry needs to be applied. It is called
	// with the entry and the extracted value from the entry. It should
	// use the value bytes as the data payload
	ApplySet(entryID []byte, entry *hexatype.Entry, value []byte) error
	// ApplyDelete is called when a delete entry to needs to be applied
	// It should remove the key and all versions given by the entry key
	ApplyDelete(entry *hexatype.Entry) error
}

FileSystemFSM implements an FSM to manage a versioned file-system. It is responsible for applying log entries to provide a VersionedFile file-system view.

type FileSystemTransport added in v0.1.4 

type FileSystemTransport interface {
	GetPath(ctx context.Context, host string, name string) (*VersionedFile, error)
}

type FileVersion added in v0.1.4 

type FileVersion struct {
	Alias string `protobuf:"bytes,1,opt,name=Alias" json:"Alias,omitempty"`
	ID    []byte `protobuf:"bytes,2,opt,name=ID,proto3" json:"ID,omitempty"`
}

func (*FileVersion) Descriptor added in v0.1.4 

func (*FileVersion) Descriptor() ([]byte, []int)

func (*FileVersion) GetAlias added in v0.1.4 

func (m *FileVersion) GetAlias() string

func (*FileVersion) GetID added in v0.1.4 

func (m *FileVersion) GetID() []byte

func (*FileVersion) MarshalJSON added in v0.1.5 

func (ver *FileVersion) MarshalJSON() ([]byte, error)

MarshalJSON marshals a file version accounting for hash ids

func (*FileVersion) ProtoMessage added in v0.1.4 

func (*FileVersion) ProtoMessage()

func (*FileVersion) Reset added in v0.1.4 

func (m *FileVersion) Reset()

func (*FileVersion) String added in v0.1.4 

func (m *FileVersion) String() string

func (*FileVersion) Text added in v0.1.4 

func (ver *FileVersion) Text() string

Text returns the text string representation of the file version

type Healer added in v0.1.4 

type Healer interface {
	Heal(key []byte, opts *hexatype.RequestOptions) error
}

Healer implements an interface to submit heal requests for a given key.

type Hexalog added in v0.1.4 

type Hexalog struct {
	// contains filtered or unexported fields
}

Hexalog is a ring/cluster aware Hexalog.

func NewHexalog added in v0.1.4 

func NewHexalog(conf *Config, logstore *hexalog.LogStore, stable hexalog.StableStore, f *FSM, remote *hexalog.NetTransport) (*Hexalog, error)

NewHexalog inits a new fidias hexalog instance attached to the ring. Remote must be registered to grpc before init'ing hexalog

func (*Hexalog) GetEntry added in v0.1.4 

func (hexlog *Hexalog) GetEntry(key, id []byte) (entry *hexatype.Entry, meta *ReMeta, err error)

GetEntry tries to get an entry from the ring. It gets the replica locations and queries upto the max allowed successors for each location.

func (*Hexalog) Heal added in v0.1.4 

func (hexlog *Hexalog) Heal(key []byte, opts *hexatype.RequestOptions) error

Heal submits a heal request for the given key to the local note. It consults the supplied PeerSet in order to perform the heal.

func (*Hexalog) Leader added in v0.1.4 

func (hexlog *Hexalog) Leader(key []byte, locs hexaring.LocationSet) (*hexalog.KeyLeader, error)

Leader returns the leader of the given location set from the underlying log.

func (*Hexalog) MinVotes added in v0.1.4 

func (hexlog *Hexalog) MinVotes() int

MinVotes returns the minimum number of required votes for a proposal and commit

func (*Hexalog) NewEntry added in v0.1.4 

func (hexlog *Hexalog) NewEntry(key []byte) (*hexatype.Entry, *hexatype.RequestOptions, error)

NewEntry returns a new Entry for the given key from Hexalog. It returns an error if the node is not part of the location set or a lookup error occurs

func (*Hexalog) NewEntryFrom added in v0.1.4 

func (hexlog *Hexalog) NewEntryFrom(entry *hexatype.Entry) (*hexatype.Entry, *hexatype.RequestOptions, error)

NewEntryFrom creates a new entry based on the given entry. It uses the given height and previous hash of the entry to determine the values for the new entry. This is essentially a compare and set

func (*Hexalog) ProposeEntry added in v0.1.4 

func (hexlog *Hexalog) ProposeEntry(entry *hexatype.Entry, opts *hexatype.RequestOptions) (err error)

ProposeEntry finds locations for the entry and proposes it to those locations. It retries the specified number of times before returning. It returns a ballot that can be waited on for the entry to be applied or an error

func (*Hexalog) RegisterDHT added in v0.1.4 

func (hexlog *Hexalog) RegisterDHT(dht DHT)

RegisterDHT registers the DHT to hexalog

type InMemKeyValueFSM 

type InMemKeyValueFSM struct {
	// contains filtered or unexported fields
}

func NewInMemKeyValueFSM 

func NewInMemKeyValueFSM(prefix string) *InMemKeyValueFSM

func (*InMemKeyValueFSM) ApplyDelete added in v0.1.4 

func (fsm *InMemKeyValueFSM) ApplyDelete(entry *hexatype.Entry) error

func (*InMemKeyValueFSM) ApplySet added in v0.1.4 

func (fsm *InMemKeyValueFSM) ApplySet(entryID []byte, entry *hexatype.Entry, value []byte) error

func (*InMemKeyValueFSM) Get 

func (fsm *InMemKeyValueFSM) Get(key []byte) (*KeyValuePair, error)

type InMemVersionedFileFSM added in v0.1.4 

type InMemVersionedFileFSM struct {
	// contains filtered or unexported fields
}

InMemVersionedFileFSM is a VersionedFile FSM

func NewInMemVersionedFileFSM added in v0.1.4 

func NewInMemVersionedFileFSM(prefix string) *InMemVersionedFileFSM

NewInMemVersionedFileFSM inits a new in-memory VersionedFile fsm. It takes a prefix that must be present in hexalog used to trim the received entry key to obtain the file path.

func (*InMemVersionedFileFSM) ApplyDelete added in v0.1.4 

func (store *InMemVersionedFileFSM) ApplyDelete(entry *hexatype.Entry) error

ApplyDelete applies a delete fsm operation for VersionedFiles. This is not to be directly used or called. It is called by the managing parent fsm when a fs delete operation entry is received.

func (*InMemVersionedFileFSM) ApplySet added in v0.1.4 

func (store *InMemVersionedFileFSM) ApplySet(entryID []byte, entry *hexatype.Entry, value []byte) error

ApplySet applies a set fsm operation for VersionedFiles. This is not to be directly used or called. It is called by the managing parent fsm when a fs set operation entry is received by the managing parent fsm.

func (*InMemVersionedFileFSM) Get added in v0.1.4 

func (store *InMemVersionedFileFSM) Get(name string) (*VersionedFile, error)

Get returns the VersionedFile by the given name

type KeyLocation added in v0.1.2 

type KeyLocation struct {
	// Actual key
	Key []byte `protobuf:"bytes,1,opt,name=Key,proto3" json:"Key,omitempty"`
	// Location id or last entry or id
	Marker []byte `protobuf:"bytes,2,opt,name=Marker,proto3" json:"Marker,omitempty"`
	// Total known height of the key
	Height uint32 `protobuf:"varint,3,opt,name=Height" json:"Height,omitempty"`
}

func (*KeyLocation) Descriptor added in v0.1.2 

func (*KeyLocation) Descriptor() ([]byte, []int)

func (*KeyLocation) GetHeight added in v0.1.2 

func (m *KeyLocation) GetHeight() uint32

func (*KeyLocation) GetKey added in v0.1.2 

func (m *KeyLocation) GetKey() []byte

func (*KeyLocation) GetMarker added in v0.1.2 

func (m *KeyLocation) GetMarker() []byte

func (*KeyLocation) ProtoMessage added in v0.1.2 

func (*KeyLocation) ProtoMessage()

func (*KeyLocation) Reset added in v0.1.2 

func (m *KeyLocation) Reset()

func (*KeyLocation) String added in v0.1.2 

func (m *KeyLocation) String() string

type KeyValueFSM 

type KeyValueFSM interface {
	// Get a key
	Get(key []byte) (*KeyValuePair, error)
	// Apply a set operation entry with value containing the data
	ApplySet(entryID []byte, entry *hexatype.Entry, value []byte) error
	// Apply a delete entry
	ApplyDelete(entry *hexatype.Entry) error
}

KeyValueFSM is an FSM for a key value store. Aside from fsm functions, it also contains read-only key-value functions needed.

type KeyValuePair 

type KeyValuePair struct {
	Key   []byte          `protobuf:"bytes,1,opt,name=Key,proto3" json:"Key,omitempty"`
	Value []byte          `protobuf:"bytes,2,opt,name=Value,proto3" json:"Value,omitempty"`
	Entry *hexatype.Entry `protobuf:"bytes,3,opt,name=Entry" json:"Entry,omitempty"`
}

func (*KeyValuePair) Descriptor 

func (*KeyValuePair) Descriptor() ([]byte, []int)

func (*KeyValuePair) GetEntry 

func (m *KeyValuePair) GetEntry() *hexatype.Entry

func (*KeyValuePair) GetKey 

func (m *KeyValuePair) GetKey() []byte

func (*KeyValuePair) GetValue 

func (m *KeyValuePair) GetValue() []byte

func (KeyValuePair) MarshalJSON added in v0.1.4 

func (kvp KeyValuePair) MarshalJSON() ([]byte, error)

MarshalJSON is a custom marshaller to handle the entry key

func (*KeyValuePair) ProtoMessage 

func (*KeyValuePair) ProtoMessage()

func (*KeyValuePair) Reset 

func (m *KeyValuePair) Reset()

func (*KeyValuePair) String 

func (m *KeyValuePair) String() string

type KeyValueStore 

type KeyValueStore interface {
	GetKey(key []byte) (*KeyValuePair, error)
}

KeyValueStore implements a key value storage interface. It is used by the network transport

type KeyValueTransport added in v0.1.4 

type KeyValueTransport interface {
	GetKey(ctx context.Context, host string, key []byte) (*KeyValuePair, error)
}

KeyValueTransport implements a transport for key-value operations

type Keyvs added in v0.1.4 

type Keyvs struct {
	// contains filtered or unexported fields
}

Keyvs is a key-value interface that relies on Hexalog and Hexaring to provide functions to perform CRUD like operations on keys

func NewKeyvs added in v0.1.4 

func NewKeyvs(namespace string, hexlog *Hexalog, kvs KeyValueStore) *Keyvs

NewKeyvs inits a new instance of Keyvs. It takes the hexalog for write ops, key-value store and network transport for read ops. namespace is used to prefix all keys.

func (*Keyvs) GetKey added in v0.1.4 

func (kv *Keyvs) GetKey(key []byte) (kvp *KeyValuePair, opt *hexatype.RequestOptions, err error)

GetKey requests a key from the nodes in the key peerset concurrently and returns the first non-errored result. If the key is not found in any of the locations, a ErrKeyNotFound is returned

func (*Keyvs) RegisterDHT added in v0.1.4 

func (kv *Keyvs) RegisterDHT(dht DHT)

RegisterDHT registers the ring to the keyvalue store

func (*Keyvs) RegisterTransport added in v0.1.4 

func (kv *Keyvs) RegisterTransport(remote KeyValueTransport)

RegisterTransport registers the remote transport to use.

func (*Keyvs) RemoveKey added in v0.1.4 

func (kv *Keyvs) RemoveKey(basekey []byte) (*hexatype.Entry, *hexatype.RequestOptions, error)

RemoveKey removes a key

func (*Keyvs) SetKey added in v0.1.4 

func (kv *Keyvs) SetKey(basekey, val []byte) (*hexatype.Entry, *hexatype.RequestOptions, error)

SetKey sets a key to the value

type LocalStore added in v0.1.4 

type LocalStore interface {
	KeyValueStore
	VersionedFileStore
}

LocalStore implements all local calls needed by the network transport

type NetTransport 

type NetTransport struct {
	// contains filtered or unexported fields
}

NetTransport implements a network transport needed for fidias

func NewNetTransport added in v0.1.2 

func NewNetTransport(localStore LocalStore, idx hexalog.IndexStore, journal device.Journal, reapInterval, maxIdle time.Duration, replicas int, hasher hexatype.Hasher) *NetTransport

NewNetTransport instantiates a new network transport using the given key-value store.

func (*NetTransport) GetKey 

func (trans *NetTransport) GetKey(ctx context.Context, host string, key []byte) (*KeyValuePair, error)

GetKey retrieves a key from a remote host

func (*NetTransport) GetKeyRPC 

func (trans *NetTransport) GetKeyRPC(ctx context.Context, in *KeyValuePair) (*KeyValuePair, error)

GetKeyRPC serves a GetKey request

func (*NetTransport) GetPath added in v0.1.4 

func (trans *NetTransport) GetPath(ctx context.Context, host string, name string) (*VersionedFile, error)

func (*NetTransport) GetPathRPC added in v0.1.4 

func (trans *NetTransport) GetPathRPC(ctx context.Context, in *PathRPC) (*PathRPC, error)

GetPathRPC serves a GetPath request

func (*NetTransport) GetRelocateBlocksStream added in v0.1.4 

func (trans *NetTransport) GetRelocateBlocksStream(local, remote *chord.Vnode) (*RelocateBlocksStream, error)

GetRelocateBlocksStream gets a stream to send relocation keys

func (*NetTransport) GetRelocateStream added in v0.1.2 

func (trans *NetTransport) GetRelocateStream(local, remote *chord.Vnode) (*RelocateStream, error)

GetRelocateStream gets a stream to send relocation keys

func (*NetTransport) Register added in v0.1.2 

func (trans *NetTransport) Register(fetLogCh, fetBlkCh chan<- *relocateReq)

Register registers a write channel used for submitting reloc. requests for keylogs and blocks.

func (*NetTransport) RelocateBlocksRPC added in v0.1.4 

func (trans *NetTransport) RelocateBlocksRPC(stream FidiasRPC_RelocateBlocksRPCServer) error

RelocateBlocksRPC serves a GetRelocateStream request stream. It initiates the process to start taking over the sent keys.

func (*NetTransport) RelocateRPC added in v0.1.2 

func (trans *NetTransport) RelocateRPC(stream FidiasRPC_RelocateRPCServer) error

RelocateRPC serves a GetRelocateStream request stream. It initiates the process to start taking over the sent keys.

func (*NetTransport) Shutdown 

func (trans *NetTransport) Shutdown()

Shutdown signals the transport to be shutdown. After shutdown no new connections can be

type PathRPC added in v0.1.4 

type PathRPC struct {
	Name     string          `protobuf:"bytes,1,opt,name=Name" json:"Name,omitempty"`
	Versions []*FileVersion  `protobuf:"bytes,2,rep,name=Versions" json:"Versions,omitempty"`
	Entry    *hexatype.Entry `protobuf:"bytes,3,opt,name=Entry" json:"Entry,omitempty"`
}

func (*PathRPC) Descriptor added in v0.1.4 

func (*PathRPC) Descriptor() ([]byte, []int)

func (*PathRPC) GetEntry added in v0.1.4 

func (m *PathRPC) GetEntry() *hexatype.Entry

func (*PathRPC) GetName added in v0.1.4 

func (m *PathRPC) GetName() string

func (*PathRPC) GetVersions added in v0.1.4 

func (m *PathRPC) GetVersions() []*FileVersion

func (*PathRPC) ProtoMessage added in v0.1.4 

func (*PathRPC) ProtoMessage()

func (*PathRPC) Reset added in v0.1.4 

func (m *PathRPC) Reset()

func (*PathRPC) String added in v0.1.4 

func (m *PathRPC) String() string

type ReMeta 

type ReMeta struct {
	Vnode   *chord.Vnode         // vnode processing the request or response
	PeerSet hexaring.LocationSet // set of peers involved
}

ReMeta contains metadata associated to a request or response

type RelocateBlocksStream added in v0.1.4 

type RelocateBlocksStream struct {
	FidiasRPC_RelocateBlocksRPCClient // grp stream client
	// contains filtered or unexported fields
}

func (RelocateBlocksStream) Recycle added in v0.1.4 

func (rs RelocateBlocksStream) Recycle()

Recycle recycles the stream returning the conn back to the pool

type RelocateStream added in v0.1.2 

type RelocateStream struct {
	FidiasRPC_RelocateRPCClient // grp stream client
	// contains filtered or unexported fields
}

RelocateStream is a stream to handle relocating of keys between nodes.

func (RelocateStream) Recycle added in v0.1.2 

func (rs RelocateStream) Recycle()

Recycle recycles the stream returning the conn back to the pool

type Relocator added in v0.1.2 

type Relocator struct {
	// contains filtered or unexported fields
}

Relocator is responsible for moving data as needed when the underlying cluster topology changes

func NewRelocator added in v0.1.2 

func NewRelocator(replicas int64, hasher hexatype.Hasher) *Relocator

NewRelocator instantiates a new Relocator

func (*Relocator) RegisterBlockJournal added in v0.1.4 

func (reb *Relocator) RegisterBlockJournal(journal device.Journal)

RegisterBlockJournal registers a block journal to the relocator to be used to determine which blocks need to be relocated.

func (*Relocator) RegisterKeylogIndex added in v0.1.4 

func (reb *Relocator) RegisterKeylogIndex(idx hexalog.IndexStore)

RegisterKeylogIndex register an index store of keylogs to the relocator to be used to determine the keys that need to be relocated

func (*Relocator) RegisterTransport added in v0.1.4 

func (reb *Relocator) RegisterTransport(trans RelocatorTransport)

RegisterTransport registers the transport to be used for relocation

type RelocatorTransport added in v0.1.2 

type RelocatorTransport interface {
	GetRelocateStream(local, remote *chord.Vnode) (*RelocateStream, error)
	GetRelocateBlocksStream(local, remote *chord.Vnode) (*RelocateBlocksStream, error)
}

RelocatorTransport implements a transport needed by the key rebalancing engine

type RingDevice added in v0.1.4 

type RingDevice struct {
	// contains filtered or unexported fields
}

RingDevice implements the blox.BlockDevice interface backed by hexaring to distribute blocks into the cluster. The filesystem uses this as its underlying device.

func NewRingDevice added in v0.1.4 

func NewRingDevice(replicas int, hasher hexatype.Hasher, trans blox.Transport) *RingDevice

NewRingDevice inits a new RingDevice that implements a BlockDevice with the given replica count, hash function and blox transport.

func (*RingDevice) Close added in v0.1.4 

func (dev *RingDevice) Close() error

Close shutdowns the underlying network transport

func (*RingDevice) GetBlock added in v0.1.4 

func (dev *RingDevice) GetBlock(id []byte) (block.Block, error)

GetBlock gets a block from the device

func (*RingDevice) Hasher added in v0.1.4 

func (dev *RingDevice) Hasher() hexatype.Hasher

Hasher returns the hash function generator for hash ids for the device

func (*RingDevice) RegisterDHT added in v0.1.4 

func (dev *RingDevice) RegisterDHT(dht DHT)

RegisterDHT registers the DHT ring device. This device is only usable once a call to register has been made.

func (*RingDevice) RemoveBlock added in v0.1.4 

func (dev *RingDevice) RemoveBlock(id []byte) error

RemoveBlock submits a request to remove a block on the device and all replicas

func (*RingDevice) SetBlock added in v0.1.4 

func (dev *RingDevice) SetBlock(blk block.Block) ([]byte, error)

SetBlock writes the block to the device

type Status added in v0.1.2 

type Status struct {
	Version string
	Hash    hexatype.HashAlgorithm
	DHT     *chord.Status
}

Status contains status information of a node.

type VersionedFile added in v0.1.4 

type VersionedFile struct {
	// contains filtered or unexported fields
}

VersionedFile contains all known versions for a given file and the hexalog entry associated with the view instance

func NewVersionedFile added in v0.1.4 

func NewVersionedFile(name string) *VersionedFile

NewVersionedFile instantiates a new VersionedFile with the given name

func (*VersionedFile) AddVersion added in v0.1.4 

func (f *VersionedFile) AddVersion(version *FileVersion) error

AddVersion adds a new version of the file. It returns an ErrVersionExists if the alias for the given version already exists.

func (*VersionedFile) GetVersion added in v0.1.4 

func (f *VersionedFile) GetVersion(alias string) (*FileVersion, error)

GetVersion gets a version by the given alias.

func (*VersionedFile) MarshalBinary added in v0.1.4 

func (f *VersionedFile) MarshalBinary() ([]byte, error)

MarshalBinary marshals the version into a byte slice. It does not include the key and entry

func (*VersionedFile) MarshalJSON added in v0.1.5 

func (f *VersionedFile) MarshalJSON() ([]byte, error)

MarshalJSON marshals the VersionedFile as json with the wanted private fields

func (*VersionedFile) String added in v0.1.4 

func (f *VersionedFile) String() string

func (*VersionedFile) UnmarshalBinary added in v0.1.4 

func (f *VersionedFile) UnmarshalBinary(b []byte) error

UnmarshalBinary unmarshal the byte slice into Versioned. It will not include the key and entry

func (*VersionedFile) UpdateVersion added in v0.1.4 

func (f *VersionedFile) UpdateVersion(alias string, id []byte) error

UpdateVersion updates a version by the alias. It returns an ErrVersionNotFound if the alias is not found,

func (*VersionedFile) Version added in v0.1.4 

func (f *VersionedFile) Version() *FileVersion

Version returns the active version

type VersionedFileStore added in v0.1.4 

type VersionedFileStore interface {
	GetPath(name string) (*VersionedFile, error)
}

VersionedFileStore implements a storage mechanism for versioned file paths. Each file path may point many versions containing an alias and id

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.