README
¶
Dave - Distributed Key-Value Store
Dave is a peer-to-peer key-value store built on UDP, designed for efficient data distribution and resource allocation using XOR metric and proof-of-work.
Core Features
- Distributed key-value storage over UDP
- XOR metric used to prioritise replicas
- Random storage challenges measure peer reliability
- Sharding for concurrent processing
- Automatic peer discovery
- Data backup and recovery
- Just two dependencies:
Architecture
Network Protocol
- Maximum Packet Size: 1424 bytes to avoid fragmentation
- Transport: UDP with optimised serialization (x4 faster than protobuf)
- Peer Management: Dynamic peer discovery with trust scoring
- Data Distribution: XOR metric for deterministic propagation
Storage System
- Sharding: Concurrent processing with configurable shard capacity
- Data Prioritization: XOR metric and time-bound storage prioritization
- Backup: Automatic data persistence and recovery with configurable backup files
Configuration
type DaveCfg struct {
// A UDP socket. Normally from net.ListenUDP. This interface can be mocked
// to build simulations.
Socket pkt.Socket
// Node private key. The last 32 bytes are the public key. The node ID is
// derived from the first 8 bytes of the public key.
PrivateKey ed25519.PrivateKey
Edges []netip.AddrPort // Bootstrap peers.
ShardCapacity int64 // Capacity of each of 256 shards in bytes.
// Time-to-live of data. Data older than this will be replaced as needed,
// if new data has a higher priority. Priority is a function of age and
// XOR distance.
TTL time.Duration
BackupFilename string // Filename of backup file. Leave blank to disable backup.
// Set to nil to disable logging, although this is not reccomended. Currently
// logging is the best way to monitor. In future, the API will be better.
Logger logger.Logger
}
Protocol Operations
- PING/PONG: Peer liveness and discovery
- PUT: Store data with proof-of-work
- GET/GET_ACK: Retrieve stored data
- GETMYADDRPORT/GETMYADDRPORT_ACK: Get own address:port from a remote
Trust System
- Reliability is measured using random storage challenges
- No gossip about reliability to prevent attack vectors
- Reliability influences peer selection
- Reliability influences re-propagation
Pruning and Maintenance
- Periodic pruning of inactive peers
- Concurrent shard processing
- Automatic backup management
Security Features
- Ed25519 signatures for data authenticity
- Proof-of-work for spam prevention
- Trust-based peer selection
- Loopback prevention
- Peer table poisoning (and by extension Eclipse attack) prevention
Edge Nodes
Edge nodes serve as bootstrap peers, and are permanently retained in the peer table.
Performance Considerations
- Concurrent packet processing
- Concurrent shard processing
- Configurable pruning intervals
- Priority heap for data prioritisation, O(log n) inserts
Documentation
¶
Overview ¶
This file contains the methods and tyoes of the main package that are not exposed.
This file contains the publicly exposed methods and types of the main package.
Index ¶
- type Dave
- func (d *Dave) ActivePeerCount() int
- func (d *Dave) BatchWriter(publicKey ed25519.PublicKey) (chan<- dat.Dat, <-chan error, error)
- func (d *Dave) Capacity() int64
- func (d *Dave) Get(ctx context.Context, get *types.Get) (*store.Entry, error)
- func (d *Dave) Kill()
- func (d *Dave) NetworkUsedSpaceAndCapacity() (usedSpace, capacity uint64)
- func (d *Dave) UsedSpace() int64
- func (d *Dave) WaitForActivePeers(ctx context.Context, count int) error
- type DaveCfg
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Dave ¶
type Dave struct {
// contains filtered or unexported fields
}
func (*Dave) ActivePeerCount ¶ added in v0.0.51
ActivePeerCount returns the number of peers that are currently activated.
func (*Dave) BatchWriter ¶ added in v0.0.51
BatchWriter finds the peers closest to the given public key, and creates TCP connections to them. A channel is returned on which the caller can send signed and proven Dats. The caller must close the channel, which will flush the buffers, and close the TCP connections.
func (*Dave) Capacity ¶ added in v0.0.51
Capacity returns the maximum memory allowance that the local store may use.
func (*Dave) Get ¶ added in v0.0.23
Get returns a store entry, or an error. If not found locally, requests will be sent to the peers that are closest to the given public key. Use the context to implement a timeout.
func (*Dave) Kill ¶ added in v0.0.48
func (d *Dave) Kill()
Kill initiates a graceful shutdown. This is important, as the store uses a buffer to write to the filesystem. This buffer must be flushed to prevent loss of the data in the buffer.
func (*Dave) NetworkUsedSpaceAndCapacity ¶ added in v0.0.51
NetworkUsedSpaceAndCapacity returns the used space and capacity of the network. This value is only an approximation. There is currently no way to know if nodes are lying. Nodes are not incentivised to lie, because there is no reward or penalty either way.
type DaveCfg ¶ added in v0.0.51
type DaveCfg struct {
// UDP listen address:port. IP must be a pure IPv6 address. For localhost, use [::1].
// A TCP listener will be created on the next port, for example if this is on port 40,
// dave will listen for TCP connections on port 41.
UdpListenAddr *net.UDPAddr
// Node private key. The last 32 bytes are the public key. The node ID is
// derived from the first 8 bytes of the public key.
PrivateKey ed25519.PrivateKey
Edges []netip.AddrPort // Bootstrap peers.
ShardCapacity int64 // Capacity of each of 256 shards in bytes.
BackupFilename string // Filename of backup file. Leave blank to disable backup.
// Set to nil to disable logging, although this is not reccomended. Currently
// logging is the best way to monitor. In future, the API will be better.
Logger logger.Logger
}
Source Files
Directories