lstoreds

README

Datastore backed Logstore

Work pretty much inspired by go-libp2p-peerstore great implementation.

This is a go-datastore backed implementation of go-threads/core/logstore:

• AddrBook
• HeadBook
• KeyBook

For testing, two go-datastore implementation are table-tested:

• badger
• leveldb

Tests leverage the /test suites used also for in-memory implementation.

Documentation

Index

• Constants
• Variables
• func NewHeadBook(ds ds.TxnDatastore) core.HeadBook
• func NewKeyBook(store ds.Datastore) (core.KeyBook, error)
• func NewLogstore(ctx context.Context, store ds.Batching, opts Options) (core.Logstore, error)
• func NewThreadMetadata(ds ds.Datastore) core.ThreadMetadata
• type DsAddrBook
  • func NewAddrBook(ctx context.Context, ds ds.Batching, opts Options) (*DsAddrBook, error)
  • func (ab *DsAddrBook) AddAddr(t thread.ID, p peer.ID, addr ma.Multiaddr, ttl time.Duration) error
  • func (ab *DsAddrBook) AddAddrs(t thread.ID, p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) error
  • func (ab *DsAddrBook) AddrStream(ctx context.Context, t thread.ID, p peer.ID) (<-chan ma.Multiaddr, error)
  • func (ab *DsAddrBook) Addrs(t thread.ID, p peer.ID) ([]ma.Multiaddr, error)
  • func (ab *DsAddrBook) AddrsEdge(t thread.ID) (uint64, error)
  • func (ab *DsAddrBook) ClearAddrs(t thread.ID, p peer.ID) error
  • func (ab *DsAddrBook) Close() error
  • func (ab *DsAddrBook) DumpAddrs() (logstore.DumpAddrBook, error)
  • func (ab *DsAddrBook) LogsWithAddrs(t thread.ID) (peer.IDSlice, error)
  • func (ab *DsAddrBook) RestoreAddrs(dump logstore.DumpAddrBook) error
  • func (ab *DsAddrBook) SetAddr(t thread.ID, p peer.ID, addr ma.Multiaddr, ttl time.Duration) error
  • func (ab *DsAddrBook) SetAddrs(t thread.ID, p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) error
  • func (ab *DsAddrBook) ThreadsFromAddrs() (thread.IDSlice, error)
  • func (ab *DsAddrBook) UpdateAddrs(t thread.ID, p peer.ID, oldTTL time.Duration, newTTL time.Duration) error
• type Options
  • func DefaultOpts() Options

Constants

const EmptyEdgeValue uint64 = 0

Variables

var AllowEmptyRestore = false

Define if storage will accept empty dumps.

Functions

func NewHeadBook

func NewHeadBook(ds ds.TxnDatastore) core.HeadBook

NewHeadBook returns a new HeadBook backed by a datastore.

func NewKeyBook

func NewKeyBook(store ds.Datastore) (core.KeyBook, error)

NewKeyBook returns a new key book for storing public and private keys of (thread.ID, peer.ID) pairs with durable guarantees by store.

func NewLogstore

func NewLogstore(ctx context.Context, store ds.Batching, opts Options) (core.Logstore, error)

NewLogstore creates a logstore backed by the provided persistent datastore.

func NewThreadMetadata

func NewThreadMetadata(ds ds.Datastore) core.ThreadMetadata

Types

type DsAddrBook

type DsAddrBook struct {
	// contains filtered or unexported fields
}

func NewAddrBook

func NewAddrBook(ctx context.Context, ds ds.Batching, opts Options) (*DsAddrBook, error)

NewAddrBook initializes a new datastore-backed address book. It serves as a drop-in replacement for pstoremem (memory-backed peerstore), and works with any datastore implementing the ds.Batching interface.

Threads and logs addresses are serialized into protobuf, storing one datastore entry per (thread, log), along with metadata to control address expiration. To alleviate disk access and serde overhead, we internally use a read/write-through ARC cache, the size of which is adjustable via Options.CacheSize.

The user has a choice of two GC algorithms:

• full-purge GC (default): performs a full visit of the store with periodicity Options.GCPurgeInterval. Useful when the range of possible TTL values is small and the values themselves are also extreme, e.g. 10 minutes or permanent, popular values used in other libp2p modules. In this cited case, optimizing with lookahead windows makes little sense.

func (*DsAddrBook) AddAddr

func (ab *DsAddrBook) AddAddr(t thread.ID, p peer.ID, addr ma.Multiaddr, ttl time.Duration) error

AddAddr will add a new address if it's not already in the AddrBook.

func (*DsAddrBook) AddAddrs

func (ab *DsAddrBook) AddAddrs(t thread.ID, p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) error

AddAddrs will add many multiple addresses if they aren't already in the AddrBook.

func (*DsAddrBook) AddrStream

func (ab *DsAddrBook) AddrStream(ctx context.Context, t thread.ID, p peer.ID) (<-chan ma.Multiaddr, error)

func (*DsAddrBook) Addrs

func (ab *DsAddrBook) Addrs(t thread.ID, p peer.ID) ([]ma.Multiaddr, error)

func (*DsAddrBook) AddrsEdge

func (ab *DsAddrBook) AddrsEdge(t thread.ID) (uint64, error)

func (*DsAddrBook) ClearAddrs

func (ab *DsAddrBook) ClearAddrs(t thread.ID, p peer.ID) error

func (*DsAddrBook) Close

func (ab *DsAddrBook) Close() error

func (*DsAddrBook) DumpAddrs

func (ab *DsAddrBook) DumpAddrs() (logstore.DumpAddrBook, error)

func (*DsAddrBook) LogsWithAddrs

func (ab *DsAddrBook) LogsWithAddrs(t thread.ID) (peer.IDSlice, error)

func (*DsAddrBook) RestoreAddrs

func (ab *DsAddrBook) RestoreAddrs(dump logstore.DumpAddrBook) error

func (*DsAddrBook) SetAddr

func (ab *DsAddrBook) SetAddr(t thread.ID, p peer.ID, addr ma.Multiaddr, ttl time.Duration) error

func (*DsAddrBook) SetAddrs

func (ab *DsAddrBook) SetAddrs(t thread.ID, p peer.ID, addrs []ma.Multiaddr, ttl time.Duration) error

func (*DsAddrBook) ThreadsFromAddrs

func (ab *DsAddrBook) ThreadsFromAddrs() (thread.IDSlice, error)

func (*DsAddrBook) UpdateAddrs

func (ab *DsAddrBook) UpdateAddrs(t thread.ID, p peer.ID, oldTTL time.Duration, newTTL time.Duration) error

type Options

type Options struct {
	// The size of the in-memory cache. A value of 0 or lower disables the cache.
	CacheSize uint

	// Sweep interval to purge expired addresses from the datastore. If this is a zero value, GC will not run
	// automatically, but it'll be available on demand via explicit calls.
	GCPurgeInterval time.Duration

	// Initial delay before GC processes start. Intended to give the system breathing room to fully boot
	// before starting GC.
	GCInitialDelay time.Duration
}

Configuration object for datastores

func DefaultOpts

func DefaultOpts() Options

DefaultOpts returns the default options for a persistent peerstore, with the full-purge GC algorithm:

* Cache size: 1024. * GC purge interval: 2 hours. * GC initial delay: 60 seconds.