protodb

README

ProtoDB

A versatile strongly typed watchable key/value store based on badger that allow you to scale when your needs evolve.

Why ?

Protocol-buffer (protobuf) is an incredible technology.

More than a fast serialization protocol, it is a networked type system with support for interfaces and reflection.

It makes it a perfect choice for strongly typed key-value store storage format.

Status

it is currently being used in its current implementation in multiple production applications. It largely lack testing, unit tests, regression tests, integration tests, end-to-end tests are missing, the API is often broken, but it will soon be stabilized. The replication is mostly done on a best effort basis and is mostly not tested as well. Metrics are mostly broken as it had significant changes in the latest badger releases.

Principles

A single interface that allows to switch with very little work between the different modes:

• In Process (with in-memory mode support)
• Server
• Replicated (alpha) available in server, in-process and in-memory modes

Features

• changes notifications

• Transactions

• Ephemeral storage (TTL)

• Filtering (based on protofilters) and token based paging.

• Defaulting (using protoc-gen-defaults annotations)

• Validation (using protoc-gen-validate)

Planned:

• Leases
• Indexes support
• Reflection based CEL validation
• CEL based filtering
• Pointers support ?
• Raft replication

Where are the binaries ?

There are no binaries at the moment.

The server implementation is provided but you will still need to implement the server with the features you want, (authentication, tls, metrics, tracing...) to use it in server mode.

That way, you may embed the protobuf descriptors in the key-value store without the need to register them. This is right now the only way to support validation.

Documentation

Index

• Constants
• Variables
• func Cmp(m1, m2 proto.Message) (*pb.MessageDiff, error)
• func WithTx(ctx context.Context, db TxProvider, fn func(ctx context.Context, tx Tx) error, ...) error
• func WithTx2[R any](ctx context.Context, db TxProvider, ...) (R, error)
• type Client
• type Committer
• type DB
• type Event
• type EventType
• type Filter
• type FilterExpr
• type GetOption
• type Leader
• type Option
• type Paging
• type PagingInfo
• type Reader
• type Registerer
• type ReplicationMode
• type ReplicationOption
• type Resolverer
• type Server
• type SetOption
• type Tx
• type TxOption
• type TxProvider
• type Watcher
• type Writer

Constants

const (
	EventTypeEnter  = pb.WatchEventEnter
	EventTypeLeave  = pb.WatchEventLeave
	EventTypeUpdate = pb.WatchEventUpdate
)

Variables

var (
	// ErrKeyNotFound is returned when key isn't found on a txn.Get.
	ErrKeyNotFound = badger.ErrKeyNotFound

	// ErrTxnTooBig is returned if too many writes are fit into a single transaction.
	ErrTxnTooBig = badger.ErrTxnTooBig

	// ErrConflict is returned when a transaction conflicts with another transaction. This can
	// happen if the read rows had been updated concurrently by another transaction.
	ErrConflict = badger.ErrConflict

	// ErrReadOnlyTxn is returned if an update function is called on a read-only transaction.
	ErrReadOnlyTxn = badger.ErrReadOnlyTxn

	// ErrEmptyKey is returned if an empty key is passed on an update function.
	ErrEmptyKey = badger.ErrEmptyKey

	// ErrInvalidKey is returned if the key has a special !badger! prefix,
	// reserved for internal usage.
	ErrInvalidKey = badger.ErrInvalidKey

	// ErrBannedKey is returned if the read/write key belongs to any banned namespace.
	ErrBannedKey = badger.ErrBannedKey

	// ErrInvalidRequest is returned if the user request is invalid.
	ErrInvalidRequest = badger.ErrInvalidRequest

	// ErrInvalidDump if a data dump made previously cannot be loaded into the database.
	ErrInvalidDump = badger.ErrInvalidDump

	// ErrWindowsNotSupported is returned when opt.ReadOnly is used on Windows
	ErrWindowsNotSupported = badger.ErrWindowsNotSupported

	// ErrPlan9NotSupported is returned when opt.ReadOnly is used on Plan 9
	ErrPlan9NotSupported = badger.ErrPlan9NotSupported

	// ErrTruncateNeeded is returned when the value log gets corrupt, and requires truncation of
	// corrupt data to allow Badger to run properly.
	ErrTruncateNeeded = badger.ErrTruncateNeeded

	// ErrBlockedWrites is returned if the user called DropAll. During the process of dropping all
	// data from Badger, we stop accepting new writes, by returning this error.
	ErrBlockedWrites = badger.ErrBlockedWrites

	// ErrEncryptionKeyMismatch is returned when the storage key is not
	// matched with the key previously given.
	ErrEncryptionKeyMismatch = badger.ErrEncryptionKeyMismatch

	// ErrInvalidDataKeyID is returned if the datakey id is invalid.
	ErrInvalidDataKeyID = badger.ErrInvalidDataKeyID

	// ErrInvalidEncryptionKey is returned if length of encryption keys is invalid.
	ErrInvalidEncryptionKey = badger.ErrInvalidEncryptionKey

	// ErrDBClosed is returned when a get operation is performed after closing the DB.
	ErrDBClosed = badger.ErrDBClosed

	ErrNotLeader = protodb.ErrNotLeader

	ErrNoLeaderConn = protodb.ErrNoLeaderConn
)

var (
	WithPath                      = db.WithPath
	WithInMemory                  = db.WithInMemory
	WithBadgerOptionsFunc         = db.WithBadgerOptionsFunc
	WithLogger                    = db.WithLogger
	WithApplyDefaults             = db.WithApplyDefaults
	WithIgnoreProtoRegisterErrors = db.WithIgnoreProtoRegisterErrors
	WithProtoRegisterErrHandler   = db.WithProtoRegisterErrHandler
	WithOnClose                   = db.WithOnClose
	WithReplication               = db.WithReplication
)

var (
	WithPaging             = protodb.WithPaging
	WithFilter             = protodb.WithFilter
	WithReadFieldMaskPaths = protodb.WithReadFieldMaskPaths
	WithReadFieldMask      = protodb.WithReadFieldMask
	WithReverse            = protodb.WithReverse
)

var (
	WithTTL                 = protodb.WithTTL
	WithWriteFieldMaskPaths = protodb.WithWriteFieldMaskPaths
	WithWriteFieldMask      = protodb.WithWriteFieldMask
)

var (
	ReplicationModeNone  = replication.ModeNone
	ReplicationModeAsync = replication.ModeAsync
	ReplicationModeSync  = replication.ModeSync
)

var (
	WithMode       = replication.WithMode
	WithName       = replication.WithName
	WithAddrs      = replication.WithAddrs
	WithGossipPort = replication.WithGossipPort
	WithGRPCPort   = replication.WithGRPCPort
	WithTick       = replication.WithTick
	// WithEncryptionKey set a key to encrypt gossip messages
	WithEncryptionKey = replication.WithEncryptionKey
	// WithRaftStartOptions are Raft start options, e.g. join, new cluster ...
	WithRaftStartOptions = replication.WithRaftStartOptions
	WithServerCert       = replication.WithServerCert
	WithServerKey        = replication.WithServerKey
	WithClientCert       = replication.WithClientCert
	WithClientKey        = replication.WithClientKey
	WithClientCA         = replication.WithClientCA
	WithTLSConfig        = replication.WithTLSConfig
)

var DefaultPath = db.DefaultPath

var NewClient = client.NewClient

var NewServer = server.NewServer

var Open = db.Open

var (
	Where = filters.Where
)

var WithReadOnly = protodb.WithReadOnly

Functions

func Cmp

func Cmp(m1, m2 proto.Message) (*pb.MessageDiff, error)

func WithTx

func WithTx(ctx context.Context, db TxProvider, fn func(ctx context.Context, tx Tx) error, opts ...TxOption) error

func WithTx2

func WithTx2[R any](ctx context.Context, db TxProvider, fn func(ctx context.Context, tx Tx) (R, error), opts ...TxOption) (R, error)

Types

type Client

type Client = client.Client

type Committer

type Committer = protodb.Committer

type DB

type DB = protodb.DB

type Event

type Event = protodb.Event

type EventType

type EventType = pb.WatchEventType

type Filter

type Filter = filters.FieldFilterer

type FilterExpr

type FilterExpr = filters.Expression

type GetOption

type GetOption = protodb.GetOption

type Leader

type Leader = protodb.Leader

type Option

type Option = db.Option

type Paging

type Paging = pb.Paging

type PagingInfo

type PagingInfo = pb.PagingInfo

type Reader

type Reader = protodb.Reader

type Registerer

type Registerer = protodb.Registerer

type ReplicationMode

type ReplicationMode = replication.Mode

type ReplicationOption

type ReplicationOption = replication.Option

type Resolverer

type Resolverer = protodb.Resolverer

type Server

type Server = server.Server

type SetOption

type SetOption = protodb.SetOption

type Tx

type Tx = protodb.Tx

type TxOption

type TxOption = protodb.TxOption

type TxProvider

type TxProvider = protodb.TxProvider

type Watcher

type Watcher = protodb.Watcher

type Writer

type Writer protodb.Writer