gsbd

Documentation

Overview

Package gsbd (short for "Gordian Server Block Data") provides utilities for hosting and retrieving block data out of band from the consensus engine.

Index

• Constants
• func DataID(height uint64, round uint32, dataLen uint32, txs []transaction.Tx) string
• func EncodeBlockData(w io.Writer, txs []transaction.Tx) (decompressedDataSize int, err error)
• func IsZeroTxDataID(id string) bool
• func ParseDataID(id string) (height uint64, round uint32, nTxs int, dataLen uint32, ...)
• func TxsHash(txs []transaction.Tx) [txsHashSize]byte
• type BlockDataDecoder
  • func NewBlockDataDecoder(dataID string, txDecoder transaction.Codec[transaction.Tx]) (*BlockDataDecoder, error)
  • func (d *BlockDataDecoder) Decode(r io.Reader) ([]transaction.Tx, error)
• type BlockDataRequest
• type Libp2pClient
  • func NewLibp2pClient(log *slog.Logger, host libp2phost.Host, ...) *Libp2pClient
  • func (c *Libp2pClient) Retrieve(ctx context.Context, ai libp2ppeer.AddrInfo, dataID string) ([]transaction.Tx, error)
• type Libp2pHost
  • func NewLibp2pProviderHost(log *slog.Logger, host libp2phost.Host) *Libp2pHost
  • func (h *Libp2pHost) Provide(ctx context.Context, height uint64, round uint32, pendingTxs []transaction.Tx) (ProvideResult, error)
• type Location
• type ProvideResult
• type Provider
• type RequestCache
  • func NewRequestCache() *RequestCache
  • func (c *RequestCache) Get(dataID string) (r *BlockDataRequest, ok bool)
  • func (c *RequestCache) Purge(dataID string)
  • func (c *RequestCache) SetImmediatelyAvailable(dataID string, txs []transaction.Tx, encodedTxs []byte)
  • func (c *RequestCache) SetInFlight(dataID string, req *BlockDataRequest)
• type Scheme

Constants

const ProposedBlockDataV1Prefix = "/gordian/proposed_blockdata/v1/"

This prefix is used by the proposer hosting block data. "Full nodes" hosting block data should use the full_block endpoint declared elsewhere.

Functions

func DataID

func DataID(
	height uint64,
	round uint32,
	dataLen uint32,
	txs []transaction.Tx,
) string

DataID returns a string formatted as:

HEIGHT:ROUND:NUM_TXs:DATA_LEN:HASH(TXs)

Where the HEIGHT, ROUND, and NUM_TXs are written as decimal numbers in ASCII, so that they are easily human-readable.

DATA_LEN is the length of the raw (uncompressed) data in bytes, encoded in lowercase hexadecimal to slightly shorten the data ID, and because it is less important that this value is human-readable. Furthermore, DATA_LEN will be larger than the stored or transmitted size of data if the stored data is compressed. The DATA_LEN is an indicator to the receiver of the buffer size required to hold the data to be decoded into transactions.

HASH(TXs) is the blake2b hash of the concatenation of every transaction' hash in txs, formatted as lowercase hex-encoded bytes.

func EncodeBlockData

func EncodeBlockData(w io.Writer, txs []transaction.Tx) (
	decompressedDataSize int, err error,
)

EncodeBlockData encodes a set of transactions as block data. The encoding format is as follows:

1. A header byte indicating the compression format, possibly indicating uncompressed.
2. A varint indicating the length of the maybe-compressed data (see binary.AppendVarint).
3. The maybe compressed data, which is currently inefficiently coded as a JSON array of base64 data (in Go, it is a [][]byte that is JSON-marshalled).

The returned decodedDataSize is the size of the uncompressed data, to be provided to the DataID function.

EncodeBlockData panics when len(txs) == 0. Use DataID with arguments (height, round, 0, nil) directly to get the data ID in that case.

func IsZeroTxDataID

func IsZeroTxDataID(id string) bool

func ParseDataID

func ParseDataID(id string) (
	height uint64, round uint32,
	nTxs int,
	dataLen uint32,
	txsHash [txsHashSize]byte,
	err error,
)

func TxsHash

func TxsHash(txs []transaction.Tx) [txsHashSize]byte

TxsHash returns the accumulated hash resulting from the hash of the concatenated transaction hashes. This is the last segment of the data ID.

Types

type BlockDataDecoder

type BlockDataDecoder struct {
	// contains filtered or unexported fields
}

BlockDataDecoder parses a "framed" set of block data. The framing format is a one-byte compression-type header, followed by possibly more header bytes depending on the compression format.

func NewBlockDataDecoder

func NewBlockDataDecoder(
	dataID string,
	txDecoder transaction.Codec[transaction.Tx],
) (*BlockDataDecoder, error)

NewBlockDataDecoder returns a new BlockDataDecoder.

Rather than doing all the DecodeBlockData work in a single call, this validates the dataID input first, so that if the dataID is malformatted, we don't waste resources opening the reader passed to DecodeBlockData.

func (*BlockDataDecoder) Decode

func (d *BlockDataDecoder) Decode(r io.Reader) ([]transaction.Tx, error)

Decode parses the encoded block data in r and returns the resulting transaction slice.

type BlockDataRequest

type BlockDataRequest struct {
	// The Ready channel is closed once the block data is available to read.
	// This lets the consumer synchronize on its availability.
	Ready <-chan struct{}

	// The deserialized transactions.
	Transactions []transaction.Tx

	// The raw data from which the Transactions were derived.
	// This value is used to write into [gcstore.BlockDataStore],
	// so identical data may be hosted to other peers.
	EncodedTransactions []byte
}

BlockDataRequest is the information associated with a request for block data, and a channel for synchronization upon the block data being ready.

type Libp2pClient

type Libp2pClient struct {
	// contains filtered or unexported fields
}

func NewLibp2pClient

func NewLibp2pClient(
	log *slog.Logger,
	host libp2phost.Host,
	decoder transaction.Codec[transaction.Tx],
) *Libp2pClient

func (*Libp2pClient) Retrieve

func (c *Libp2pClient) Retrieve(
	ctx context.Context,
	ai libp2ppeer.AddrInfo,
	dataID string,
) ([]transaction.Tx, error)

type Libp2pHost

type Libp2pHost struct {
	// contains filtered or unexported fields
}

func NewLibp2pProviderHost

func NewLibp2pProviderHost(
	log *slog.Logger,
	host libp2phost.Host,
) *Libp2pHost

func (*Libp2pHost) Provide

func (h *Libp2pHost) Provide(
	ctx context.Context,
	height uint64, round uint32,
	pendingTxs []transaction.Tx,
) (ProvideResult, error)

type Location

type Location struct {
	// Scheme indicates the method of hosting the data.
	Scheme Scheme

	// Addr is the scheme-specific connection string.
	// This is usually only the address of the host;
	// the scheme and the data ID together imply the full connection string.
	Addr string
}

Location indicates where a particular instance of block data is hosted.

type ProvideResult

type ProvideResult struct {
	// DataID should be set at the DataID field in a [tmconsensus.Proposal].
	DataID string

	// Addrs is the list of locations where the data has been provided.
	// TODO: rename this field to Locations.
	// The name Addrs is confusing.
	Addrs []Location

	// The serialized form of the transactions provided.
	Encoded []byte
}

ProvideResult is the return value from [Provider.Provide].

type Provider

type Provider interface {
	// Make the provided transactions available for retrieval.
	// The dataID return value should be set as the DataID in a [tmconsensus.Proposal].
	// The addrs can be included in the proposal's driver annotation
	// to indicate to the other validators where to retrieve the data.
	Provide(ctx context.Context, height uint64, round uint32, txs []transaction.Tx) (
		ProvideResult, error,
	)
}

Provider has one method to provide block data.

type RequestCache

type RequestCache struct {
	// contains filtered or unexported fields
}

RequestCache is an in-memory cache for in-flight block data requests.

The RequestCache only operates in driver-space. A BlockDataRequest entry is added to the RequestCache in the following circumstances:

1. The consensus strategy is proposing a block, and it is immediately marking the data available.
2. The consensus strategy is deciding whether to vote on a proposed block, and it is initiating a request to the proposer's specified addresses in the proposal annotations, in order to evaluate whether the proposed block can be applied.
3. The mirror is performing catchup, and the block data may or may not be immediately available.

The first downside of the RequestCache is that entries must be manually purged. If multiple proposed blocks are received, the driver is designed to only purge the block data for the block it finalizes. This means the consensus strategy should purge the other blocks; otherwise the other block data will never be garbage collected. (Perhaps this implies this cache is a good candidate for a weak map in Go.)

There are some possible race conditions if a block data request may be created from multiple originators, or if the same block data is present in different requests. The latter case could be addressed by including a height and round in the map key.

func NewRequestCache

func NewRequestCache() *RequestCache

func (*RequestCache) Get

func (c *RequestCache) Get(dataID string) (r *BlockDataRequest, ok bool)

Get returns the BlockDataRequest corresponding to dataID. The ok return value follows the idiomatic "comma, ok" pattern in Go.

func (*RequestCache) Purge

func (c *RequestCache) Purge(dataID string)

Purge removes the entry keyed by dataID from the cache. If no such entry exists, Purge panics.

func (*RequestCache) SetImmediatelyAvailable

func (c *RequestCache) SetImmediatelyAvailable(
	dataID string, txs []transaction.Tx, encodedTxs []byte,
)

SetImmediatelyAvailable marks the given dataID as ready to read, with the given txs and encodedTxs. This is equivalent to calling *RequestCache.SetInFlight with a BlockDataRequest with those fields and whose Ready channel is already closed.

func (*RequestCache) SetInFlight

func (c *RequestCache) SetInFlight(dataID string, req *BlockDataRequest)

SetInFlight adds the given BlockDataRequest to the cache, under the key specified by dataID. If there is already an entry for this dataID, SetInFlight panics.

type Scheme

type Scheme uint8

const (
	InvalidScheme Scheme = 0

	Libp2pScheme Scheme = 1
)