blob

README

x/blob

Abstract

The x/blob module enables users to pay for arbitrary data to be published to the Celestia blockchain. This module's name is derived from Binary Large Object (blob).

To use the blob module, users create and submit a BlobTx that is composed of:

1. A single sdk.Tx which encapsulates a message of type MsgPayForBlobs.
2. Multiple Blobs: the data they wish to publish.

After the BlobTx is submitted to the network, a block producer separates the the sdk.Tx from the blob(s). Both components get included in the data square in different namespaces:

1. The sdk.Tx and some metadata about the separated blobs gets included in the PayForBlobNamespace (one of the reserved namespaces).
2. The blob(s) get included in the namespace specified by each blob.

After a block has been created, the user can verify that their data was included in a block via a blob inclusion proof. A blob inclusion proof uses the ShareCommitment in the original sdk.Tx transaction and subtree roots of the block's data square to prove to the user that the shares that compose their original data do in fact exist in a particular block.

TODO: link to blob inclusion (and fraud) proof

State

The blob module doesn't maintain it's own state outside of two params. Meaning that the blob module only uses the params and auth module stores.

Params

// Params defines the parameters for the module.
message Params {
  option (gogoproto.goproto_stringer) = false;
  uint32 gas_per_blob_byte = 1
      [ (gogoproto.moretags) = "yaml:\"gas_per_blob_byte\"" ];
  uint64 gov_max_square_size = 2
      [ (gogoproto.moretags) = "yaml:\"gov_max_square_size\"" ];
}

GasPerBlobByte

GasPerBlobByte is the amount of gas that is consumed per byte of blob data when a MsgPayForBlobs is processed. Currently, the default value is 8. This value is set below that of normal transaction gas consumption, which is 10.

GovMaxSquareSize

GovMaxSquareSize is a governance modifiable parameter that is used to determine the max effective square size. See ADR021 for more details.

Messages

MsgPayForBlobs pays for a set of blobs to be included in the block. Blob transactions that contain this sdk.Msg are also referred to as "PFBs".

// MsgPayForBlobs pays for the inclusion of a blob in the block.
message MsgPayForBlobs {
  // signer is the bech32 encoded signer address
  string signer = 1;
  // namespaces is a list of namespaces that the blobs are associated with. A
  // namespace is a byte slice of length 29 where the first byte is the
  // namespaceVersion and the subsequent 28 bytes are the namespaceId.
  repeated bytes namespaces = 2;
  // blob_sizes is a list of blob sizes (one per blob). Each size is in bytes.
  repeated uint32 blob_sizes = 3;
  // share_commitments is a list of share commitments (one per blob).
  repeated bytes share_commitments = 4;
  // share_versions are the versions of the share format that the blobs
  // associated with this message should use when included in a block. The
  // share_versions specified must match the share_versions used to generate the
  // share_commitment in this message.
  repeated uint32 share_versions = 8;
}

[!NOTE] The internal representation of share versions is always uint8. Since protobuf doesn't support the uint8 type, they are encoded and decoded as uint32.

Generating the ShareCommitment

The share commitment is the commitment to share encoded blobs. It can be used for cheap inclusion checks for some data by light clients. More information and rational can be found in the data square layout specs.

1. Split the blob into shares of size shareSize
2. Determine the SubtreeWidth by dividing the length in shares by the SubtreeRootThreshold.
3. Generate each subtree root by diving the blob shares into SubtreeWidth sized sets, then take the binary namespaced merkle tree (NMT) root of each set of shares.
4. Calculate the final share commitment by taking the merkle root (note: not an NMT, just a normal binary merkle root) of the subtree roots from the previous step.

See CreateCommitment for an implementation. See data square layout and ADR013 for details on the rational of the square layout.

Validity Rules

In order for a proposal block to be considered valid, each BlobTx, and thus each PFB, to be included in a block must follow a set of validity rules.

1. Signatures: All blob transactions must have valid signatures. This is state-dependent because correct signatures require using the correct sequence number(aka nonce).
2. Single SDK.Msg: There must be only a single sdk.Msg encoded in the sdk.Tx field of the blob transaction BlobTx.
3. Namespace Validity: The namespace of each blob in a blob transaction BlobTx must be valid. This validity is determined by the following sub-rules:
   1. The namespace of each blob must match the respective (same index) namespace in the MsgPayForBlobs sdk.Msg field namespaces.
   2. The namespace is not reserved for protocol use.
4. Blob Size: No blob can have a size of 0.
5. Blob Count: There must be one or more blobs included in the transaction.
6. Share Commitment Validity: Each share commitment must be valid.
   1. The size of each of the share commitments must be equal to the digest of the hash function used (sha256 so 32 bytes).
   2. The share commitment must be calculated using the steps specified above in Generating the Share Commitment
7. Share Versions: The versions of the shares must be supported.
8. Signer Address: The signer address must be a valid Celestia address.
9. Proper Encoding: The blob transactions must be properly encoded.
10. Size Consistency: The sizes included in the PFB field blob_sizes, and each must match the actual size of the respective (same index) blob in bytes.

IndexWrappedTx

When a block producer is preparing a block, they must perform an extra step for BlobTxs so that end-users can find the blob shares relevant to their submitted BlobTx. In particular, block proposers wrap the BlobTx in the PFB namespace with the index of the first share of the blob in the data square. See Blob share commitment rules for more details.

Since BlobTxs can contain multiple blobs, the sdk.Tx portion of the BlobTx is wrapped with one share index per blob in the transaction. The index wrapped transaction is called an IndexWrapper and this is the struct that gets marshalled and written to the PayForBlobNamespace.

Events

The blob module emits the following events:

Blob Events

EventPayForBlobs

Attribute Key	Attribute Value
signer	{bech32 encoded signer address}
blob_sizes	{sizes of blobs in bytes}
namespaces	{namespaces the blobs should be published to}

Parameters

Key	Type	Default
GasPerBlobByte	uint32	8

Usage

celestia-appd tx blob PayForBlobs <hex encoded namespace> <hex encoded data> [flags]

For submitting PFB transaction via a light client's rpc, see celestia-node's documentation.

The steps in the SubmitPayForBlobs function can be reverse engineered to submit blobs programmatically.

FAQ

Q: Why do the PFB transactions in the response from Comet BFT API endpoints fail to decode to valid transaction hashes?

The response of CometBFT API endpoints (e.g. /cosmos/base/tendermint/v1beta1/blocks/{block_number}) will contain a field called txs with base64 encoded transactions. In Celestia, transactions may have one of the two possible types of sdk.Tx or BlobTx (which wraps around a sdk.Tx). As such, each transaction should be first decoded and then gets unmarshalled according to its type, as explained below:

1. Base64 decode the transaction
2. Check to see if the transaction is a BlobTx by unmarshalling it into a BlobTx type.
   1. If it is a BlobTx, then unmarshal the BlobTx's Tx field into a sdk.Tx type.
   2. If it is not a BlobTx, then unmarshal the transaction into a sdk.Tx type.

See test/decode_blob_tx_test.go for an example of how to do this.

Documentation

Overview

blob is a Cosmos SDK module that enables users to pay for data to be published to the Celestia blockchain. Please see ./README.md for the full specification of this module.

Index

• func ExportGenesis(ctx sdk.Context, k keeper.Keeper) *types.GenesisState
• func InitGenesis(ctx sdk.Context, k keeper.Keeper, genState types.GenesisState)
• func NewHandler(k keeper.Keeper) sdk.Handler
• type AppModule
  • func NewAppModule(cdc codec.Codec, keeper keeper.Keeper) AppModule
  • func (am AppModule) BeginBlock(_ sdk.Context, _ abci.RequestBeginBlock)
  • func (AppModule) ConsensusVersion() uint64
  • func (am AppModule) EndBlock(_ sdk.Context, _ abci.RequestEndBlock) []abci.ValidatorUpdate
  • func (am AppModule) ExportGenesis(ctx sdk.Context, cdc codec.JSONCodec) json.RawMessage
  • func (am AppModule) InitGenesis(ctx sdk.Context, cdc codec.JSONCodec, gs json.RawMessage) []abci.ValidatorUpdate
  • func (am AppModule) LegacyQuerierHandler(_ *codec.LegacyAmino) sdk.Querier
  • func (am AppModule) Name() string
  • func (AppModule) QuerierRoute() string
  • func (am AppModule) RegisterInvariants(_ sdk.InvariantRegistry)
  • func (am AppModule) RegisterServices(cfg module.Configurator)
  • func (am AppModule) Route() sdk.Route
• type AppModuleBasic
  • func NewAppModuleBasic(cdc codec.BinaryCodec) AppModuleBasic
  • func (AppModuleBasic) DefaultGenesis(cdc codec.JSONCodec) json.RawMessage
  • func (AppModuleBasic) GetQueryCmd() *cobra.Command
  • func (a AppModuleBasic) GetTxCmd() *cobra.Command
  • func (AppModuleBasic) Name() string
  • func (AppModuleBasic) RegisterGRPCGatewayRoutes(clientCtx client.Context, mux *runtime.ServeMux)
  • func (a AppModuleBasic) RegisterInterfaces(reg cdctypes.InterfaceRegistry)
  • func (AppModuleBasic) RegisterLegacyAminoCodec(cdc *codec.LegacyAmino)
  • func (AppModuleBasic) RegisterRESTRoutes(_ client.Context, _ *mux.Router)
  • func (AppModuleBasic) ValidateGenesis(cdc codec.JSONCodec, _ client.TxEncodingConfig, bz json.RawMessage) error

Functions

func ExportGenesis

func ExportGenesis(ctx sdk.Context, k keeper.Keeper) *types.GenesisState

ExportGenesis returns the capability module's exported genesis.

func InitGenesis

func InitGenesis(ctx sdk.Context, k keeper.Keeper, genState types.GenesisState)

InitGenesis initializes the capability module's state from a provided genesis state.

func NewHandler

func NewHandler(k keeper.Keeper) sdk.Handler

NewHandler uses the provided blob keeper to create an sdk.Handler

Types

type AppModule

type AppModule struct {
	AppModuleBasic
	// contains filtered or unexported fields
}

AppModule implements the AppModule interface for the capability module.

func NewAppModule

func NewAppModule(cdc codec.Codec, keeper keeper.Keeper) AppModule

func (AppModule) BeginBlock

func (am AppModule) BeginBlock(_ sdk.Context, _ abci.RequestBeginBlock)

BeginBlock executes all ABCI BeginBlock logic respective to the capability module.

func (AppModule) ConsensusVersion

func (AppModule) ConsensusVersion() uint64

ConsensusVersion implements ConsensusVersion.

func (AppModule) EndBlock

func (am AppModule) EndBlock(_ sdk.Context, _ abci.RequestEndBlock) []abci.ValidatorUpdate

EndBlock executes all ABCI EndBlock logic respective to the capability module. It returns an empty list of validator updates.

func (AppModule) ExportGenesis

func (am AppModule) ExportGenesis(ctx sdk.Context, cdc codec.JSONCodec) json.RawMessage

ExportGenesis returns the capability module's exported genesis state as raw JSON bytes.

func (AppModule) InitGenesis

func (am AppModule) InitGenesis(ctx sdk.Context, cdc codec.JSONCodec, gs json.RawMessage) []abci.ValidatorUpdate

InitGenesis performs the capability module's genesis initialization. It returns an empty list of validator updates.

func (AppModule) LegacyQuerierHandler

func (am AppModule) LegacyQuerierHandler(_ *codec.LegacyAmino) sdk.Querier

LegacyQuerierHandler returns the capability module's Querier.

func (AppModule) Name

func (am AppModule) Name() string

Name returns the capability module's name.

func (AppModule) QuerierRoute

func (AppModule) QuerierRoute() string

QuerierRoute returns the capability module's query routing key.

func (AppModule) RegisterInvariants

func (am AppModule) RegisterInvariants(_ sdk.InvariantRegistry)

RegisterInvariants registers the capability module's invariants.

func (AppModule) RegisterServices

func (am AppModule) RegisterServices(cfg module.Configurator)

RegisterServices registers a GRPC query service to respond to the module-specific GRPC queries.

func (AppModule) Route

func (am AppModule) Route() sdk.Route

Route returns the capability module's message routing key.

type AppModuleBasic

type AppModuleBasic struct {
	// contains filtered or unexported fields
}

AppModuleBasic implements the AppModuleBasic interface for the capability module.

func NewAppModuleBasic

func NewAppModuleBasic(cdc codec.BinaryCodec) AppModuleBasic

func (AppModuleBasic) DefaultGenesis

func (AppModuleBasic) DefaultGenesis(cdc codec.JSONCodec) json.RawMessage

DefaultGenesis returns the capability module's default genesis state.

func (AppModuleBasic) GetQueryCmd

func (AppModuleBasic) GetQueryCmd() *cobra.Command

GetQueryCmd returns the capability module's root query command.

func (AppModuleBasic) GetTxCmd

func (a AppModuleBasic) GetTxCmd() *cobra.Command

GetTxCmd returns the capability module's root tx command.

func (AppModuleBasic) Name

func (AppModuleBasic) Name() string

Name returns the capability module's name.

func (AppModuleBasic) RegisterGRPCGatewayRoutes

func (AppModuleBasic) RegisterGRPCGatewayRoutes(clientCtx client.Context, mux *runtime.ServeMux)

RegisterGRPCGatewayRoutes registers the gRPC Gateway routes for the module.

func (AppModuleBasic) RegisterInterfaces

func (a AppModuleBasic) RegisterInterfaces(reg cdctypes.InterfaceRegistry)

RegisterInterfaces registers the module's interface types

func (AppModuleBasic) RegisterLegacyAminoCodec

func (AppModuleBasic) RegisterLegacyAminoCodec(cdc *codec.LegacyAmino)

func (AppModuleBasic) RegisterRESTRoutes

func (AppModuleBasic) RegisterRESTRoutes(_ client.Context, _ *mux.Router)

RegisterRESTRoutes registers the capability module's REST service handlers.

func (AppModuleBasic) ValidateGenesis

func (AppModuleBasic) ValidateGenesis(cdc codec.JSONCodec, _ client.TxEncodingConfig, bz json.RawMessage) error

ValidateGenesis performs genesis state validation for the capability module.