blobstorage

README

blob-storage

Repository for BLOB storage (archive and serve data)

Prerequisites

• Docker engine up and running.
• Go installed on your system.

Configuration and Setup

Setting up MySQL

1. Start MySQL:

   Navigate to the docker-compose directory and start the MySQL service:

   cd ./docker-compose
   docker-compose up -d
   

   This command starts your MySQL instance as defined in your docker-compose.yml file.

2. Migrate Database Schema:

   Navigate to the migrations directory to apply database migrations:

   cd ./migrations
   goose mysql "<user>:<password>@tcp(localhost:3306)/blobs" status
   goose mysql "<user>:<password>@tcp(localhost:3306)/blobs" up
   

   These commands apply migrations to the blobs database.

Environment Configuration

Ensure your .default.indexer.env and .default.server.env files are configured with the correct database credentials, host, and any other necessary environment variables.

Running the Application

1. Start the Indexer:

   With the environment file configured, start the indexer:

   ENV_FILE=.default.indexer.env go run cmd/main.go indexer
   

   This starts the app from the latest block height by default. Adjust the STARTING_BLOCK_ID in the environment file if needed.

2. Start the Server:

   Similarly, start the server:

   ENV_FILE=.default.server.env go run cmd/main.go server
   

Testing and Usage

When the DB, blob-catcher and server are running, the blob-catcher is outputting the blobHash to the terminal (with the networkName variable too, though it is not written into the DB). Use that blobHash (including the 0x) in

1. Querying Blob Data via HTTP Request:

   To retrieve blob data, you can execute a curl command. This allows for querying multiple blobHashes simultaneously, separated by commas. A single request can yield an array of results:

   curl -X GET "http://localhost:3282/getBlob?blobHash=0x01a2a1cdc7ad221934061642a79a760776a013d0e6fa1a1c6b642ace009c372a,0xWRONG_HASH"
   

   Expected Output:

   {"data":[{"blob":"0x123...00","kzg_commitment":"0xabd68b406920aa74b83cf19655f1179d373b5a8cba21b126b2c18baf2096c8eb9ab7116a89b375546a3c30038485939e"}, {"blob":"NOT_FOUND","kzg_commitment":"NOT_FOUND"}]}
   

2. Backtesting with a Python Script:

   This script facilitates querying the database directly based on a specified blob_hash. Modify the blob_hash variable in the script to match the hash you wish to query.

   To run the script:

   python3 python_query.py
   

Todos

What is still missing is:

• small refinements and DevOps (prod-grade DB with credentials, proper containerization)

Documentation

Index

• Variables
• type BlobHash
• type BlobHashRepository
• type BlockMeta
• type BlockMetaRepository
• type DB
• type DBConnectionOpts
• type SaveBlobHashOpts
• type SaveBlockMetaOpts

Variables

var (
	ErrNoDB = errors.Validation.NewWithKeyAndDetail("ERR_NO_DB", "DB is required")
)

Types

type BlobHash

type BlobHash struct {
	BlobHash      string
	KzgCommitment string
	BlobData      string
}

type BlobHashRepository

type BlobHashRepository interface {
	Save(opts SaveBlobHashOpts) error
	FirstByBlobHash(blobHash string) (*BlobHash, error)
	DeleteAllAfterBlockID(blockID uint64) error
}

type BlockMeta

type BlockMeta struct {
	BlobHash       string
	BlockID        uint64
	EmittedBlockID uint64
}

type BlockMetaRepository

type BlockMetaRepository interface {
	Save(opts SaveBlockMetaOpts) error
	FindLatestBlockID() (uint64, error)
	DeleteAllAfterBlockID(blockID uint64) error
}

type DB

type DB interface {
	DB() (*sql.DB, error)
	GormDB() *gorm.DB
}

type DBConnectionOpts

type DBConnectionOpts struct {
	Name            string
	Password        string
	Host            string
	Database        string
	MaxIdleConns    uint64
	MaxOpenConns    uint64
	MaxConnLifetime uint64
	OpenFunc        func(dsn string) (DB, error)
}

type SaveBlobHashOpts

type SaveBlobHashOpts struct {
	BlobHash      string
	KzgCommitment string
	BlobData      string
}

type SaveBlockMetaOpts

type SaveBlockMetaOpts struct {
	BlobHash       string
	BlockID        uint64
	EmittedBlockID uint64
}