crawler

README

Crawler docs: https://pkg.go.dev/github.com/newity/crawler

Blocklib docs: https://pkg.go.dev/github.com/newity/crawler/blocklib

Crawler is a framework for parsing Hyperledger Fabric blockchain. You can implement any storage, parser or storage adapter or use default implementations.

This is how it looks like:

// get instance of BadgerDB storage
stor, err := storage.NewBadger(path)
if err != nil {
	logrus.Fatal(err)
}

// create Crawler instance with connection to all Fabric channels specified in connection profile (crawler.WithAutoConnect(USER, ORG)),
// using BadgerDB storage (crawler.WithStorage(stor)) and with default storage adapter (crawler.WithStorageAdapter(adapter.NewSimpleAdapter(stor)))
engine, err := crawler.New("connection.yaml", crawler.WithAutoConnect(USER, ORG), crawler.WithStorage(stor), crawler.WithStorageAdapter(adapter.NewSimpleAdapter(stor)))
if err != nil {
	logrus.Error(err)
}

// start block events (*fab.BlockEvent) listeners
err = engine.Listen(crawler.FromBlock(), crawler.WithBlockNum(0))
if err != nil {
	logrus.Error(err)
}

// start parsing and saving info
go engine.Run()

And that's all!

Here are the main parts of a crawler:

• Storage is responsible for saving data fetched from blockchain. Default is BadgerDB.

• Parser is responsible for processing data from the blockchain. Simply put, this is about how exactly and into what constituent parts we will disassemble the blocks. You can find default implementation in https://github.com/newity/crawler/tree/master/parser/parser.go. Default parser just packs all txs with type ENDORSER_TRANSACTION and all events into parser.Data format.

• StorageAdapter is used for implementation specific logic of saving parsed data into the storage. Default implementation saves gob-serialized parser.Data with block number as the key and retrieves parser.Data by block number specified.

You can replace any of these components with your own implementation.

Crawler uses Blocklib under the hood. You can use it too for your own parsing needs.

Examples:

• parse block creator (orderer), tx creator and tx endorsers identities and signatures: https://github.com/newity/crawler/tree/master/examples/getsignatures
• find actual configuration blocks for each block in the network: https://github.com/newity/crawler/tree/master/examples/readlastconfig
• run crawler with BadgerDB storage and retrieve specific block from storage: https://github.com/newity/crawler/tree/master/examples/saveblocksandread
• Google Pub/Sub usage as Crawler storage: https://github.com/newity/crawler/tree/master/examples/pubsub
• NATS Streaming usage as Crawler storage: https://github.com/newity/crawler/tree/master/examples/nats
• Blocklib basic usage example: https://github.com/newity/crawler/tree/master/examples/blocklib
• Blocklib usage for getting chaincode events: https://github.com/newity/crawler/tree/master/examples/blocklib-events

FAQ

We have already a database behind fabric peer, thus we can see BadgeDB as a cache storage, is it?

BadgerDB used by Crawler as just a storage for parsed info. Depending on the implementation of the parser, the data in the Crawler's storage may differ more or less from the data in peer's LevelDB. By the way, at the moment there are storage implementations available not only for BadgerDB, but also for NATS and Google Pub/Sub.

I guess this project is based on block replay support powered by ChannelEventHub. Is it?

Yes. More precisely, the implementation of this interface in the go sdk is used: https://pkg.go.dev/github.com/hyperledger/fabric-sdk-go@v1.0.0/pkg/common/providers/fab#EventService

What will be project target user? What interests a developer to use this than to use fabric-sdk-go?

This project helps any developer who needs to collect data from the blockchain to get rid of the need to write boilerplate code. Crawler offers simple abstractions (and default implementations of each) to cover any need for collecting data from the blockchain. I tried to make the tool as simple as possible and will stick to this principle in the future.

Documentation

Index

• Constants
• type Crawler
  • func New(connectionProfile string, opts ...Option) (*Crawler, error)
  • func (c *Crawler) ConfigProvider() core.ConfigProvider
  • func (c *Crawler) Connect(ch, username, org string) error
  • func (c *Crawler) ConnectWithIdentity(ch, org string, identity msp.SigningIdentity) error
  • func (c *Crawler) GetFromStorage(key string) (*parser.Data, error)
  • func (c *Crawler) Listen(opts ...ListenOpt) error
  • func (c *Crawler) ListenerForChannel(channel string) <-chan *fab.BlockEvent
  • func (c *Crawler) ReadStreamFromStorage(key string) (<-chan *parser.Data, <-chan error)
  • func (c *Crawler) Run()
  • func (c *Crawler) SDK() *fabsdk.FabricSDK
  • func (c *Crawler) StopListenAll()
  • func (c *Crawler) StopListenChannel(channel string)
• type ListenOpt
  • func FromBlock() ListenOpt
  • func Newest() ListenOpt
  • func Oldest() ListenOpt
  • func WithBlockNum(block uint64) ListenOpt
• type Option
  • func WithAutoConnect(username, org string, identity msp.SigningIdentity) Option
  • func WithConfigProvider(configProvider core.ConfigProvider) Option
  • func WithParser(p parser.Parser) Option
  • func WithSDK(sdk *fabsdk.FabricSDK) Option
  • func WithStorage(s storage.Storage) Option
  • func WithStorageAdapter(a storageadapter.StorageAdapter) Option

Constants

const (
	LISTEN_FROM   = "from"
	LISTEN_NEWEST = "newest"
	LISTEN_OLDEST = "oldest"
)

Types

type Crawler

type Crawler struct {
	// contains filtered or unexported fields
}

Crawler is responsible for fetching info from blockchain

func New

func New(connectionProfile string, opts ...Option) (*Crawler, error)

New creates Crawler instance from HLF connection profile and returns pointer to it. "connectionProfile" is a path to HLF connection profile

func (*Crawler) ConfigProvider

func (c *Crawler) ConfigProvider() core.ConfigProvider

ConfigProvider returns core.ConfigProvider instance.

func (*Crawler) Connect

func (c *Crawler) Connect(ch, username, org string) error

Connect connects crawler to channel 'ch' as identity specified in 'username' from organization with name 'org'

func (*Crawler) ConnectWithIdentity

func (c *Crawler) ConnectWithIdentity(ch, org string, identity msp.SigningIdentity) error

ConnectWithIdentity connects crawler to channel 'ch' as passed signing identity from specified organization

func (*Crawler) GetFromStorage

func (c *Crawler) GetFromStorage(key string) (*parser.Data, error)

GetBlock retrieves specified data from a storage by specified key and returns it in the form of parser.Data.

func (*Crawler) Listen

func (c *Crawler) Listen(opts ...ListenOpt) error

Listen starts blocks listener starting from block with num 'from'. All consumed blocks will be hadled by the provided parser (or default parser ParserImpl).

func (*Crawler) ListenerForChannel

func (c *Crawler) ListenerForChannel(channel string) <-chan *fab.BlockEvent

func (*Crawler) ReadStreamFromStorage

func (c *Crawler) ReadStreamFromStorage(key string) (<-chan *parser.Data, <-chan error)

func (*Crawler) Run

func (c *Crawler) Run()

Run starts parsing blocks and saves them to storage. The parsing strategy is determined by the implementation of the parser. What and in what form will be stored in the storage is determined by the storage adapter implementation.

func (*Crawler) SDK

func (c *Crawler) SDK() *fabsdk.FabricSDK

SDK returns fabsdk.FabricSDK instance.

func (*Crawler) StopListenAll

func (c *Crawler) StopListenAll()

StopListenAll removes the registration for block events from all channels and closes these channels

func (*Crawler) StopListenChannel

func (c *Crawler) StopListenChannel(channel string)

StopListenChannel removes the registration for block events from channel and closes the channel

type ListenOpt

type ListenOpt func() interface{}

func FromBlock

func FromBlock() ListenOpt

func Newest

func Newest() ListenOpt

func Oldest

func Oldest() ListenOpt

func WithBlockNum

func WithBlockNum(block uint64) ListenOpt

type Option

type Option func(crawler *Crawler) error

func WithAutoConnect

func WithAutoConnect(username, org string, identity msp.SigningIdentity) Option

WithAutoConnect connects crawler to all channels specified in connection profile 'username' is a Fabric identity name and 'org' is a Fabric organization ti which the identity belongs

func WithConfigProvider

func WithConfigProvider(configProvider core.ConfigProvider) Option

WithConfigProvider injects prepared core.ConfigProvider to the Crawler instance.

func WithParser

func WithParser(p parser.Parser) Option

WithParser injects a specific parser that satisfies the Parser interface to the Crawler instance. If no parser is specified, the default parser ParserImpl will be used.

func WithSDK

func WithSDK(sdk *fabsdk.FabricSDK) Option

WithSDK injects prepared *fabsdk.FabricSDK to the Crawler instance.

func WithStorage

func WithStorage(s storage.Storage) Option

WithStorage adds a specific storage that satisfies the Storage interface to the Crawler instance. If no storage is specified, the default storage Badger (BadgerDB) will be used.

func WithStorageAdapter

func WithStorageAdapter(a storageadapter.StorageAdapter) Option

WithStorageAdapter adds storage adapter that satisfies the StorageAdapter interface to the Crawler instance. If no storage adapter is specified, the default SimpleAdapter will be used.