redgla

README

redgla

Scalable Ethereum Read Requester

redgla stores multiple Ethereum client endpoints, and if more than the specified number of requests are received, the requests are equally divided and sent, and the results are combined and returned to the client. See the Scalable. If scalability is not required, it can also be used to make requests to one alive node of the candidate endpoints instead of split-transmitting to multiple nodes. See the Stable.

Table of Contents

• Install
• Usage
• Scalable
• Stable
• Heartbeat

Install

Required: Go (version 1.16 or later)

$ go get -u github.com/dbadoy/redgla

Usage

Configuration

• Config field details (https://github.com/dbadoy/redgla/blob/main/config.go#L28-L46)
  

ws, wss as endpoints are not allowed yet. Since the close processing for websockets does not work properly, we'll allow it when the implementation is complete.

// https://github.com/dbadoy/redgla/blob/main/config.go#L16-L19
cfg := redgla.DefaultConfig()

cfg.Endpoints = append(cfg.Endpoints, "http://127.0.0.1:8545", "http://mynode.io")

redgla, err := redgla.New(redgla.DefaultHeartbeatFn, cfg)
if err != nil {
  panic(err)
}

HeartbeatFn

DefaultHeartbeatFn checks if the chain ID is successfully obtained from the client. A method that checks whether a node is operating normally can be declared and injected externally. However, you must set the timeout through the context.(e.g. implement methods such as determining that a node is an 'abnormal node' if the chain ID is not the mainnet chain ID)

func fn(ctx context.Context, endpoint string) error {
  client, err := ethclient.DialContext(ctx, endpoint)
  if err != nil {
    return err
  }

  chainID, err = client.ChainID(ctx)
  if err != nil {
    return err
  }

  if chainID.Uint64() != 1 {
    return errors.New("invalid chain id")
  }

  return nil
}

//
redgla, err := redgla.New(fn, cfg)

Execute

// Even given a normal endpoint, it takes some time to determine
// it. If a request is made immediately after calling Run(),
// ErrNoAliveNode may occur. We could also implement a
// notification via a channel after the first heartbeat is over.
redgla.Run()

redgla.Stop()

Scalable

What does 'scalability' mean when making read requests to Ethereum clients? A service that makes read requests doesn't consume much computing power. Actual resource consumption comes from Ethereum nodes processing requested blocks, transactions and receipts. Therefore, scale-up or scale-out for the subject requesting reads will not be very effective, and it is necessary to work on the node that will handle the request. However, scaling up of nodes cannot be considered from the requester's point of view, so let's consider a scaling method that divides requests into multiple nodes and then aggregates them.

Stable

You may not need scalability. However, specifying one endpoint in the service and making a request to it puts service in a single point of failure(SPOF) state. To solve this, it is necessary to register multiple endpoints and send heartbeats periodically to check whether they are normal nodes. This is a native feature of redgla and for this we just need to make a request to one of the endpoints without splitting the request.

Heartbeat

Let's consider the process of making requests to multiple nodes. I need to get receipts for 1000 transactions. Therefore, after sending the request to each of the five nodes in groups of 200, we try to receive them and count them. However, what if a specific node's resources are exhausted or a network failure occurs and the request cannot be properly processed? The result of an incomplete (missing) requested value is an error, and we must resend this request. If one of the five nodes continues to fail, even if four return correct results, the requester is not satisfied. To solve this problem, it maintains a list of healthy nodes by periodically sending low-resource requests to nodes. If we send requests to nodes that are considered to be functioning normally, the possibility that a particular node's operation will be in vain is reduced. Of course, there is a possibility that the node will change to an unhealthy state immediately after sending the request assuming it is normal. However, this will eventually be resolved by sending a request to the newly updated normal node list after the next 'heartbeat interval time'. Since speed is matched to the slowest response, it may be more effective to remove nodes that are too slow from the node list. To help manage the list, it would also be nice to provide response times for requests per node.

Documentation

Overview

Copyright (c) 2023, redgla authors <dbadoy4874@gmail.com> All rights reserved.

Use of this source code is governed by a BSD-style license that can be found in the LICENSE file.

Copyright (c) 2023, redgla authors <dbadoy4874@gmail.com> All rights reserved.

Use of this source code is governed by a BSD-style license that can be found in the LICENSE file.

Copyright (c) 2023, redgla authors <dbadoy4874@gmail.com> All rights reserved.

Use of this source code is governed by a BSD-style license that can be found in the LICENSE file.

Copyright (c) 2023, redgla authors <dbadoy4874@gmail.com> All rights reserved.

Use of this source code is governed by a BSD-style license that can be found in the LICENSE file.

Index

• Variables
• func DefaultHeartbeatFn(ctx context.Context, endpoint string) error
• type Config
  • func DefaultConfig() *Config
• type HeartbeatFn
• type Redgla
  • func New(fn HeartbeatFn, cfg *Config) (*Redgla, error)
  • func (r *Redgla) AddNode(endpoint string) error
  • func (r *Redgla) Benchmark(height uint64, cnt int) (map[string]time.Duration, error)
  • func (r *Redgla) BlockByRange(start uint64, end uint64) (map[uint64]*types.Block, error)
  • func (r *Redgla) BlockByRangeWithBatch(start uint64, end uint64) (map[uint64]*types.Block, error)
  • func (r *Redgla) DelNode(endpoint string) error
  • func (r *Redgla) ReceiptByTxs(txs []*types.Transaction) (map[common.Hash]*types.Receipt, error)
  • func (r *Redgla) ReceiptByTxsWithBatch(txs []*types.Transaction) (map[common.Hash]*types.Receipt, error)
  • func (r *Redgla) Run()
  • func (r *Redgla) Stop()
  • func (r *Redgla) TransactionByHashes(hashes []common.Hash) (map[common.Hash]*types.Transaction, error)
  • func (r *Redgla) TransactionByHashesWithBatch(hashes []common.Hash) (map[common.Hash]*types.Transaction, error)

Variables

var (
	ErrNoAliveNode  = errors.New("there is no alive node")
	ErrBatchFailure = errors.New("batch request failure")
)

Functions

func DefaultHeartbeatFn

func DefaultHeartbeatFn(ctx context.Context, endpoint string) error

Types

type Config

type Config struct {
	// A list of endpoints to send batch requests to.
	Endpoints []string

	// Threshold to send a batch request. If the number of requests is
	// greater than the value, they are converted to batch requests.
	Threshold int

	// This is the timeout of the request to the Ethereum node. It also
	// seems okay to give a very large value and rely on the Ethereum
	// node's request timeout.
	RequestTimeout time.Duration

	// Ping interval for checks alive endpoints.
	HeartbeatInterval time.Duration

	// Timeout for requests to determine 'alive'.
	HeartbeatTimeout time.Duration
}

func DefaultConfig

func DefaultConfig() *Config

There is no default value for Endpoints. Set the Endpoints on the created default Config.

type HeartbeatFn

type HeartbeatFn func(ctx context.Context, endpoint string) error

HeartbeatFn is a method that can check whether the endpoint is working or not. Since it is not possible to specify which service endpoint it is, it is appropriately injected from the outside according to the usage.

type Redgla

type Redgla struct {
	// contains filtered or unexported fields
}

func New

func New(fn HeartbeatFn, cfg *Config) (*Redgla, error)

func (*Redgla) AddNode

func (r *Redgla) AddNode(endpoint string) error

AddNode adds the target endpoint to the list of batch processing nodes. The endpoint entered will take effect starting from the next HeartbeatInterval.

func (*Redgla) Benchmark

func (r *Redgla) Benchmark(height uint64, cnt int) (map[string]time.Duration, error)

Benchmark measures and returns the response time of each node.

Batch request performance is matched to the speed of the slowest node. Removing nodes that are too slow to respond from the list can help improve performance. This method performs a benchmark for requests to fetch 'cnt' times a random number of block numbers less than 'height'.

func (*Redgla) BlockByRange

func (r *Redgla) BlockByRange(start uint64, end uint64) (map[uint64]*types.Block, error)

BlockByRange requests blocks from a range to a node.

func (*Redgla) BlockByRangeWithBatch

func (r *Redgla) BlockByRangeWithBatch(start uint64, end uint64) (map[uint64]*types.Block, error)

BlockByRangeWithBatch transmits and receives batch requests to healthy nodes among the list of registered nodes.

func (*Redgla) DelNode

func (r *Redgla) DelNode(endpoint string) error

DelNode removes the target endpoint from the list of batch processing nodes. Removed endpoints take effect from the next HeartbeatInterval.

func (*Redgla) ReceiptByTxs

func (r *Redgla) ReceiptByTxs(txs []*types.Transaction) (map[common.Hash]*types.Receipt, error)

ReceiptByTxs requests receipts from given transactions to a node.

func (*Redgla) ReceiptByTxsWithBatch

func (r *Redgla) ReceiptByTxsWithBatch(txs []*types.Transaction) (map[common.Hash]*types.Receipt, error)

ReceiptByTxsWithBatch transmits and receives batch requests to healthy nodes among the list of registered nodes.

func (*Redgla) Run

func (r *Redgla) Run()

func (*Redgla) Stop

func (r *Redgla) Stop()

func (*Redgla) TransactionByHashes

func (r *Redgla) TransactionByHashes(hashes []common.Hash) (map[common.Hash]*types.Transaction, error)

TransactionByHashes requests transactions from given hashes to a node.

func (*Redgla) TransactionByHashesWithBatch

func (r *Redgla) TransactionByHashesWithBatch(hashes []common.Hash) (map[common.Hash]*types.Transaction, error)

TransactionByHashesWithBatch transmits and receives batch requests to healthy nodes among the list of registered nodes.