indexer

package
v0.6.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 13, 2024 License: MIT Imports: 24 Imported by: 2

Documentation

Overview

Package indexer defines indexer server and client. The indexer server is a service that maintain a trusted storage node list, client can fetch the list from server and select nodes in list to upload data.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func InitDefaultIPLocationManager added in v0.3.1 

func InitDefaultIPLocationManager(config IPLocationConfig)

InitDefaultIPLocationManager initializes the default `IPLocationManager`.

func InitDefaultNodeManager added in v0.3.1 

func InitDefaultNodeManager(config NodeManagerConfig) (closable func(), err error)

InitDefaultNodeManager initializes the default `NodeManager`.

func InitFileLocationCache added in v0.4.0 

func InitFileLocationCache(config FileLocationCacheConfig) (closable func(), err error)

Types

type Client 

type Client struct {
	interfaces.Provider
	// contains filtered or unexported fields
}

Client indexer client

func NewClient 

func NewClient(url string, option ...IndexerClientOption) (*Client, error)

NewClient create new indexer client, url is indexer service url

func (*Client) BatchUpload added in v0.3.1 

func (c *Client) BatchUpload(ctx context.Context, w3Client *web3go.Client, datas []core.IterableData, waitForLogEntry bool, option ...transfer.BatchUploadOption) (eth_common.Hash, []eth_common.Hash, error)

BatchUpload submit multiple data to 0g storage contract batchly in single on-chain transaction, then transfer the data to the storage nodes selected from indexer service.

func (*Client) Download added in v0.4.0 

func (c *Client) Download(ctx context.Context, root, filename string, withProof bool) error

Download download file by given data root

func (*Client) GetFileLocations added in v0.4.0 

func (c *Client) GetFileLocations(ctx context.Context, root string) (locations []*shard.ShardedNode, err error)

GetFileLocations return locations info of given file.

func (*Client) GetNodeLocations added in v0.3.1 

func (c *Client) GetNodeLocations(ctx context.Context) (locations map[string]*IPLocation, err error)

GetNodes return storage nodes with IP location information.

func (*Client) GetShardedNodes added in v0.3.1 

func (c *Client) GetShardedNodes(ctx context.Context) (nodes ShardedNodes, err error)

GetNodes get node list from indexer service

func (*Client) NewUploaderFromIndexerNodes added in v0.3.1 

func (c *Client) NewUploaderFromIndexerNodes(ctx context.Context, w3Client *web3go.Client, expectedReplica uint, dropped []string) (*transfer.Uploader, error)

NewUploaderFromIndexerNodes return an uploader with selected storage nodes from indexer service.

func (*Client) SelectNodes added in v0.3.1 

func (c *Client) SelectNodes(ctx context.Context, expectedReplica uint, dropped []string) ([]*node.ZgsClient, error)

SelectNodes get node list from indexer service and select a subset of it, which is sufficient to store expected number of replications.

func (*Client) Upload added in v0.3.1 

func (c *Client) Upload(ctx context.Context, w3Client *web3go.Client, data core.IterableData, option ...transfer.UploadOption) (eth_common.Hash, error)

Upload submit data to 0g storage contract, then transfer the data to the storage nodes selected from indexer service.

type FileLocation added in v0.4.0 

type FileLocation struct {
	Url         string            `json:"url"`
	ShardConfig shard.ShardConfig `json:"shardConfig"`
}

type FileLocationCache added in v0.4.0 

type FileLocationCache struct {
	// contains filtered or unexported fields
}

func (*FileLocationCache) GetFileLocations added in v0.4.0 

func (c *FileLocationCache) GetFileLocations(ctx context.Context, txSeq uint64) ([]*shard.ShardedNode, error)

type FileLocationCacheConfig added in v0.4.0 

type FileLocationCacheConfig struct {
	CacheSize      int
	Expiry         time.Duration
	DiscoveryNode  string
	DiscoveryPorts []int
}

type IPLocation added in v0.3.1 

type IPLocation struct {
	City     string `json:"city"`
	Region   string `json:"region"`
	Country  string `json:"country"`
	Location string `json:"loc"`
	Timezone string `json:"timezone"`
}

type IPLocationConfig added in v0.3.1 

type IPLocationConfig struct {
	CacheFile          string
	CacheWriteInterval time.Duration
	AccessToken        string
}

type IPLocationManager added in v0.3.1 

type IPLocationManager struct {
	// contains filtered or unexported fields
}

IPLocationManager manages IP locations.

func (*IPLocationManager) All added in v0.3.1 

func (manager *IPLocationManager) All() map[string]*IPLocation

All returns all cached IP locations.

func (*IPLocationManager) Query added in v0.3.1 

func (manager *IPLocationManager) Query(ip string) (*IPLocation, error)

Query returns the cached IP location if any. Otherwise, retrieve from web API.

type IndexerApi 

type IndexerApi struct {
	Namespace string
}

IndexerApi indexer service configuration

func NewIndexerApi 

func NewIndexerApi() *IndexerApi

NewIndexerApi creates indexer service configuration

func (*IndexerApi) GetFileLocations added in v0.4.0 

func (api *IndexerApi) GetFileLocations(ctx context.Context, root string) (locations []*shard.ShardedNode, err error)

GetFileLocations return locations info of given file.

func (*IndexerApi) GetNodeLocations added in v0.3.1 

func (api *IndexerApi) GetNodeLocations(ctx context.Context) (map[string]*IPLocation, error)

GetNodeLocations return IP locations of all nodes.

func (*IndexerApi) GetShardedNodes added in v0.3.1 

func (api *IndexerApi) GetShardedNodes(ctx context.Context) (ShardedNodes, error)

GetShardedNodes return storage node list

type IndexerClientOption added in v0.3.1 

type IndexerClientOption struct {
	ProviderOption providers.Option
	LogOption      common.LogOption // log option when uploading data
}

IndexerClientOption indexer client option

type Interface 

type Interface interface {
	GetShardedNodes(ctx context.Context) (ShardedNodes, error)

	GetNodeLocations(ctx context.Context) (map[string]*IPLocation, error)

	GetFileLocations(ctx context.Context, root string) ([]*shard.ShardedNode, error)
}

type NodeManager added in v0.3.1 

type NodeManager struct {
	// contains filtered or unexported fields
}

NodeManager manages trusted storage nodes and auto discover peers from network.

func (*NodeManager) AddTrustedNodes added in v0.3.1 

func (nm *NodeManager) AddTrustedNodes(nodes ...string) error

AddTrustedNodes add trusted storage nodes.

func (*NodeManager) Discovered added in v0.3.1 

func (nm *NodeManager) Discovered() []*shard.ShardedNode

Discovered returns discovered sharded nodes.

func (*NodeManager) Trusted added in v0.3.1 

func (nm *NodeManager) Trusted() ([]*shard.ShardedNode, error)

Trusted returns trusted sharded nodes.

func (*NodeManager) TrustedClients added in v0.4.0 

func (nm *NodeManager) TrustedClients() []*node.ZgsClient

TrustedClients returns trusted clients.

type NodeManagerConfig added in v0.3.1 

type NodeManagerConfig struct {
	TrustedNodes []string

	DiscoveryNode     string
	DiscoveryInterval time.Duration
	DiscoveryPorts    []int

	UpdateInterval time.Duration
}

type ShardedNodes added in v0.3.1 

type ShardedNodes struct {
	Trusted    []*shard.ShardedNode `json:"trusted"`
	Discovered []*shard.ShardedNode `json:"discovered"`
}

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.