crawler

Documentation

Index

• type Crawler
  • func New(host host.Host, opts ...Option) (*Crawler, error)
  • func (c *Crawler) Run(ctx context.Context, startingPeers []*peer.AddrInfo, ...)
• type HandleQueryFail
• type HandleQueryResult
• type Option
  • func AvgBitsImprovedPerStep(avgBitsImprovedPerStep float64) Option
  • func AvgRoundTripPerStepWithNewPeer(avgRoundTripsPerStepWithNewPeer float64) Option
  • func EnableConsiderLatency() Option
  • func WithConnectTimeout(timeout time.Duration) Option
  • func WithMsgTimeout(timeout time.Duration) Option
  • func WithParallelism(parallelism int) Option
  • func WithProtocols(protocols []protocol.ID) Option

Types

type Crawler

type Crawler struct {
	// contains filtered or unexported fields
}

Crawler connects to hosts in the DHT to track routing tables of peers.

func New

func New(host host.Host, opts ...Option) (*Crawler, error)

New creates a new Crawler

func (*Crawler) Run

func (c *Crawler) Run(ctx context.Context, startingPeers []*peer.AddrInfo, handleSuccess HandleQueryResult, handleFail HandleQueryFail)

Run crawls dht peers from an initial seed of `startingPeers`

type HandleQueryFail

type HandleQueryFail func(p peer.ID, err error)

HandleQueryFail is a callback on failed peer query

type HandleQueryResult

type HandleQueryResult func(p peer.ID, rtPeers []*peer.AddrInfo)

HandleQueryResult is a callback on successful peer query

type Option

type Option func(*options) error

Option DHT Crawler option type.

func AvgBitsImprovedPerStep

func AvgBitsImprovedPerStep(avgBitsImprovedPerStep float64) Option

AvgBitsImprovedPerStep configures the estimated average number of bits improved per lookup step. If not set will use the default value calculated using the bucket size.

func AvgRoundTripPerStepWithNewPeer

func AvgRoundTripPerStepWithNewPeer(avgRoundTripsPerStepWithNewPeer float64) Option

AvgRoundTripPerStepWithNewPeer configures the average RTT count needed per lookup step to connect to a new peer and execute the lookup query, varies among transport protocols, reference values: For TCP+TLS1.3 : 4 For QUIC : 2

If not set will default to 4 (TCP+TLS1.3 settings) which will value the xor distance more in sorting.

func EnableConsiderLatency

func EnableConsiderLatency() Option

If enabled, DHT will find the nearest peers to query by taking into account not only the xor distance to the target peer, but also the latency to the local peer (measured in RTT). This strategy can be tuned with AvgBitsImprovedPerStep and AvgRoundTripPerStepWithNewPeer.

Defaults to disabled.

func WithConnectTimeout

func WithConnectTimeout(timeout time.Duration) Option

WithConnectTimeout defines the time for peer connection before timing out

func WithMsgTimeout

func WithMsgTimeout(timeout time.Duration) Option

WithMsgTimeout defines the amount of time a single DHT message is allowed to take before it's deemed failed

func WithParallelism

func WithParallelism(parallelism int) Option

WithParallelism defines the number of queries that can be issued in parallel

func WithProtocols

func WithProtocols(protocols []protocol.ID) Option

WithProtocols defines the ordered set of protocols the crawler will use to talk to other nodes