sia-node-scanner

README

Sia-Node-Scanner

The Sia-Node-Scanner tries to learn the IP addresses of all nodes in the Sia network in order to get an accurate count of current active nodes.

Overview

The Sia-Node-Scanner uses a Sia gateway to explore the Sia peer-to-peer network. It attempts to connect to as many Sia nodes as possible, and uses the ShareNodes RPC with Sia nodes to learn the IP addresses of more nodes in the network. ShareNodes returns only 10 randomly selected peers, so the node scanner uses this RPC multiple times with each peer to learn more from each of them.

Starting with the set of bootstrap nodes, the node scanner uses multiple worker goroutines to scan each IP address it learns about from the network. The main goroutine makes sure addresses are not scanned more than once in each scan, and also logs any error messages generated by a worker's scan. If a worker learned of some new IP addresses, the main goroutine adds them to a work queue so that workers can scan them during the run.

Stored Data

During each scan the scanner creates a log of all IP addresses it attempts to connect to, the Unix timestamp from the time of the attempt, and any errors that resulted from the either the connection attempt or ShareNodes RPC attempts.

In addition to a log for each scan, the scanner also maintains a persistent set of all nodes it has successfully connected to. After the first run (which starts with the bootstrap nodes), all future runs start off the scan by filling the queue with this persistent set. For each node in the persisted set, we also store the first connection time, the last successful connection time, recent uptime (number of seconds last node was last down or since first connection), total uptime, and also uptime percentage.

If a node has not been connected to successfully in over 30 days, it will be pruned from the set the next time a scan is started.

Usage

Running the make utils command will create and install the binary for you. The command make test-utils (and make test) will run the tests for the node scanner. Then running sia-node-scanner will create a SiaNodeScanner directory in the working directory which contains a directory for the gateway module use for the scan, which can be used for debugging, a persisted-node-set.json and persisted-node-set.json_temp, and timestamped files for each scan scan-07-12:15:41.json.

Use the -dir flag to specify which directory to make the SiaNodeScanner directory in. The flag defaults to the current working directory.

Ex:

SiaNodeScanner
  persisted-node-set.json
  persisted-node-set.json_temp
  scan-07-12:15:41.json
  scan-07-13:15:41.json
  ...
  gateway
    |    
    |
    └───(gateway module internals)

Implementation details

There is 1 main goroutine, and 10 worker goroutines. The main routine starts the workers and creates workAssignments which are sent down a channel. The workers read from this channel to start scanning different IP addresses. When a scan is completed (either successfully or because an error was encountered) each worker sends a NodeScanResult back down a result channel. The worker then reads from this channel, logging the result and any errors, and creating new work assignments if new IP addresses were discovered.

The main goroutine also periodically (every 10 seconds) prints counters and connection stats to standard output. These counters are also logged at the end of each scan in the scan file.

Every minute the main goroutine also updates the persisted node set file with any additions or changes from the current scan. It also does this one last time at the end of the run.

The main goroutine finishes the scan when all work assignments have been completed, and when the work queues are empty. To do this, it keeps track of the total number of assignments created and the total number of worker results read back. When these numbers are equal, and both the assignment and result channels are empty, the main goroutine finishes the scan by writing the last changes to the scan log and persisted node set files.