hydra-booster

README

Hydra Booster

A DHT Indexer node & Peer Router

A new type of DHT node designed to accelerate the Content Resolution & Content Providing on the IPFS Network. A (cute) Hydra with one belly full of records and many heads (Peer IDs) to tell other nodes about them, charged with rocket boosters to transport other nodes to their destination faster.

Read the RFC. Disclaimer: We are at Stage 1 of the RFC. Kanban

Install

[openssl support (lower CPU usage)]
go get -u -tags=openssl github.com/libp2p/hydra-booster

[standard (sub-optimal)]
go get -u github.com/libp2p/hydra-booster

Usage

hydra-booster has two modes. A 'single head' mode that has a nicer UI, this is intended to be run in a tmux window or something so you can see statistics about your contribution to the network.

go run ./main.go

The second mode is called 'many heads'. Passing the -nheads=N allows you to run N heads at a time in the same process. It periodically prints out a status line with information about total peers, uptime, and memory usage.

go run ./main.go -nheads=5

Alternatively you can use the HYDRA_NHEADS environment var to specify the number of heads. Note the -nheads flag takes precedence.

Best Practices

Only run a hydra-booster on machines with public IP addresses. Having more DHT nodes behind NATs makes DHT queries in general slower, as connecting in generally takes longer and sometimes doesnt even work (resulting in a timeout).

When running with -nheads, please make sure to bump the ulimit to something fairly high. Expect ~500 connections per node youre running (so with -nheads=10, try setting ulimit -n 5000)

Running Multiple Hydras

The total number of heads a single Hydra can have depends on the resources of the machine it's running on. To get the desired number of heads you may need to run multiple Hydras on multiple machines. There's a couple of challenges with this:

• Peer IDs of Hydra heads are balanced in the DHT. When running multiple Hydras it's necessary to designate one of the Hydras to be the "idgen server" and the rest to be "idgen clients" so that all Peer IDs in the Hydra swarm are balanced. Use the -idgen-addr flag or HYDRA_IDGEN_ADDR environment variable to ensure all Peer IDs in the Hydra swarm are balanced perfectly.
• A datastore is shared by all Hydra heads but not by all Hydras. Use the -db flag or HYDRA_DB environment variable to specify a PostgreSQL database connection string that can be shared by all Hydras in the swarm.

Developers

Release a new version

1. Update version number in version.go.
2. Create a semver tag with "v" prefix e.g. git tag v0.1.7.
3. See deployment.md#continuous-deployment for what happens next.

Publish a new image

# Build your container
docker build -t hydra-booster .

# Get it to run
docker run hydra-booster

# Commit new version
docker commit -m="some commit message" <CONTAINER_ID> libp2p/hydra-booster

# Push to docker hub (must be logged in, do docker login)
docker push libp2p/hydra-booster

Metrics collection with Prometheus

Install Prometheus and then start it using the provided config:

prometheus --config.file=promconfig.yaml --storage.tsdb.path=prometheus-data

Next start the Hydra Booster, specifying the port to run metrics on:

go run ./main.go -nheads=5 -metrics-port=8888

You should now be able to access metrics at http://127.0.0.1:9090.

API

HTTP API

By default the HTTP API is available at http://127.0.0.1:7779.

GET /heads

Returns an ndjson list of peers created by the Hydra: their IDs and mulitaddrs. Example output:

{"Addrs":["/ip4/127.0.0.1/tcp/50277","/ip4/192.168.0.3/tcp/50277"],"ID":"12D3KooWHacdCMnm4YKDJHn72HPTxc6LRGNzbrbyVEnuLFA3FXCZ"}
{"Addrs":["/ip4/127.0.0.1/tcp/50280","/ip4/192.168.0.3/tcp/50280","/ip4/90.198.150.147/tcp/50280"],"ID":"12D3KooWQnUpnw6xS2VrJw3WuCP8e92fsEDnh4tbqyrXW5AVJ7oe"}
...

GET /records/list

Returns an ndjson list of provider records stored by the Hydra Booster node.

GET /records/fetch/{cid}?nProviders=1

Fetches provider record(s) available on the network by CID. Use the nProviders query string parameter to signal the number of provider records to find. Returns an ndjson list of provider peers: their IDs and mulitaddrs. Will return HTTP status code 404 if no records were found.

POST /idgen/add

Generate and add a balanced Peer ID to the server's xor trie and return it for use by another Hydra Booster peer. Returns a base64 encoded JSON string. Example output:

"CAESQNcYNr0ENfml2IaiE97Kf3hGTqfB5k5W+C2/dW0o0sJ7b7zsvxWMedz64vKpS2USpXFBKKM9tWDmcc22n3FBnow="

POST /idgen/remove

Remove a balanced Peer ID from the server's xor trie. Accepts a base64 encoded JSON string.

License

The hydra-booster project is dual-licensed under Apache 2.0 and MIT terms:

• Apache License, Version 2.0, (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)