libp2p

README

The Go implementation of the libp2p Networking Stack.

Project status

Weekly Core Dev Calls

Table of Contents

• Background
• Bundles
• Usage
  • Install
  • API
  • Examples
• Development
  • Tests
  • Packages
• Contribute
• License

Background

libp2p is a networking stack and library modularized out of The IPFS Project, and bundled separately for other tools to use.

libp2p is the product of a long, and arduous quest of understanding -- a deep dive into the internet's network stack, and plentiful peer-to-peer protocols from the past. Building large scale peer-to-peer systems has been complex and difficult in the last 15 years, and libp2p is a way to fix that. It is a "network stack" -- a protocol suite -- that cleanly separates concerns, and enables sophisticated applications to only use the protocols they absolutely need, without giving up interoperability and upgradeability. libp2p grew out of IPFS, but it is built so that lots of people can use it, for lots of different projects.

We will be writing a set of docs, posts, tutorials, and talks to explain what p2p is, why it is tremendously useful, and how it can help your existing and new projects. But in the meantime, check out

• The libp2p Specification
• go-libp2p implementation
• js-libp2p implementation

Bundles

There is currently only one bundle of go-libp2p, this package. This bundle is used by go-ipfs.

Usage

go-libp2p repo is a place holder for the list of Go modules that compose Go libp2p, as well as its entry point.

Install

> go get -u -d github.com/libp2p/go-libp2p/...
> cd $GOPATH/src/github.com/libp2p/go-libp2p
> make
> make deps

API

Examples

Examples can be found in the examples repo.

Development

Dependencies

While developing, you need to use gx to install and link your dependencies, to do that, run:

> make deps

Before commiting and pushing to Github, make sure to rewind the gx'ify of dependencies. You can do that with:

> make publish

Tests

Running of individual tests is done through gx test <path to test>

$ cd $GOPATH/src/github.com/libp2p/go-libp2p
$ make deps
$ gx test ./p2p/<path of module you want to run tests for>

Packages

This table is generated using the module package-table with package-table --data=package-list.json.

List of packages currently in existence for libp2p:

Name	CI/Travis	CI/Jenkins	Coverage	Description
Libp2p
go-libp2p				go-libp2p entry point
go-libp2p-host		N/A		libp2p "host" interface
go-libp2p-blankhost		N/A		minimal implementation of the "host" interface
Network
go-libp2p-net		N/A		libp2p connection and "network" interfaces
go-libp2p-swarm				reference implementation
Transport
go-libp2p-transport		N/A		interfaces
go-ws-transport		N/A		WebSocket transport
go-tcp-transport		N/A		TCP transport
go-libp2p-quic-transport		N/A		QUIC transport
go-udp-transport		N/A		UDP transport
go-utp-transport		N/A		uTorrent transport (UTP)
go-libp2p-circuit				relay transport
go-libp2p-transport-upgrader		N/A		upgrades multiaddr-net connections into full libp2p transports
go-libp2p-reuseport-transport		N/A		partial transport for building transports that reuse ports
Encrypted Channels
go-conn-security		N/A		interfaces
go-libp2p-secio		N/A		SecIO crypto channel
go-conn-security-multistream		N/A		multistream multiplexed meta crypto channel
Private Network
go-libp2p-interface-pnet		N/A		interfaces
go-libp2p-pnet		N/A		reference implementation
Stream Muxers
go-stream-muxer		N/A		interfaces
go-smux-yamux		N/A		YAMUX stream multiplexer
go-smux-mplex		N/A		MPLEX stream multiplexer
NAT Traversal
go-libp2p-nat		N/A
Peerstore
go-libp2p-peerstore		N/A		interfaces and reference implementation
Connection Manager
go-libp2p-interface-connmgr		N/A		interface
go-libp2p-connmgr		N/A		reference implementation
Routing
go-libp2p-routing		N/A		routing interfaces
go-libp2p-record		N/A		record type and validator logic
go-libp2p-routing-helpers		N/A		helpers for composing routers
go-libp2p-kad-dht				Kademlia-like router
go-libp2p-pubsub-router		N/A		record-store over pubsub adapter
Consensus
go-libp2p-consensus		N/A		consensus protocols interfaces
go-libp2p-raft				consensus implementation over raft
Pubsub
go-floodsub				pubsub implementation
Metrics
go-libp2p-metrics		N/A		libp2p metrics interfaces/collectors
Data Types
go-libp2p-peer		N/A		libp2p peer-ID datatype
go-libp2p-crypto		N/A		libp2p key types
go-libp2p-protocol		N/A		libp2p protocol datatype
go-libp2p-kbucket		N/A		Kademlia routing table helper types
Utilities/miscellaneous
go-libp2p-loggables		N/A		logging helpers
go-maddr-filter		N/A		multiaddr filtering helpers
go-libp2p-netutil		N/A		misc utilities
go-msgio		N/A		length prefixed data channel
go-addr-util		N/A		address utilities for libp2p swarm
go-buffer-pool		N/A		a variable size buffer pool for go
go-libp2p-loggables		N/A		logging helpers
go-libp2p-routing-helpers		N/A		routing helpers
go-maddr-filter		N/A		a library to perform filtering of multiaddrs.
go-reuseport		N/A		enables reuse of addresses
go-sockaddr		N/A		utils for sockaddr conversions
go-flow-metrics		N/A		metrics library
Testing and examples
go-testutil		N/A		a collection of testing utilities for ipfs and libp2p
go-libp2p-examples				go-libp2p examples and tutorials
go-libp2p-circuit-progs		N/A		testing programs for go-libp2p-circuit

Contribute

go-libp2p is part of The IPFS Project, and is MIT licensed open source software. We welcome contributions big and small! Take a look at the community contributing notes. Please make sure to check the issues. Search the closed ones before reporting things, and help us with the open ones.

Guidelines:

• read the libp2p spec
• please make branches + pull-request, even if working on the main repository
• ask questions or talk about things in Issues or #ipfs on freenode.
• ensure you are able to contribute (no legal issues please-- we use the DCO)
• run go fmt before pushing any code
• run golint and go vet too -- some things (like protobuf files) are expected to fail.
• get in touch with @jbenet and @diasdavid about how best to contribute
• have fun!

There's a few things you can do right now to help out:

• Go through the modules below and check out existing issues. This would be especially useful for modules in active development. Some knowledge of IPFS/libp2p may be required, as well as the infrasture behind it - for instance, you may need to read up on p2p and more complex operations like muxing to be able to help technically.
• Perform code reviews.
• Add tests. There can never be enough tests.

Modularizing go-libp2p

We have currently a work in progress of modularizing go-libp2p from a repo monolith to several packages in different repos that can be reused for other projects of swapped for custom libp2p builds.

We want to maintain history, so we'll use git-subtree for extracting packages. Find instructions below:

# 1) create the extracted tree (has the directory specified as -P as its root)
> cd go-libp2p/
> git subtree split -P p2p/crypto/secio/ -b libp2p-secio
62b0a5c21574bcbe06c422785cd5feff378ae5bd
# important to delete the tree now, so that outdated imports fail in step 5
> git rm -r p2p/crypto/secio/
> git commit
> cd ../

# 2) make the new repo
> mkdir go-libp2p-secio
> cd go-libp2p-secio/
> git init && git commit --allow-empty

# 3) fetch the extracted tree from the previous repo
> git remote add libp2p ../go-libp2p
> git fetch libp2p
> git reset --hard libp2p/libp2p-secio

# 4) update self import paths
> sed -someflagsidontknow 'go-libp2p/p2p/crypto/secio' 'golibp2p-secio'
> git commit

# 5) create package.json and check all imports are correct
> vim package.json
> gx --verbose install --global
> gx-go rewrite
> go test ./...
> gx-go rewrite --undo
> git commit

# 4) make the package ready
> vim README.md LICENSE
> git commit

# 5) bump the version separately
> vim package.json
> gx publish
> git add package.json .gx/
> git commit -m 'Publish 1.2.3'

# 6) clean up and push
> git remote rm libp2p
> git push origin master

Documentation

Index

• Variables
• func New(ctx context.Context, opts ...Option) (host.Host, error)
• func NewWithoutDefaults(ctx context.Context, opts ...Option) (host.Host, error)
• type Config
• type Option
  • func AddrsFactory(factory config.AddrsFactory) Option
  • func BandwidthReporter(rep metrics.Reporter) Option
  • func ChainOptions(opts ...Option) Option
  • func ConnectionManager(connman ifconnmgr.ConnManager) Option
  • func EnableRelay(options ...circuit.RelayOpt) Option
  • func FilterAddresses(addrs ...*net.IPNet) Option
  • func Identity(sk crypto.PrivKey) Option
  • func ListenAddrStrings(s ...string) Option
  • func ListenAddrs(addrs ...ma.Multiaddr) Option
  • func Muxer(name string, tpt interface{}) Option
  • func NATManager(nm config.NATManagerC) Option
  • func NATPortMap() Option
  • func Peerstore(ps pstore.Peerstore) Option
  • func PrivateNetwork(prot pnet.Protector) Option
  • func Security(name string, tpt interface{}) Option
  • func Transport(tpt interface{}) Option

Variables

var DefaultListenAddrs = func(cfg *Config) error {
	defaultIP4ListenAddr, err := multiaddr.NewMultiaddr("/ip4/0.0.0.0/tcp/0")
	if err != nil {
		return err
	}

	defaultIP6ListenAddr, err := multiaddr.NewMultiaddr("/ip6/::/tcp/0")
	if err != nil {
		return err
	}
	return cfg.Apply(ListenAddrs(
		defaultIP4ListenAddr,
		defaultIP6ListenAddr,
	))
}

DefaultListenAddrs configures libp2p to use default listen address

var DefaultMuxers = ChainOptions(
	Muxer("/yamux/1.0.0", yamux.DefaultTransport),
	Muxer("/mplex/6.7.0", mplex.DefaultTransport),
)

DefaultMuxers configures libp2p to use the stream connection multiplexers.

Use this option when you want to *extend* the set of multiplexers used by libp2p instead of replacing them.

var DefaultSecurity = Security(secio.ID, secio.New)

DefaultSecurity is the default security option.

Useful when you want to extend, but not replace, the supported transport security protocols.

var DefaultTransports = ChainOptions(
	Transport(tcp.NewTCPTransport),
	Transport(ws.New),
)

DefaultTransports are the default libp2p transports.

Use this option when you want to *extend* the set of transports used by libp2p instead of replacing them.

var NoListenAddrs = func(cfg *Config) error {
	cfg.ListenAddrs = []ma.Multiaddr{}
	return nil
}

NoListenAddrs will configure libp2p to not listen by default.

This will both clear any configured listen addrs and prevent libp2p from applying the default listen address option.

var NoTransports = func(cfg *Config) error {
	cfg.Transports = []config.TptC{}
	return nil
}

NoTransports will configure libp2p to not enable any transports.

This will both clear any configured transports (specified in prior libp2p options) and prevent libp2p from applying the default transports.

var RandomIdentity = func(cfg *Config) error {
	priv, _, err := crypto.GenerateKeyPairWithReader(crypto.RSA, 2048, rand.Reader)
	if err != nil {
		return err
	}
	return cfg.Apply(Identity(priv))
}

RandomIdentity generates a random identity (default behaviour)

Functions

func New

func New(ctx context.Context, opts ...Option) (host.Host, error)

New constructs a new libp2p node with the given options, falling back on reasonable defaults. The defaults are:

- If no transport and listen addresses are provided, the node listens to the multiaddresses "/ip4/0.0.0.0/tcp/0" and "/ip6/::/tcp/0";

- If no transport options are provided, the node uses TCP and websocket transport protocols;

- If no multiplexer configuration is provided, the node is configured by default to use the "yamux/1.0.0" and "mplux/6.7.0" stream connection multiplexers;

- If no security transport is provided, the host uses the go-libp2p's secio encrypted transport to encrypt all traffic;

- If no peer identity is provided, it generates a random RSA 2048 key-par and derives a new identity from it;

- If no peerstore is provided, the host is initialized with an empty peerstore.

Canceling the passed context will stop the returned libp2p node.

func NewWithoutDefaults

func NewWithoutDefaults(ctx context.Context, opts ...Option) (host.Host, error)

NewWithoutDefaults constructs a new libp2p node with the given options but *without* falling back on reasonable defaults.

Warning: This function should not be considered a stable interface. We may choose to add required services at any time and, by using this function, you opt-out of any defaults we may provide.

Types

type Config

type Config = config.Config

Config describes a set of settings for a libp2p node

type Option

type Option = config.Option

Option is a libp2p config option that can be given to the libp2p constructor (`libp2p.New`).

var DefaultPeerstore Option = func(cfg *Config) error {
	return cfg.Apply(Peerstore(pstoremem.NewPeerstore()))
}

DefaultPeerstore configures libp2p to use the default peerstore.

var Defaults Option = func(cfg *Config) error {
	for _, def := range defaults {
		if err := cfg.Apply(def.opt); err != nil {
			return err
		}
	}
	return nil
}

Defaults configures libp2p to use the default options. Can be combined with other options to *extend* the default options.

var FallbackDefaults Option = func(cfg *Config) error {
	for _, def := range defaults {
		if !def.fallback(cfg) {
			continue
		}
		if err := cfg.Apply(def.opt); err != nil {
			return err
		}
	}
	return nil
}

FallbackDefaults applies default options to the libp2p node if and only if no other relevent options have been applied. will be appended to the options passed into New.

var NoSecurity Option = func(cfg *Config) error {
	if len(cfg.SecurityTransports) > 0 {
		return fmt.Errorf("cannot use security transports with an insecure libp2p configuration")
	}
	cfg.Insecure = true
	return nil
}

NoSecurity is an option that completely disables all transport security. It's incompatible with all other transport security protocols.

func AddrsFactory

func AddrsFactory(factory config.AddrsFactory) Option

AddrsFactory configures libp2p to use the given address factory.

func BandwidthReporter

func BandwidthReporter(rep metrics.Reporter) Option

BandwidthReporter configures libp2p to use the given bandwidth reporter.

func ChainOptions

func ChainOptions(opts ...Option) Option

ChainOptions chains multiple options into a single option.

func ConnectionManager

func ConnectionManager(connman ifconnmgr.ConnManager) Option

ConnectionManager configures libp2p to use the given connection manager.

func EnableRelay

func EnableRelay(options ...circuit.RelayOpt) Option

EnableRelay configures libp2p to enable the relay transport.

func FilterAddresses

func FilterAddresses(addrs ...*net.IPNet) Option

FilterAddresses configures libp2p to never dial nor accept connections from the given addresses.

func Identity

func Identity(sk crypto.PrivKey) Option

Identity configures libp2p to use the given private key to identify itself.

func ListenAddrStrings

func ListenAddrStrings(s ...string) Option

ListenAddrStrings configures libp2p to listen on the given (unparsed) addresses.

func ListenAddrs

func ListenAddrs(addrs ...ma.Multiaddr) Option

ListenAddrs configures libp2p to listen on the given addresses.

func Muxer

func Muxer(name string, tpt interface{}) Option

Muxer configures libp2p to use the given stream multiplexer (or stream multiplexer constructor).

Name is the protocol name.

The transport can be a constructed mux.Transport or a function taking any subset of this libp2p node's: * Peer ID * Host * Network * Peerstore

func NATManager

func NATManager(nm config.NATManagerC) Option

NATManager will configure libp2p to use the requested NATManager. This function should be passed a NATManager *constructor* that takes a libp2p Network.

func NATPortMap

func NATPortMap() Option

NATPortMap configures libp2p to use the default NATManager. The default NATManager will attempt to open a port in your network's firewall using UPnP.

func Peerstore

func Peerstore(ps pstore.Peerstore) Option

Peerstore configures libp2p to use the given peerstore.

func PrivateNetwork

func PrivateNetwork(prot pnet.Protector) Option

PrivateNetwork configures libp2p to use the given private network protector.

func Security

func Security(name string, tpt interface{}) Option

Security configures libp2p to use the given security transport (or transport constructor).

Name is the protocol name.

The transport can be a constructed security.Transport or a function taking any subset of this libp2p node's: * Public key * Private key * Peer ID * Host * Network * Peerstore

func Transport

func Transport(tpt interface{}) Option

Transport configures libp2p to use the given transport (or transport constructor).

The transport can be a constructed transport.Transport or a function taking any subset of this libp2p node's: * Transport Upgrader (*tptu.Upgrader) * Host * Stream muxer (muxer.Transport) * Security transport (security.Transport) * Private network protector (pnet.Protector) * Peer ID * Private Key * Public Key * Address filter (filter.Filter) * Peerstore