internal/

README

Directory github.com/ooni/probe-cli/internal

This directory contains private Go packages.

Useful commands

You can read the Go documentation of a package by using go doc -all.

For example:

go doc -all ./internal/netxlite

You can get a graph of the dependencies using kisielk/godepgraph.

For example:

godepgraph -s -novendor -p golang.org,gitlab.com ./internal/engine | dot -Tpng -o deps.png

You can further tweak which packages to exclude by appending prefixes to the list passed to the -p flag.

Tutorials

The tutorial package contains tutorials on writing new experiments, using measurements libraries, and networking code.

Network extensions

This section briefly describes the overall design of the network extensions (aka netx) inside ooni/probe-cli. In OONI, we have two distinct but complementary needs:

1. speaking with our backends or accessing other services useful to bootstrap OONI probe and perform measurements;

2. implementing network experiments.

We originally implemented these functionality into a separate repository: ooni/netx. The original design document still provides a good overview of the problems we wanted to solve. The newer dd-002-step-by-step.md design document describes the current architecture (as of 2022-06-17) and the future trajectory for netx.

The general idea of netx has always been to provide interfaces replacing standard library objects that we could further wrap to perform network measurements without deviating from the normal APIs expected by Go programmers.

For example,

type Dialer interface {
	DialContext(ctx context.Context, network, address string) (net.Conn, error)
}

is a generic dialer that could be a &net.Dialer{} but could also be a saving dialer that saves the results of dial events. So, you could write something like:

saver := &Saver{}
var dialer Dialer = NewDialer()
dialer = saver.WrapDialer(dialer)
conn, err := dialer.DialContext(ctx, network, address)
events := saver.ExtractEvents()

In short, with the original netx you could write measurement code resembling ordinary Go code but you could also save network events from which to derive whether there was censorship.

Since then, the architecture itself has evolved and netx has been merged into ooni/probe-engine and later ooni/probe-cli. As of 2022-06-06, these are the fundamental netx packages:

• model/netx.go: contains the interfaces and structs patterned after the Go standard library used by netx;

• netxlite: implements error wrapping (i.e., mapping Go errors to OONI errors), enforces timeouts, and generally ensures that we're using a stdlib-like network API that meet all our constraints and requirements (e.g., logging);

• bytecounter: provides support for counting the number of bytes consumed by network interactions;

• multierror: defines an error type that contains a list of errors for representing the results of operations where multiple sub-operations may fail (e.g., TCP connect fails for all the IP addresses associated with a domain name);

• tracex: support for collecting events during operations such as TCP connect, QUIC handshake, HTTP round trip. Collecting events allows us to analyze such events and determine whether there was blocking. This measurement strategy is called tracing because we wrap fundamental types (e.g., a dialer or an HTTP transport) to save the result of each operation into a "list of events" type called `Saver;

• engine/netx: code surviving from the original netx implementation that we're still using for measuring. Issue ooni/probe#2121 describes a slow refactoring process where we'll move code outside of netx and inside netxlite or other packages. We are currently experimenting with step-by-step measurements, an alternative measurement approach where we break down operations in simpler building blocks. This alternative approach may eventually make netx obsolete.