Using the measurex package to write network experiments
This tutorial teaches you how to write OONI network
experiments using the primitives in the ./internal/measurex
package. The name of this package means either "measure
extensions" or "measure crossover".
The measure extension interpretation of the name explains
what this package does. It contains extensions to our
basic networking code (./internal/netxlite
) that allow
us to perform OONI measurements.
The measure crossover interpretation explains its history. Since
OONI has been written in Go, we've mostly performed measurements
using "tracing". That is, by registering hooks that run when
specific operations happen (e.g., TCP connect or TLS handshake)
and then making sense of the network trace. This is the approach
with which most experiments are written as of 2021-09-24. But
we have also seen that in several cases a step-by-step approach
is preferrable. Under this approach, you perform individual
operations and record their result right away. So, for example,
you have a connection and a TLS config, you perform a TLS
handshake, and immediately after you create and store somewhere
a data structure containing the result. This package is at the
crossover of these two approaches, because basically it contains
enough primitives to support both.
What we are going to do in this tutorial is the following:
-
we will start from very simple-step-by-step measurements such
as TCP connect, DNS lookup, and TLS handshake;
-
we will see how measurex
provides support for composing
these primitives in larger steps, which will lead us to
eventually perform all the measurements that matter for a
given input URL (including discovering QUIC endpoints
and following redirections);
-
finally, as an exercise, we will use the knowledge
acquired in the rest of the tutorial to rewrite a
subset of the Web Connectivity experiment (as of 2021-09-24
the flagship OONI experiment). This will be an opportunity
to explore more low level aspects of measurex
.
As part of the process, we'll introduce you to the data
format used by OONI and there will be proposed exercises
where we simulate censorship conditions and we see how
that impacts the generated measurements.
Every chapter will show how to write a simple main.go
program that explains how to use some primitives. The
chapter text itself is autogenerated from comments inside
the actual main.go
the we describe in the chapter.
For this reason, if you need to change a chapter, you
need to change the corresponding main.go
file and then
follow the instructions at ./internal/tutorial/generate
to regenerate the markdown text of the chapter.
More in detail, here's the index:
-
chapter01 explains how to use the "system" resolver
-
chapter02 deals with establishing TCP connections
-
chapter03 is about using custom DNS-over-UDP resolvers
-
chapter04 shows how to measure TLS handshakes
-
chapter05 is about the QUIC handshake
-
chapter06 shows how to get a webpage knowing its
URL and the endpoint (i.e., IP address and TCP/UDP port)
-
chapter07 shows how to extend what we did in
chapter06 to cover all the IP addresses in the URL's domain
-
chapter08 is about HTTPSSvc DNS queries and
how they can be used to discover and test QUIC endpoints, thus
extending the work done in chapter07
-
chapter09 improves upon chapter08 showing
how to run endpoints measurements in parallel
-
chapter10 improves upon chapter09 by
also running DNS queries in parallel
-
chapter11 tells you that all the code we
have been writing so far, and specifically the code we have
in chapter10, is actually the implementation of an API
of measurex
called MeasureURL
, and then shows you how
you can simplify the code in chapter10 by using this API.
-
chapter12 extends the work done in
chapter11 by teaching you about a more high-level API
that discovers and follows all redirections, calling
MeasureURL
for each redirection.
-
chapter13 contains the exercise regarding
rewriting WebConnectivity using all the tools you have
learned so far and pointing you at additional measurex
API that could be useful to solve the problem.
-
chapter14 contains our solution to the exercise.