measurex/

README

Using the measurex package to write network experiments

Note: measurex is deprecated. We will remove this tutorial once we have a tutorial for the new DSL-based measurement package.

This tutorial teaches you how to write OONI network experiments using the primitives in the ./internal/measurex package. The name of this package means either "measure extensions" or "measure crossover".

The measure extension interpretation of the name explains what this package does. It contains extensions to our basic networking code (./internal/netxlite) that allow us to perform OONI measurements.

The measure crossover interpretation explains its history. Since OONI has been written in Go, we've mostly performed measurements using "tracing". That is, by registering hooks that run when specific operations happen (e.g., TCP connect or TLS handshake) and then making sense of the network trace. This is the approach with which most experiments are written as of 2021-09-24. But we have also seen that in several cases a step-by-step approach is preferrable. Under this approach, you perform individual operations and record their result right away. So, for example, you have a connection and a TLS config, you perform a TLS handshake, and immediately after you create and store somewhere a data structure containing the result. This package is at the crossover of these two approaches, because basically it contains enough primitives to support both.

What we are going to do in this tutorial is the following:

• we will start from very simple-step-by-step measurements such as TCP connect, DNS lookup, and TLS handshake;

• we will see how measurex provides support for composing these primitives in larger steps, which will lead us to eventually perform all the measurements that matter for a given input URL (including discovering QUIC endpoints and following redirections);

• finally, as an exercise, we will use the knowledge acquired in the rest of the tutorial to rewrite a subset of the Web Connectivity experiment (as of 2021-09-24 the flagship OONI experiment). This will be an opportunity to explore more low level aspects of measurex.

As part of the process, we'll introduce you to the data format used by OONI and there will be proposed exercises where we simulate censorship conditions and we see how that impacts the generated measurements.

Every chapter will show how to write a simple main.go program that explains how to use some primitives. The chapter text itself is autogenerated from comments inside the actual main.go the we describe in the chapter.

For this reason, if you need to change a chapter, you need to change the corresponding main.go file and then follow the instructions at ./internal/tutorial/generate to regenerate the markdown text of the chapter.

More in detail, here's the index:

• chapter01 explains how to use the "system" resolver

• chapter02 deals with establishing TCP connections

• chapter03 is about using custom DNS-over-UDP resolvers

• chapter04 shows how to measure TLS handshakes

• chapter05 is about the QUIC handshake

• chapter06 shows how to get a webpage knowing its URL and the endpoint (i.e., IP address and TCP/UDP port)

• chapter07 shows how to extend what we did in chapter06 to cover all the IP addresses in the URL's domain

• chapter08 is about HTTPSSvc DNS queries and how they can be used to discover and test QUIC endpoints, thus extending the work done in chapter07

• chapter09 improves upon chapter08 showing how to run endpoints measurements in parallel

• chapter10 improves upon chapter09 by also running DNS queries in parallel

• chapter11 tells you that all the code we have been writing so far, and specifically the code we have in chapter10, is actually the implementation of an API of measurex called MeasureURL, and then shows you how you can simplify the code in chapter10 by using this API.

• chapter12 extends the work done in chapter11 by teaching you about a more high-level API that discovers and follows all redirections, calling MeasureURL for each redirection.

• chapter13 contains the exercise regarding rewriting WebConnectivity using all the tools you have learned so far and pointing you at additional measurex API that could be useful to solve the problem.

• chapter14 contains our solution to the exercise.