README
¶
OONI Probe Measurement Engine and CLI
User guide · API docs · Report bug · Request feature · Tutorials
Table of Contents
About this project
The Open Observatory of Network Interference (OONI) is a non-profit free software project that aims to empower decentralized efforts in documenting Internet censorship around the world.
This repository contains the following Go packages:
-
the command line client (cmd/ooniprobe);
-
the test helper server (internal/cmd/oohelperd);
-
the mobile library (pkg/oonimkall);
-
the measurement-engine library (internal).
Every top-level directory in this repository contains an explanatory README file.
Install instructions
Follow the instructions at ooni.org/install/cli
to install ooniprobe precompiled binaries for Windows, macOS, and
Debian/Ubuntu. Once ooniprobe is installed, refer to the
user guide.
Developer instructions
This repository requires exactly the Go version mentioned by the GOVERSION file (i.e., go1.20.10). Using a different version of Go may work as intended but is not recommended: we depend on packages forked from the standard library; so, it is more robust to use the same version of Go from which we forked those packages from.
You will also need a C compiler. On Linux and other Unix systems both GCC and Clang will work. If you're using Windows, we recommend installing Ubuntu or Debian on the Windows Subsystem for Linux. If you're targeting Windows, you should also install the mingw-w64 cross-compiler.
Debian developer setup
The following commands show how to setup a development environment using Debian 12 ("bookworm"). The same instructions should also work for Debian-based distribution (e.g., Ubuntu).
# install the compilers, git, and the root CA
sudo apt install golang build-essential ca-certificates git
# [optional] install mingw-w64 if you're targeting windows
sudo apt install mingw-w64
# install the required go version binary
go install -v golang.org/dl/go1.20.10@latest
# fetch the whole go distribution
$HOME/go/bin/go1.20.10 download
Fedora developer setup
The following commands show how to setup a development environment using Fedora.
# install the compilers and git
sudo dnf install golang make gcc gcc-c++ git
# [optional] install mingw-w64 if you're targeting windows
sudo dnf install mingw64-gcc mingw64-gcc-c++
# install the required go version binary
go install -v golang.org/dl/go1.20.10@latest
# fetch the whole go distribution
$HOME/go/bin/go1.20.10 download
macOS developer setup
The following commands show how to setup a development environment using macOS. We assume you have already installed Homebrew, which should also install the Xcode command line tools.
Then, you need to follow these instructions:
# install the compiler
brew install go
# install the required go version binary
go install -v golang.org/dl/go1.20.10@latest
# fetch the whole go distribution
$HOME/go/bin/go1.20.10 download
Build instructions
Once you have installed the correct Go version and a C compiler,
you can compile ooniprobe using:
$HOME/go/bin/go1.20.10 build -v -ldflags '-s -w' ./cmd/ooniprobe
This command will generate a stripped binary called ooniprobe
in the toplevel directory.
Likewise, you can compile miniooni using:
$HOME/go/bin/go1.20.10 build -v -ldflags '-s -w' ./internal/cmd/miniooni
This command will generate a stripped binary called miniooni
in the toplevel directory.
And oohelperd using:
$HOME/go/bin/go1.20.10 build -v -ldflags '-s -w' ./internal/cmd/oohelperd
This command will generate a stripped binary called oohelperd
in the toplevel directory.
Contributing
Please, see CONTRIBUTING.md.
License
SPDX-License-Identifier: GPL-3.0-or-later
Updating dependencies
go get -t -u -v ./... && go mod tidy
Releasing
Create an issue according to the routine release template and perform any item inside the check-list.
We build releases using Makefile, which requires GNU make. Run
make help for detailed usage.
Semantic versioning policy
The mobile library is a public package for technical reasons. Go mobile tools require
a public package to build from. Yet, we don't consider API breakages happening in
such a package to be sufficient to bump our major version number. For us, the mobile
library is just a mean to implement OONI Probe Android and OONI Probe iOS. We'll
only bump the major version number if we change ./cmd/ooniprobe's CLI.
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
ooniprobe/internal/autorun
Package autorun contains code to manage automatic runs
|
Package autorun contains code to manage automatic runs |
|
ooniprobe/internal/log/handlers/cli/progress
Package progress provides a simple terminal progress bar.
|
Package progress provides a simple terminal progress bar. |
|
ooniprobe/internal/log/handlers/syslog
Package syslog contains a syslog handler.
|
Package syslog contains a syslog handler. |
|
ooniprobe/internal/oonitest
Package oonitest contains code used for testing.
|
Package oonitest contains code used for testing. |
|
internal
|
|
|
bytecounter
Package bytecounter contains code to track the number of bytes sent and received by a probe.
|
Package bytecounter contains code to track the number of bytes sent and received by a probe. |
|
checkincache
Package checkincache contains an on-disk cache for check-in responses.
|
Package checkincache contains an on-disk cache for check-in responses. |
|
cmd/apitool
Command apitool is a simple tool to fetch individual OONI measurements.
|
Command apitool is a simple tool to fetch individual OONI measurements. |
|
cmd/buildtool/internal/buildtoolmodel
Package buildtoolmodel contains the model for buildtool.
|
Package buildtoolmodel contains the model for buildtool. |
|
cmd/buildtool/internal/buildtooltest
Package buildtooltest contains code for testing buildtool.
|
Package buildtooltest contains code for testing buildtool. |
|
cmd/gardener
Command gardener helps with test-lists management.
|
Command gardener helps with test-lists management. |
|
cmd/gardener/internal/aggregationapi
Package aggregationapi allows one to query the OONI aggregation API.
|
Package aggregationapi allows one to query the OONI aggregation API. |
|
cmd/gardener/internal/dnsfix
Package dnsfix implements the dnsfix command.
|
Package dnsfix implements the dnsfix command. |
|
cmd/gardener/internal/dnsreport
Package dnsreport implements the dnsreport subcommand.
|
Package dnsreport implements the dnsreport subcommand. |
|
cmd/gardener/internal/sync
Package sync implements the sync subcommand.
|
Package sync implements the sync subcommand. |
|
cmd/gardener/internal/testlists
Package testlists contains code to read/rewrite the test lists.
|
Package testlists contains code to read/rewrite the test lists. |
|
cmd/getresources
Command getresources downloads the resources
|
Command getresources downloads the resources |
|
cmd/ghgen
Command ghgen regenerates selected GitHub actions.
|
Command ghgen regenerates selected GitHub actions. |
|
cmd/miniooni
Command miniooni is a simple binary for research and QA purposes with a CLI interface similar to MK and OONI Probe v2.x.
|
Command miniooni is a simple binary for research and QA purposes with a CLI interface similar to MK and OONI Probe v2.x. |
|
cmd/oohelper
Command oohelper contains a simple command line client for the Web Connectivity test helper.
|
Command oohelper contains a simple command line client for the Web Connectivity test helper. |
|
cmd/oohelperd
Command oohelperd implements the Web Connectivity test helper.
|
Command oohelperd implements the Web Connectivity test helper. |
|
cmd/oonireport
Command oonireport uploads reports stored on disk to the OONI collector.
|
Command oonireport uploads reports stored on disk to the OONI collector. |
|
cmd/ooporthelper
command ooporthelper implements the Port Filtering test helper
|
command ooporthelper implements the Port Filtering test helper |
|
cmd/printversion
Command printversion prints the current version of this repository.
|
Command printversion prints the current version of this repository. |
|
cmd/tinyjafar
Command tinyjafar implements a subset of the CLI flags of the original jafar tool.
|
Command tinyjafar implements a subset of the CLI flags of the original jafar tool. |
|
dslx
Package dslx implements a domain specific language for writing network experiments.
|
Package dslx implements a domain specific language for writing network experiments. |
|
engine
Package engine contains the engine API.
|
Package engine contains the engine API. |
|
enginelocate
Package enginelocate implements IP lookup, resolver lookup, and geolocation.
|
Package enginelocate implements IP lookup, resolver lookup, and geolocation. |
|
enginenetx
Package enginenetx contains engine-specific network-extensions.
|
Package enginenetx contains engine-specific network-extensions. |
|
engineresolver
Package engineresolver contains the resolver used by the OONI engine.
|
Package engineresolver contains the resolver used by the OONI engine. |
|
experiment/dash
Package dash implements the DASH network experiment.
|
Package dash implements the DASH network experiment. |
|
experiment/dnscheck
Package dnscheck contains the DNS check experiment.
|
Package dnscheck contains the DNS check experiment. |
|
experiment/dnsping
Package dnsping is the experimental dnsping experiment.
|
Package dnsping is the experimental dnsping experiment. |
|
experiment/echcheck
Package echcheck contains the ECH blocking network experiment.
|
Package echcheck contains the ECH blocking network experiment. |
|
experiment/example
Package example contains a simple example experiment.
|
Package example contains a simple example experiment. |
|
experiment/fbmessenger
Package fbmessenger contains the Facebook Messenger network experiment.
|
Package fbmessenger contains the Facebook Messenger network experiment. |
|
experiment/hhfm
Package hhfm contains the HTTP Header Field Manipulation network experiment.
|
Package hhfm contains the HTTP Header Field Manipulation network experiment. |
|
experiment/hirl
Package hirl contains the HTTP Invalid Request Line network experiment.
|
Package hirl contains the HTTP Invalid Request Line network experiment. |
|
experiment/httphostheader
Package httphostheader contains the HTTP host header network experiment.
|
Package httphostheader contains the HTTP host header network experiment. |
|
experiment/ndt7
Package ndt7 contains the ndt7 network experiment.
|
Package ndt7 contains the ndt7 network experiment. |
|
experiment/portfiltering
Package portfiltering implements the portfiltering experiment
|
Package portfiltering implements the portfiltering experiment |
|
experiment/psiphon
Package psiphon implements the psiphon network experiment.
|
Package psiphon implements the psiphon network experiment. |
|
experiment/quicping
Package quicping implements the quicping network experiment.
|
Package quicping implements the quicping network experiment. |
|
experiment/riseupvpn
Package riseupvpn contains the RiseupVPN network experiment.
|
Package riseupvpn contains the RiseupVPN network experiment. |
|
experiment/run
Package run contains code to run other experiments.
|
Package run contains code to run other experiments. |
|
experiment/signal
Package signal contains the Signal network experiment.
|
Package signal contains the Signal network experiment. |
|
experiment/simplequicping
Package simplequicping is the experimental simplequicping experiment.
|
Package simplequicping is the experimental simplequicping experiment. |
|
experiment/sniblocking
Package sniblocking contains the SNI blocking network experiment.
|
Package sniblocking contains the SNI blocking network experiment. |
|
experiment/stunreachability
Package stunreachability contains the STUN reachability experiment.
|
Package stunreachability contains the STUN reachability experiment. |
|
experiment/tcpping
Package tcpping is the experimental tcpping experiment.
|
Package tcpping is the experimental tcpping experiment. |
|
experiment/telegram
Package telegram contains the Telegram network experiment.
|
Package telegram contains the Telegram network experiment. |
|
experiment/tlsmiddlebox
Package tlsmiddlebox implements the tlsmiddlebox experiment
|
Package tlsmiddlebox implements the tlsmiddlebox experiment |
|
experiment/tlsping
Package tlsping is the experimental tlsping experiment.
|
Package tlsping is the experimental tlsping experiment. |
|
experiment/tlstool
Package tlstool contains a TLS tool that we are currently using for running quick and dirty experiments.
|
Package tlstool contains a TLS tool that we are currently using for running quick and dirty experiments. |
|
experiment/tlstool/internal
Package internal contains the implementation of tlstool.
|
Package internal contains the implementation of tlstool. |
|
experiment/tor
Package tor contains the tor experiment.
|
Package tor contains the tor experiment. |
|
experiment/torsf
Package torsf contains the torsf experiment.
|
Package torsf contains the torsf experiment. |
|
experiment/urlgetter
Package urlgetter implements a nettest that fetches a URL.
|
Package urlgetter implements a nettest that fetches a URL. |
|
experiment/vanillator
Package vanillator contains the vanilla_tor experiment.
|
Package vanillator contains the vanilla_tor experiment. |
|
experiment/webconnectivity
Package webconnectivity implements OONI's Web Connectivity experiment.
|
Package webconnectivity implements OONI's Web Connectivity experiment. |
|
experiment/webconnectivity/internal
Package internal contains internal code.
|
Package internal contains internal code. |
|
experiment/webconnectivitylte
Package webconnectivitylte implements the web_connectivity experiment.
|
Package webconnectivitylte implements the web_connectivity experiment. |
|
experiment/webconnectivityqa
Package webconnectivityqa contains code to perform Web Connectivity QA.
|
Package webconnectivityqa contains code to perform Web Connectivity QA. |
|
experiment/whatsapp
Package whatsapp contains the WhatsApp network experiment.
|
Package whatsapp contains the WhatsApp network experiment. |
|
flagx
Package flagx contains extensions for the standard library flag package.
|
Package flagx contains extensions for the standard library flag package. |
|
fsx
Package fsx contains io/fs extensions.
|
Package fsx contains io/fs extensions. |
|
geoipx
Package geoipx contains code to use the embedded MaxMind-like databases.
|
Package geoipx contains code to use the embedded MaxMind-like databases. |
|
httpapi
Package httpapi contains code for calling HTTP APIs.
|
Package httpapi contains code for calling HTTP APIs. |
|
httpx
Package httpx contains http extensions.
|
Package httpx contains http extensions. |
|
hujsonx
Package hujsonx contains github.com/tailscale/hujson extensions.
|
Package hujsonx contains github.com/tailscale/hujson extensions. |
|
humanize
Package humanize is like dustin/go-humanize.
|
Package humanize is like dustin/go-humanize. |
|
inputparser
Package inputparser contains code to parse experiments input.
|
Package inputparser contains code to parse experiments input. |
|
kvstore
Package kvstore implements model.KeyValueStore.
|
Package kvstore implements model.KeyValueStore. |
|
legacy/assetsdir
Package assetsdir contains code to cleanup the assets dir.
|
Package assetsdir contains code to cleanup the assets dir. |
|
legacy/legacymodel
Package legacymodel contains legacy content that used to be in internal/model
|
Package legacymodel contains legacy content that used to be in internal/model |
|
legacy/measurex
Package measurex contains measurement extensions.
|
Package measurex contains measurement extensions. |
|
legacy/mockable
Package mockable contains mockable objects
|
Package mockable contains mockable objects |
|
legacy/netx
Package netx contains code to perform network measurements.
|
Package netx contains code to perform network measurements. |
|
legacy/tracex
Package tracex performs measurements using tracing.
|
Package tracex performs measurements using tracing. |
|
logx
Package logx contains logging extensions.
|
Package logx contains logging extensions. |
|
measurexlite
Package measurexlite contains measurement extensions.
|
Package measurexlite contains measurement extensions. |
|
memoryless
Package memoryless helps repeated calls to a function be distributed across time in a memoryless fashion.
|
Package memoryless helps repeated calls to a function be distributed across time in a memoryless fashion. |
|
mlablocatev2
Package mlablocatev2 implements m-lab locate services API v2.
|
Package mlablocatev2 implements m-lab locate services API v2. |
|
mocks
Package mocks contains mocks for implementing unit tests.
|
Package mocks contains mocks for implementing unit tests. |
|
model
Package model contains the shared interfaces and data structures.
|
Package model contains the shared interfaces and data structures. |
|
multierror
Package multierror contains code to manage multiple errors.
|
Package multierror contains code to manage multiple errors. |
|
must
Package must contains functions that panic on error.
|
Package must contains functions that panic on error. |
|
netemx
Package netemx contains code to run integration tests using github.com/ooni/netem.
|
Package netemx contains code to run integration tests using github.com/ooni/netem. |
|
netxlite
Package netxlite contains network extensions.
|
Package netxlite contains network extensions. |
|
ooapi
Package ooapi describes how the OONI API works.
|
Package ooapi describes how the OONI API works. |
|
oonirun
Package oonirun contains code to run OONI experiments.
|
Package oonirun contains code to run OONI experiments. |
|
optional
Package optional contains safer code to handle optional values.
|
Package optional contains safer code to handle optional values. |
|
platform
Package platform allows you to obtain the platform name.
|
Package platform allows you to obtain the platform name. |
|
probeservices
Package probeservices contains code to contact OONI probe services.
|
Package probeservices contains code to contact OONI probe services. |
|
progress
Package progress contains utilities to emit progress.
|
Package progress contains utilities to emit progress. |
|
ptx
Package ptx contains code to use pluggable transports.
|
Package ptx contains code to use pluggable transports. |
|
randx
Package randx contains math/rand extensions.
|
Package randx contains math/rand extensions. |
|
registry
Package registry contains a registry of all the available experiments.
|
Package registry contains a registry of all the available experiments. |
|
runtimex
Package runtimex contains runtime extensions.
|
Package runtimex contains runtime extensions. |
|
shellx
Package shellx helps to write shell-like Go code.
|
Package shellx helps to write shell-like Go code. |
|
shellx/shellxtesting
Package shellxtesting supports shellx testing.
|
Package shellxtesting supports shellx testing. |
|
strcasex
Package strcasex converts strings to various cases.
|
Package strcasex converts strings to various cases. |
|
stuninput
Package stuninput contains stun targets as well as code to format such targets according to various conventions.
|
Package stuninput contains stun targets as well as code to format such targets according to various conventions. |
|
testingproxy
Package testingproxy contains shared test cases for the proxies.
|
Package testingproxy contains shared test cases for the proxies. |
|
testingquic
Package testingquic allows to retrieve the domain and endpoint to use for all the integration tests that use QUIC.
|
Package testingquic allows to retrieve the domain and endpoint to use for all the integration tests that use QUIC. |
|
testingsocks5
Package testingsock5 is a netem-aware fork of https://github.com/armon/go-socks5.
|
Package testingsock5 is a netem-aware fork of https://github.com/armon/go-socks5. |
|
testingx
Package testingx contains code useful for testing.
|
Package testingx contains code useful for testing. |
|
throttling
Package throttling wraps connections to measure throttling.
|
Package throttling wraps connections to measure throttling. |
|
torlogs
Package torlogs contains code to read tor logs.
|
Package torlogs contains code to read tor logs. |
|
tunnel
Package tunnel allows to create tunnels to speak with OONI backends and other services.
|
Package tunnel allows to create tunnels to speak with OONI backends and other services. |
|
tunnel/mocks
Package mocks contains mocks for tunnel.
|
Package mocks contains mocks for tunnel. |
|
tutorial/dslx/chapter02
## Getting started
|
## Getting started |
|
tutorial/experiment/torsf/chapter01
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/experiment/torsf/chapter02
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/generator
Command generator generates or re-generates the tutorial chapters.
|
Command generator generates or re-generates the tutorial chapters. |
|
tutorial/measurex/chapter01
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/measurex/chapter02
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/measurex/chapter03
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/measurex/chapter04
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/measurex/chapter05
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/measurex/chapter06
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/measurex/chapter07
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/measurex/chapter08
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/measurex/chapter09
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/measurex/chapter10
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/measurex/chapter11
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/measurex/chapter12
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/measurex/chapter13
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/measurex/chapter14
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/netxlite/chapter01
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/netxlite/chapter02
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/netxlite/chapter03
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/netxlite/chapter04
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/netxlite/chapter05
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/netxlite/chapter06
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/netxlite/chapter07
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/netxlite/chapter08
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
version
Package version contains version information.
|
Package version contains version information. |
|
pkg
|
|
|
oonimkall
Package oonimkall implements APIs used by OONI mobile apps.
|
Package oonimkall implements APIs used by OONI mobile apps. |
Click to show internal directories.
Click to hide internal directories.