README
¶
OONI Probe Client Library and CLI
The Open Observatory of Network Interference (OONI) is a non-profit free software project that aims to empower decentralized efforts in documenting Internet censorship around the world.
This repository contains core OONI tools written in Go:
-
the CLI client (cmd/ooniprobe);
-
the test helper server (internal/cmd/oohelperd);
-
the mobile library (pkg/oonimkall);
-
and all the related support packages (inside internal).
Every top-level directory in this repository contains an explanatory README file.
User setup
Please, follow the instructions at ooni.org/install/cli
to install ooniprobe. If we do not support your use case, please let us know. Once
ooniprobe is installed, try ooniprobe help to get interactive help.
Reporting issues
Report issues at github.com/ooni/probe.
Please, make sure you add the ooni/probe-cli label.
Build instructions
ooniprobe
Be sure you have golang >= 1.17 and a C compiler (Mingw-w64 for Windows). You can build using:
go build -v ./cmd/ooniprobe
This will generate a binary called ooniprobe in the current directory.
Android bindings
Make sure you have GNU make installed, then run:
./mk android
to build bindings for Android. (Add OONI_PSIPHON_TAGS="" if you
cannot clone private repositories in the https://github.com/ooni namespace.)
The generated bindings are (manually) pushed to the Maven Central package repository. The instructions explaining how to integrate these bindings are published along with the release notes.
iOS bindings
Make sure you have GNU make installed, then run:
./mk ios
to build bindings for iOS. (Add OONI_PSIPHON_TAGS="" if you
cannot clone private repositories in the https://github.com/ooni namespace.)
The generated bindings are (manually) added to GitHub releases. The instructions explaining how to integrate these bindings are published along with the release notes.
miniooni
Miniooni is the experimental OONI client used for research. Compile using:
go build -v ./internal/cmd/miniooni
This will generate a binary called miniooni in the current directory.
oohelperd
Oohelperd is the test helper server. Compile using:
go build -v ./internal/cmd/oohelperd
This will generate a binary called oohelperd in the current directory.
Specifications
Every nettest (aka experiment) implemented in this repository has a companion spec in the ooni/spec repository.
Contributing
Please, see CONTRIBUTING.md.
Updating dependencies
go get -u -v ./... && go mod tidy
Releasing
Create an issue according to the routine release template and perform any item inside the check-list.
We build releases using ./mk, which requires GNU make. Try
the ./mk help|less command for detailed usage.
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
ooniprobe/internal/autorun
Package autorun contains code to manage automatic runs
|
Package autorun contains code to manage automatic runs |
|
ooniprobe/internal/enginex
Package enginex contains ooni/probe-engine extensions.
|
Package enginex contains ooni/probe-engine extensions. |
|
ooniprobe/internal/log/handlers/cli/progress
Package progress provides a simple terminal progress bar.
|
Package progress provides a simple terminal progress bar. |
|
ooniprobe/internal/log/handlers/syslog
Package syslog contains a syslog handler.
|
Package syslog contains a syslog handler. |
|
ooniprobe/internal/oonitest
Package oonitest contains code used for testing.
|
Package oonitest contains code used for testing. |
|
internal
|
|
|
atomicx
Package atomicx extends sync/atomic.
|
Package atomicx extends sync/atomic. |
|
bytecounter
Package bytecounter contains code to track the number of bytes sent and received by a probe.
|
Package bytecounter contains code to track the number of bytes sent and received by a probe. |
|
cmd/apitool
Command apitool is a simple tool to fetch individual OONI measurements.
|
Command apitool is a simple tool to fetch individual OONI measurements. |
|
cmd/getresources
Command getresources downloads the resources
|
Command getresources downloads the resources |
|
cmd/jafar
Jafar is a censorship simulation tool used for testing OONI.
|
Jafar is a censorship simulation tool used for testing OONI. |
|
cmd/jafar/badproxy
Package badproxy implements misbehaving proxies.
|
Package badproxy implements misbehaving proxies. |
|
cmd/jafar/flagx
Package flagx contains extensions for the standard library flag package.
|
Package flagx contains extensions for the standard library flag package. |
|
cmd/jafar/httpproxy
Package httpproxy contains a censoring HTTP proxy.
|
Package httpproxy contains a censoring HTTP proxy. |
|
cmd/jafar/iptables
Package iptables contains code for managing firewall rules.
|
Package iptables contains code for managing firewall rules. |
|
cmd/jafar/resolver
Package resolver contains a censoring DNS resolver.
|
Package resolver contains a censoring DNS resolver. |
|
cmd/jafar/tlsproxy
Package tlsproxy contains a censoring TLS proxy.
|
Package tlsproxy contains a censoring TLS proxy. |
|
cmd/jafar/uncensored
Package uncensored contains code used by Jafar to evade its own censorship efforts by taking alternate routes.
|
Package uncensored contains code used by Jafar to evade its own censorship efforts by taking alternate routes. |
|
cmd/miniooni
Command miniooni is a simple binary for research and QA purposes with a CLI interface similar to MK and OONI Probe v2.x.
|
Command miniooni is a simple binary for research and QA purposes with a CLI interface similar to MK and OONI Probe v2.x. |
|
cmd/oohelper
Command oohelper contains a simple command line client for the Web Connectivity test helper.
|
Command oohelper contains a simple command line client for the Web Connectivity test helper. |
|
cmd/oohelperd
Command oohelperd contains the Web Connectivity test helper.
|
Command oohelperd contains the Web Connectivity test helper. |
|
cmd/oohelperd/internal/websteps
Package websteps implements the websteps test helper.
|
Package websteps implements the websteps test helper. |
|
cmd/printversion
Command printversion prints the current version of this repository.
|
Command printversion prints the current version of this repository. |
|
cmd/ptxclient
Command ptxclient implements a pluggable transports CLI client.
|
Command ptxclient implements a pluggable transports CLI client. |
|
engine
Package engine contains the engine API.
|
Package engine contains the engine API. |
|
engine/experiment/dash
Package dash implements the DASH network experiment.
|
Package dash implements the DASH network experiment. |
|
engine/experiment/dnscheck
Package dnscheck contains the DNS check experiment.
|
Package dnscheck contains the DNS check experiment. |
|
engine/experiment/example
Package example contains a simple example experiment.
|
Package example contains a simple example experiment. |
|
engine/experiment/fbmessenger
Package fbmessenger contains the Facebook Messenger network experiment.
|
Package fbmessenger contains the Facebook Messenger network experiment. |
|
engine/experiment/hhfm
Package hhfm contains the HTTP Header Field Manipulation network experiment.
|
Package hhfm contains the HTTP Header Field Manipulation network experiment. |
|
engine/experiment/hirl
Package hirl contains the HTTP Invalid Request Line network experiment.
|
Package hirl contains the HTTP Invalid Request Line network experiment. |
|
engine/experiment/httphostheader
Package httphostheader contains the HTTP host header network experiment.
|
Package httphostheader contains the HTTP host header network experiment. |
|
engine/experiment/ndt7
Package ndt7 contains the ndt7 network experiment.
|
Package ndt7 contains the ndt7 network experiment. |
|
engine/experiment/psiphon
Package psiphon implements the psiphon network experiment.
|
Package psiphon implements the psiphon network experiment. |
|
engine/experiment/riseupvpn
Package riseupvpn contains the RiseupVPN network experiment.
|
Package riseupvpn contains the RiseupVPN network experiment. |
|
engine/experiment/run
Package run contains code to run other experiments.
|
Package run contains code to run other experiments. |
|
engine/experiment/signal
Package signal contains the Signal network experiment.
|
Package signal contains the Signal network experiment. |
|
engine/experiment/sniblocking
Package sniblocking contains the SNI blocking network experiment.
|
Package sniblocking contains the SNI blocking network experiment. |
|
engine/experiment/stunreachability
Package stunreachability contains the STUN reachability experiment.
|
Package stunreachability contains the STUN reachability experiment. |
|
engine/experiment/telegram
Package telegram contains the Telegram network experiment.
|
Package telegram contains the Telegram network experiment. |
|
engine/experiment/tlstool
Package tlstool contains a TLS tool that we are currently using for running quick and dirty experiments.
|
Package tlstool contains a TLS tool that we are currently using for running quick and dirty experiments. |
|
engine/experiment/tlstool/internal
Package internal contains the implementation of tlstool.
|
Package internal contains the implementation of tlstool. |
|
engine/experiment/tor
Package tor contains the tor experiment.
|
Package tor contains the tor experiment. |
|
engine/experiment/torsf
Package torsf contains the torsf experiment.
|
Package torsf contains the torsf experiment. |
|
engine/experiment/urlgetter
Package urlgetter implements a nettest that fetches a URL.
|
Package urlgetter implements a nettest that fetches a URL. |
|
engine/experiment/webconnectivity
Package webconnectivity implements OONI's Web Connectivity experiment.
|
Package webconnectivity implements OONI's Web Connectivity experiment. |
|
engine/experiment/webconnectivity/internal
Package internal contains internal code.
|
Package internal contains internal code. |
|
engine/experiment/websteps
Package websteps implements the websteps experiment.
|
Package websteps implements the websteps experiment. |
|
engine/experiment/webstepsx
Package webstepsx contains a websteps implementation based on the internal/measurex package.
|
Package webstepsx contains a websteps implementation based on the internal/measurex package. |
|
engine/experiment/whatsapp
Package whatsapp contains the WhatsApp network experiment.
|
Package whatsapp contains the WhatsApp network experiment. |
|
engine/geolocate
Package geolocate implements IP lookup, resolver lookup, and geolocation.
|
Package geolocate implements IP lookup, resolver lookup, and geolocation. |
|
engine/httpheader
Package httpheader contains code to set common HTTP headers.
|
Package httpheader contains code to set common HTTP headers. |
|
engine/httpx
Package httpx contains http extensions.
|
Package httpx contains http extensions. |
|
engine/internal/httpfailure
Package httpfailure groups a bunch of extra HTTP failures.
|
Package httpfailure groups a bunch of extra HTTP failures. |
|
engine/internal/sessionresolver
Package sessionresolver contains the resolver used by the session.
|
Package sessionresolver contains the resolver used by the session. |
|
engine/legacy/assetsdir
Package assetsdir contains code to cleanup the assets dir.
|
Package assetsdir contains code to cleanup the assets dir. |
|
engine/legacy/errorsx
Package errorsx contains error extensions.
|
Package errorsx contains error extensions. |
|
engine/legacy/netx
Package netx contains OONI's net extensions.
|
Package netx contains OONI's net extensions. |
|
engine/legacy/netx/handlers
Package handlers contains default modelx.Handler handlers.
|
Package handlers contains default modelx.Handler handlers. |
|
engine/legacy/netx/modelx
Package modelx contains the data modelx.
|
Package modelx contains the data modelx. |
|
engine/legacy/netx/oldhttptransport
Package oldhttptransport contains HTTP transport extensions.
|
Package oldhttptransport contains HTTP transport extensions. |
|
engine/legacy/netxlogger
Package netxlogger is a logger for netx events.
|
Package netxlogger is a logger for netx events. |
|
engine/legacy/oonidatamodel
Package oonidatamodel contains the OONI data model.
|
Package oonidatamodel contains the OONI data model. |
|
engine/legacy/oonitemplates
Package oonitemplates contains templates for experiments.
|
Package oonitemplates contains templates for experiments. |
|
engine/mockable
Package mockable contains mockable objects
|
Package mockable contains mockable objects |
|
engine/model
Package model defines shared data structures and interfaces.
|
Package model defines shared data structures and interfaces. |
|
engine/netx
Package netx contains code to perform network measurements.
|
Package netx contains code to perform network measurements. |
|
engine/netx/archival
Package archival contains data formats used for archival.
|
Package archival contains data formats used for archival. |
|
engine/netx/dialer
Package dialer allows you to create a net.Dialer-compatible DialContext-enabled dialer with error wrapping, optional logging, optional network-events saving, and optional proxying.
|
Package dialer allows you to create a net.Dialer-compatible DialContext-enabled dialer with error wrapping, optional logging, optional network-events saving, and optional proxying. |
|
engine/netx/httptransport
Package httptransport contains HTTP transport extensions.
|
Package httptransport contains HTTP transport extensions. |
|
engine/netx/tlsdialer
Package tlsdialer contains code to establish TLS connections.
|
Package tlsdialer contains code to establish TLS connections. |
|
engine/probeservices
Package probeservices contains code to contact OONI probe services.
|
Package probeservices contains code to contact OONI probe services. |
|
engine/probeservices/testorchestra
Package testorchestra helps with testing the OONI orchestra API.
|
Package testorchestra helps with testing the OONI orchestra API. |
|
fsx
Package fsx contains io/fs extensions.
|
Package fsx contains io/fs extensions. |
|
humanize
Package humanize is like dustin/go-humanize.
|
Package humanize is like dustin/go-humanize. |
|
kvstore
Package kvstore contains key-value stores.
|
Package kvstore contains key-value stores. |
|
measurex
Package measurex contains measurement extensions.
|
Package measurex contains measurement extensions. |
|
mlablocate
Package mlablocate contains a locate.measurementlab.net client implementing v1 of the locate API.
|
Package mlablocate contains a locate.measurementlab.net client implementing v1 of the locate API. |
|
mlablocatev2
Package mlablocatev2 implements m-lab locate services API v2.
|
Package mlablocatev2 implements m-lab locate services API v2. |
|
multierror
Package multierror contains code to manage multiple errors.
|
Package multierror contains code to manage multiple errors. |
|
netxlite
Package netxlite contains network extensions.
|
Package netxlite contains network extensions. |
|
netxlite/dnsx
Package dnsx contains DNS extension types.
|
Package dnsx contains DNS extension types. |
|
netxlite/filtering
Package filtering allows to implement self-censorship.
|
Package filtering allows to implement self-censorship. |
|
netxlite/mocks
Package mocks contains mocks for netx types.
|
Package mocks contains mocks for netx types. |
|
netxlite/quictesting
Package quictesting contains code useful to test QUIC.
|
Package quictesting contains code useful to test QUIC. |
|
netxlite/quicx
Package quicx contains lucas-clemente/quic-go extensions.
|
Package quicx contains lucas-clemente/quic-go extensions. |
|
ooapi
Package ooapi contains a client for the OONI API.
|
Package ooapi contains a client for the OONI API. |
|
ooapi/apimodel
Package apimodel describes the data types used by OONI's API.
|
Package apimodel describes the data types used by OONI's API. |
|
ooapi/internal/generator
Command generator generates code in the ooapi package.
|
Command generator generates code in the ooapi package. |
|
ooapi/internal/openapi
Package openapi contains data structures for Swagger v2.0.
|
Package openapi contains data structures for Swagger v2.0. |
|
platform
Package platform allows you to obtain the platform name.
|
Package platform allows you to obtain the platform name. |
|
ptx
Package ptx contains code to use pluggable transports.
|
Package ptx contains code to use pluggable transports. |
|
randx
Package randx contains math/rand extensions.
|
Package randx contains math/rand extensions. |
|
runtimex
Package runtimex contains runtime extensions.
|
Package runtimex contains runtime extensions. |
|
shellx
Package shellx runs external commands.
|
Package shellx runs external commands. |
|
stuninput
Package stuninput contains stun targets as well as code to format such targets according to various conventions.
|
Package stuninput contains stun targets as well as code to format such targets according to various conventions. |
|
tunnel
Package tunnel allows to create tunnels to speak with OONI backends and other services.
|
Package tunnel allows to create tunnels to speak with OONI backends and other services. |
|
tutorial/experiment/torsf/chapter01
-=-=- StartHere -=-=- # Chapter I: main.go using the real torsf implementation In this chapter we will write together a `main.go` file that uses the real `torsf` implementation to run the experiment.
|
-=-=- StartHere -=-=- # Chapter I: main.go using the real torsf implementation In this chapter we will write together a `main.go` file that uses the real `torsf` implementation to run the experiment. |
|
tutorial/experiment/torsf/chapter02
-=-=- StartHere -=-=- ## The torsf.go file This file will contain the implementation of the `NewExperimentMeasurer` function.
|
-=-=- StartHere -=-=- ## The torsf.go file This file will contain the implementation of the `NewExperimentMeasurer` function. |
|
tutorial/generator
Command generator generates or re-generates the tutorial chapters.
|
Command generator generates or re-generates the tutorial chapters. |
|
tutorial/measurex/chapter01
-=-=- StartHere -=-=- # Chapter I: using the system resolver In this chapter we explain how to measure DNS resolutions performed using the system resolver.
|
-=-=- StartHere -=-=- # Chapter I: using the system resolver In this chapter we explain how to measure DNS resolutions performed using the system resolver. |
|
tutorial/measurex/chapter02
-=-=- StartHere -=-=- # Chapter II: establishing TCP connections In this chapter we explain how to measure establishing TCP connections.
|
-=-=- StartHere -=-=- # Chapter II: establishing TCP connections In this chapter we explain how to measure establishing TCP connections. |
|
tutorial/measurex/chapter03
-=-=- StartHere -=-=- # Chapter III: using a custom DNS-over-UDP resolver In this chapter we learn how to measure sending DNS queries to a DNS server speaking the DNS-over-UDP protocol.
|
-=-=- StartHere -=-=- # Chapter III: using a custom DNS-over-UDP resolver In this chapter we learn how to measure sending DNS queries to a DNS server speaking the DNS-over-UDP protocol. |
|
tutorial/measurex/chapter04
-=-=- StartHere -=-=- # Chapter IV: TLS handshaking This chapter describes measuring TLS handshakes.
|
-=-=- StartHere -=-=- # Chapter IV: TLS handshaking This chapter describes measuring TLS handshakes. |
|
tutorial/measurex/chapter05
-=-=- StartHere -=-=- # Chapter V: QUIC handshaking This chapter describes measuring QUIC handshakes.
|
-=-=- StartHere -=-=- # Chapter V: QUIC handshaking This chapter describes measuring QUIC handshakes. |
|
tutorial/measurex/chapter06
-=-=- StartHere -=-=- # Chapter VI: Getting a webpage from an HTTP/HTTPS/HTTP3 endpoint.
|
-=-=- StartHere -=-=- # Chapter VI: Getting a webpage from an HTTP/HTTPS/HTTP3 endpoint. |
|
tutorial/measurex/chapter07
-=-=- StartHere -=-=- # Chapter VII: Measuring all the HTTPEndpoints for a domain We are now going to combine DNS resolutions with getting HTTPEndpoints.
|
-=-=- StartHere -=-=- # Chapter VII: Measuring all the HTTPEndpoints for a domain We are now going to combine DNS resolutions with getting HTTPEndpoints. |
|
tutorial/measurex/chapter08
-=-=- StartHere -=-=- # Chapter VII: HTTPSSvc DNS queries The program we see here is _really_ similar to the one we discussed in the previous chapter.
|
-=-=- StartHere -=-=- # Chapter VII: HTTPSSvc DNS queries The program we see here is _really_ similar to the one we discussed in the previous chapter. |
|
tutorial/measurex/chapter09
-=-=- StartHere -=-=- # Chapter IX: Parallel HTTPEndpoint measurements The program we see here is _really_ similar to the one we discussed in the previous chapter.
|
-=-=- StartHere -=-=- # Chapter IX: Parallel HTTPEndpoint measurements The program we see here is _really_ similar to the one we discussed in the previous chapter. |
|
tutorial/measurex/chapter10
-=-=- StartHere -=-=- # Chapter IX: Parallel DNS lookups The program we see here is _really_ similar to the one we discussed in the previous chapter.
|
-=-=- StartHere -=-=- # Chapter IX: Parallel DNS lookups The program we see here is _really_ similar to the one we discussed in the previous chapter. |
|
tutorial/measurex/chapter11
-=-=- StartHere -=-=- # Chapter XI: Measuring a URL This program shows how to measure an HTTP/HTTPS URL.
|
-=-=- StartHere -=-=- # Chapter XI: Measuring a URL This program shows how to measure an HTTP/HTTPS URL. |
|
tutorial/measurex/chapter12
-=-=- StartHere -=-=- # Chapter XII: Following redirections.
|
-=-=- StartHere -=-=- # Chapter XII: Following redirections. |
|
tutorial/measurex/chapter13
-=-=- StartHere -=-=- # Chapter XIII: Rewriting Web Connectivity This chapter contains an exercise.
|
-=-=- StartHere -=-=- # Chapter XIII: Rewriting Web Connectivity This chapter contains an exercise. |
|
tutorial/measurex/chapter14
-=-=- StartHere -=-=- # Chapter XIV: A possible rewrite of Web Connectivity In this chapter we try to solve the exercise laid out in the previous chapter, using `measurex` primitives.
|
-=-=- StartHere -=-=- # Chapter XIV: A possible rewrite of Web Connectivity In this chapter we try to solve the exercise laid out in the previous chapter, using `measurex` primitives. |
|
tutorial/netxlite/chapter01
-=-=- StartHere -=-=- # Chapter I: establishing TCP connections In this chapter we will write together a `main.go` file that uses netxlite to establish a new TCP connection.
|
-=-=- StartHere -=-=- # Chapter I: establishing TCP connections In this chapter we will write together a `main.go` file that uses netxlite to establish a new TCP connection. |
|
tutorial/netxlite/chapter02
-=-=- StartHere -=-=- # Chapter I: TLS handshakes In this chapter we will write together a `main.go` file that uses netxlite to establish a new TCP connection and then performs a TLS handshake using the established connection.
|
-=-=- StartHere -=-=- # Chapter I: TLS handshakes In this chapter we will write together a `main.go` file that uses netxlite to establish a new TCP connection and then performs a TLS handshake using the established connection. |
|
tutorial/netxlite/chapter03
-=-=- StartHere -=-=- # Chapter I: TLS parroting In this chapter we will write together a `main.go` file that uses netxlite to establish a new TCP connection and then performs a TLS handshake using the established connection.
|
-=-=- StartHere -=-=- # Chapter I: TLS parroting In this chapter we will write together a `main.go` file that uses netxlite to establish a new TCP connection and then performs a TLS handshake using the established connection. |
|
tutorial/netxlite/chapter04
-=-=- StartHere -=-=- # Chapter I: Using QUIC In this chapter we will write together a `main.go` file that uses netxlite to establish a new QUIC session with an UDP endpoint.
|
-=-=- StartHere -=-=- # Chapter I: Using QUIC In this chapter we will write together a `main.go` file that uses netxlite to establish a new QUIC session with an UDP endpoint. |
|
tutorial/netxlite/chapter05
-=-=- StartHere -=-=- # Chapter I: Using the "system" DNS resolver In this chapter we will write together a `main.go` file that uses the "system" DNS resolver to lookup domain names.
|
-=-=- StartHere -=-=- # Chapter I: Using the "system" DNS resolver In this chapter we will write together a `main.go` file that uses the "system" DNS resolver to lookup domain names. |
|
tutorial/netxlite/chapter06
-=-=- StartHere -=-=- # Chapter I: Using a custom UDP resolver In this chapter we will write together a `main.go` file that uses a custom UDP DNS resolver to lookup domain names.
|
-=-=- StartHere -=-=- # Chapter I: Using a custom UDP resolver In this chapter we will write together a `main.go` file that uses a custom UDP DNS resolver to lookup domain names. |
|
tutorial/netxlite/chapter07
-=-=- StartHere -=-=- # Chapter I: HTTP GET with TLS conn In this chapter we will write together a `main.go` file that uses netxlite to establish a TLS connection to a remote endpoint and then fetches a webpage from it using GET.
|
-=-=- StartHere -=-=- # Chapter I: HTTP GET with TLS conn In this chapter we will write together a `main.go` file that uses netxlite to establish a TLS connection to a remote endpoint and then fetches a webpage from it using GET. |
|
tutorial/netxlite/chapter08
-=-=- StartHere -=-=- # Chapter I: HTTP GET with QUIC sess In this chapter we will write together a `main.go` file that uses netxlite to establish a QUIC session to a remote endpoint and then fetches a webpage from it using GET.
|
-=-=- StartHere -=-=- # Chapter I: HTTP GET with QUIC sess In this chapter we will write together a `main.go` file that uses netxlite to establish a QUIC session to a remote endpoint and then fetches a webpage from it using GET. |
|
version
Package version contains version information
|
Package version contains version information |
|
pkg
|
|
|
oonimkall
Package oonimkall implements APIs used by OONI mobile apps.
|
Package oonimkall implements APIs used by OONI mobile apps. |
Click to show internal directories.
Click to hide internal directories.