netx

README

Package github.com/ooni/probe-engine/netx

OONI extensions to the net and net/http packages. This code is used by ooni/probe-engine as a low level library to collect network, DNS, and HTTP events occurring during OONI measurements.

This library contains replacements for commonly used standard library interfaces that facilitate seamless network measurements. By using such replacements, as opposed to standard library interfaces, we can:

• save the timing of HTTP events (e.g. received response headers)
• save the timing and result of every Connect, Read, Write, Close operation
• save the timing and result of the TLS handshake (including certificates)

By default, this library uses the system resolver. In addition, it is possible to configure alternative DNS transports and remote servers. We support DNS over UDP, DNS over TCP, DNS over TLS (DoT), and DNS over HTTPS (DoH). When using an alternative transport, we are also able to intercept and save DNS messages, as well as any other interaction with the remote server (e.g., the result of the TLS handshake for DoT and DoH).

This package is a fork of github.com/ooni/netx.

Documentation

Overview

Package netx contains code to perform network measurements.

This library contains replacements for commonly used standard library interfaces that facilitate seamless network measurements. By using such replacements, as opposed to standard library interfaces, we can:

* save the timing of HTTP events (e.g. received response headers) * save the timing and result of every Connect, Read, Write, Close operation * save the timing and result of the TLS handshake (including certificates)

By default, this library uses the system resolver. In addition, it is possible to configure alternative DNS transports and remote servers. We support DNS over UDP, DNS over TCP, DNS over TLS (DoT), and DNS over HTTPS (DoH). When using an alternative transport, we are also able to intercept and save DNS messages, as well as any other interaction with the remote server (e.g., the result of the TLS handshake for DoT and DoH).

We described the design and implementation of the most recent version of this package at <https://github.com/ooni/probe-engine/issues/359>. Such issue also links to a previous design document.

Index

• func NewDNSClient(config Config, URL string) (model.Resolver, error)
• func NewDNSClientWithOverrides(config Config, URL, hostOverride, SNIOverride, TLSVersion string) (model.Resolver, error)
• func NewDialer(config Config) model.Dialer
• func NewHTTPTransport(config Config) model.HTTPTransport
• func NewQUICDialer(config Config) model.QUICDialer
• func NewResolver(config Config) model.Resolver
• func NewTLSDialer(config Config) model.TLSDialer
• type Config

Functions

func NewDNSClient

func NewDNSClient(config Config, URL string) (model.Resolver, error)

NewDNSClient creates a new DNS client. The config argument is used to create the underlying Dialer and/or HTTP transport, if needed. The URL argument describes the kind of client that we want to make:

- if the URL is `doh://powerdns`, `doh://google` or `doh://cloudflare` or the URL starts with `https://`, then we create a DoH client.

- if the URL is “ or `system:///`, then we create a system client, i.e. a client using the system resolver.

- if the URL starts with `udp://`, then we create a client using a resolver that uses the specified UDP endpoint.

We return error if the URL does not parse or the URL scheme does not fall into one of the cases described above.

If config.ResolveSaver is not nil and we're creating an underlying resolver where this is possible, we will also save events.

func NewDNSClientWithOverrides

func NewDNSClientWithOverrides(config Config, URL, hostOverride, SNIOverride,
	TLSVersion string) (model.Resolver, error)

NewDNSClientWithOverrides creates a new DNS client, similar to NewDNSClient, with the option to override the default Hostname and SNI.

func NewDialer

func NewDialer(config Config) model.Dialer

NewDialer creates a new Dialer from the specified config

func NewHTTPTransport

func NewHTTPTransport(config Config) model.HTTPTransport

NewHTTPTransport creates a new HTTPRoundTripper. You can further extend the returned HTTPRoundTripper before wrapping it into an http.Client.

func NewQUICDialer

func NewQUICDialer(config Config) model.QUICDialer

NewQUICDialer creates a new DNS Dialer for QUIC, with the resolver from the specified config

func NewResolver

func NewResolver(config Config) model.Resolver

NewResolver creates a new resolver from the specified config

func NewTLSDialer

func NewTLSDialer(config Config) model.TLSDialer

NewTLSDialer creates a new TLSDialer from the specified config

Types

type Config

type Config struct {
	BaseResolver        model.Resolver       // default: system resolver
	BogonIsError        bool                 // default: bogon is not error
	ByteCounter         *bytecounter.Counter // default: no explicit byte counting
	CacheResolutions    bool                 // default: no caching
	CertPool            *x509.CertPool       // default: use vendored gocertifi
	ContextByteCounting bool                 // default: no implicit byte counting
	DNSCache            map[string][]string  // default: cache is empty
	DialSaver           *trace.Saver         // default: not saving dials
	Dialer              model.Dialer         // default: dialer.DNSDialer
	FullResolver        model.Resolver       // default: base resolver + goodies
	QUICDialer          model.QUICDialer     // default: quicdialer.DNSDialer
	HTTP3Enabled        bool                 // default: disabled
	HTTPSaver           *trace.Saver         // default: not saving HTTP
	Logger              model.DebugLogger    // default: no logging
	NoTLSVerify         bool                 // default: perform TLS verify
	ProxyURL            *url.URL             // default: no proxy
	ReadWriteSaver      *trace.Saver         // default: not saving read/write
	ResolveSaver        *trace.Saver         // default: not saving resolves
	TLSConfig           *tls.Config          // default: attempt using h2
	TLSDialer           model.TLSDialer      // default: dialer.TLSDialer
	TLSSaver            *trace.Saver         // default: not saving TLS
}

Config contains configuration for creating a new transport. When any field of Config is nil/empty, we will use a suitable default.

We use different savers for different kind of events such that the user of this library can choose what to save.