netx

README

Package github.com/ooni/probe-engine/netx

OONI extensions to the net and net/http packages. This code is used by ooni/probe-engine as a low level library to collect network, DNS, and HTTP events occurring during OONI measurements.

This library contains replacements for commonly used standard library interfaces that facilitate seamless network measurements. By using such replacements, as opposed to standard library interfaces, we can:

• save the timing of HTTP events (e.g. received response headers)
• save the timing and result of every Connect, Read, Write, Close operation
• save the timing and result of the TLS handshake (including certificates)

By default, this library uses the system resolver. In addition, it is possible to configure alternative DNS transports and remote servers. We support DNS over UDP, DNS over TCP, DNS over TLS (DoT), and DNS over HTTPS (DoH). When using an alternative transport, we are also able to intercept and save DNS messages, as well as any other interaction with the remote server (e.g., the result of the TLS handshake for DoT and DoH).

This package is a fork of github.com/ooni/netx.

Documentation

Overview

Package netx contains code to perform network measurements.

This library contains replacements for commonly used standard library interfaces that facilitate seamless network measurements. By using such replacements, as opposed to standard library interfaces, we can:

* save the timing of HTTP events (e.g. received response headers) * save the timing and result of every Connect, Read, Write, Close operation * save the timing and result of the TLS handshake (including certificates)

By default, this library uses the system resolver. In addition, it is possible to configure alternative DNS transports and remote servers. We support DNS over UDP, DNS over TCP, DNS over TLS (DoT), and DNS over HTTPS (DoH). When using an alternative transport, we are also able to intercept and save DNS messages, as well as any other interaction with the remote server (e.g., the result of the TLS handshake for DoT and DoH).

We described the design and implementation of the most recent version of this package at <https://github.com/ooni/probe-engine/issues/359>. Such issue also links to a previous design document.

Index

• type Config
• type DNSClient
  • func NewDNSClient(config Config, URL string) (DNSClient, error)
  • func NewDNSClientWithOverrides(config Config, URL, hostOverride, SNIOverride, TLSVersion string) (DNSClient, error)
  • func (c DNSClient) CloseIdleConnections()
• type Dialer
  • func NewDialer(config Config) Dialer
• type HTTPRoundTripper
  • func NewHTTPTransport(config Config) HTTPRoundTripper
• type Logger
• type QUICDialer
  • func NewQUICDialer(config Config) QUICDialer
• type Resolver
  • func NewResolver(config Config) Resolver
• type TLSDialer
  • func NewTLSDialer(config Config) TLSDialer

Types

type Config

type Config struct {
	BaseResolver        Resolver             // default: system resolver
	BogonIsError        bool                 // default: bogon is not error
	ByteCounter         *bytecounter.Counter // default: no explicit byte counting
	CacheResolutions    bool                 // default: no caching
	CertPool            *x509.CertPool       // default: use vendored gocertifi
	ContextByteCounting bool                 // default: no implicit byte counting
	DNSCache            map[string][]string  // default: cache is empty
	DialSaver           *trace.Saver         // default: not saving dials
	Dialer              Dialer               // default: dialer.DNSDialer
	FullResolver        Resolver             // default: base resolver + goodies
	QUICDialer          QUICDialer           // default: quicdialer.DNSDialer
	HTTP3Enabled        bool                 // default: disabled
	HTTPSaver           *trace.Saver         // default: not saving HTTP
	Logger              Logger               // default: no logging
	NoTLSVerify         bool                 // default: perform TLS verify
	ProxyURL            *url.URL             // default: no proxy
	ReadWriteSaver      *trace.Saver         // default: not saving read/write
	ResolveSaver        *trace.Saver         // default: not saving resolves
	TLSConfig           *tls.Config          // default: attempt using h2
	TLSDialer           TLSDialer            // default: dialer.TLSDialer
	TLSSaver            *trace.Saver         // default: not saving TLS
}

Config contains configuration for creating a new transport. When any field of Config is nil/empty, we will use a suitable default.

We use different savers for different kind of events such that the user of this library can choose what to save.

type DNSClient

type DNSClient struct {
	Resolver
	// contains filtered or unexported fields
}

DNSClient is a DNS client. It wraps a Resolver and it possibly also wraps an HTTP client, but only when we're using DoH.

func NewDNSClient

func NewDNSClient(config Config, URL string) (DNSClient, error)

NewDNSClient creates a new DNS client. The config argument is used to create the underlying Dialer and/or HTTP transport, if needed. The URL argument describes the kind of client that we want to make:

- if the URL is `doh://powerdns`, `doh://google` or `doh://cloudflare` or the URL starts with `https://`, then we create a DoH client.

- if the URL is “ or `system:///`, then we create a system client, i.e. a client using the system resolver.

- if the URL starts with `udp://`, then we create a client using a resolver that uses the specified UDP endpoint.

We return error if the URL does not parse or the URL scheme does not fall into one of the cases described above.

If config.ResolveSaver is not nil and we're creating an underlying resolver where this is possible, we will also save events.

func NewDNSClientWithOverrides

func NewDNSClientWithOverrides(config Config, URL, hostOverride, SNIOverride,
	TLSVersion string) (DNSClient, error)

NewDNSClientWithOverrides creates a new DNS client, similar to NewDNSClient, with the option to override the default Hostname and SNI.

func (DNSClient) CloseIdleConnections

func (c DNSClient) CloseIdleConnections()

CloseIdleConnections closes idle connections, if any.

type Dialer

type Dialer interface {
	DialContext(ctx context.Context, network, address string) (net.Conn, error)
}

Dialer is the definition of dialer assumed by this package.

func NewDialer

func NewDialer(config Config) Dialer

NewDialer creates a new Dialer from the specified config

type HTTPRoundTripper

type HTTPRoundTripper interface {
	RoundTrip(req *http.Request) (*http.Response, error)
	CloseIdleConnections()
}

HTTPRoundTripper is the definition of http.HTTPRoundTripper used by this package.

func NewHTTPTransport

func NewHTTPTransport(config Config) HTTPRoundTripper

NewHTTPTransport creates a new HTTPRoundTripper. You can further extend the returned HTTPRoundTripper before wrapping it into an http.Client.

type Logger

type Logger interface {
	Debugf(format string, v ...interface{})
	Debug(message string)
}

Logger is the logger assumed by this package

type QUICDialer

type QUICDialer interface {
	DialContext(ctx context.Context, network, addr string, tlsCfg *tls.Config, cfg *quic.Config) (quic.EarlySession, error)
}

QUICDialer is the definition of a dialer for QUIC assumed by this package.

func NewQUICDialer

func NewQUICDialer(config Config) QUICDialer

NewQUICDialer creates a new DNS Dialer for QUIC, with the resolver from the specified config

type Resolver

type Resolver interface {
	LookupHost(ctx context.Context, hostname string) (addrs []string, err error)
	Network() string
	Address() string
}

Resolver is the interface we expect from a resolver

func NewResolver

func NewResolver(config Config) Resolver

NewResolver creates a new resolver from the specified config

type TLSDialer

type TLSDialer interface {
	DialTLSContext(ctx context.Context, network, address string) (net.Conn, error)
}

TLSDialer is the definition of a TLS dialer assumed by this package.

func NewTLSDialer

func NewTLSDialer(config Config) TLSDialer

NewTLSDialer creates a new TLSDialer from the specified config