measurex

package
v3.15.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 5, 2022 License: GPL-3.0 Imports: 24 Imported by: 0

Documentation

Overview

Package measurex contains measurement extensions.

Index

Constants

View Source 
const (
	// NetworkTCP identifies endpoints using TCP.
	NetworkTCP = EndpointNetwork("tcp")

	// NetworkQUIC identifies endpoints using QUIC.
	NetworkQUIC = EndpointNetwork("quic")
)
View Source 
const DefaultDNSLookupTimeout = 4 * time.Second

DefaultDNSLookupTimeout is the default DNS lookup timeout.

View Source 
const DefaultHTTPMaxBodySnapshotSize = 1 << 11

DefaultHTTPMaxBodySnapshotSize is the default size used when saving HTTP body snapshots. We only save a small snapshot of the body to keep measurements lean, since we're mostly interested in TLS interference nowadays and much less in full bodies.

View Source 
const DefaultHTTPRoundTripTimeout = 15 * time.Second

DefaultHTTPRoundTripTimeout is the default HTTP round-trip timeout.

View Source 
const DefaultQUICHandshakeTimeout = 10 * time.Second

DefaultQUICHandshakeTimeout is the default QUIC handshake timeout.

View Source 
const DefaultTCPConnectTimeout = 15 * time.Second

DefaultTCPConnectTimeout is the default TCP connect timeout.

View Source 
const DefaultTLSHandshakeTimeout = 10 * time.Second

DefaultTLSHandshakeTimeout is the default TLS handshake timeout.

Variables

View Source 
var (
	// ResolverSystem is the system resolver (i.e., getaddrinfo)
	ResolverSystem = ResolverNetwork("system")

	// ResolverUDP is a resolver using DNS-over-UDP
	ResolverUDP = ResolverNetwork("udp")

	// ResolverForeign is a resolver that is not managed by
	// this package. We can wrap it, but we don't be able to
	// observe any event but Lookup{Host,HTTPSvc}
	ResolverForeign = ResolverNetwork("foreign")
)
View Source 
var (
	// tcp.connect
	OddityTCPConnectTimeout         = Oddity("tcp.connect.timeout")
	OddityTCPConnectRefused         = Oddity("tcp.connect.refused")
	OddityTCPConnectHostUnreachable = Oddity("tcp.connect.host_unreachable")
	OddityTCPConnectOher            = Oddity("tcp.connect.other")

	// tls.handshake
	OddityTLSHandshakeTimeout          = Oddity("tls.handshake.timeout")
	OddityTLSHandshakeReset            = Oddity("tls.handshake.reset")
	OddityTLSHandshakeOther            = Oddity("tls.handshake.other")
	OddityTLSHandshakeUnexpectedEOF    = Oddity("tls.handshake.unexpected_eof")
	OddityTLSHandshakeInvalidHostname  = Oddity("tls.handshake.invalid_hostname")
	OddityTLSHandshakeUnknownAuthority = Oddity("tls.handshake.unknown_authority")

	// quic.handshake
	OddityQUICHandshakeTimeout         = Oddity("quic.handshake.timeout")
	OddityQUICHandshakeHostUnreachable = Oddity("quic.handshake.host_unreachable")
	OddityQUICHandshakeOther           = Oddity("quic.handshake.other")

	// dns.lookup
	OddityDNSLookupNXDOMAIN = Oddity("dns.lookup.nxdomain")
	OddityDNSLookupTimeout  = Oddity("dns.lookup.timeout")
	OddityDNSLookupRefused  = Oddity("dns.lookup.refused")
	OddityDNSLookupBogon    = Oddity("dns.lookup.bogon")
	OddityDNSLookupOther    = Oddity("dns.lookup.other")

	// http.status
	OddityStatus403   = Oddity("http.status.403")
	OddityStatus404   = Oddity("http.status.404")
	OddityStatus503   = Oddity("http.status.503")
	OddityStatusOther = Oddity("http.status.other")
)

This enumeration lists all known oddities.

View Source 
var ErrCannotDeterminePortFromURL = errors.New("cannot determine port from URL")

ErrCannotDeterminePortFromURL indicates that we could not determine the correct port from the URL authority and scheme.

View Source 
var ErrHTTPTooManyRedirects = errors.New("stopped after 10 redirects")

ErrHTTPTooManyRedirects is the unexported error that the standard library would return when hitting too many redirects.

View Source 
var (

	// ErrUnknownHTTPEndpointNetwork means that the given endpoint's
	// network is of a type that we don't know how to handle.
	ErrUnknownHTTPEndpointNetwork = errors.New("unknown HTTPEndpoint.Network")
)

Functions

func ALPNForHTTPEndpoint 

func ALPNForHTTPEndpoint(network EndpointNetwork) []string

ALPNForHTTPEndpoint selects the correct ALPN for an HTTP endpoint given the network. On failure, we return a nil list.

func NewCookieJar 

func NewCookieJar() http.CookieJar

NewCookieJar is a convenience factory for creating an http.CookieJar that is aware of the effective TLS / public suffix list. This means that the jar won't allow a domain to set cookies for another unrelated domain (in the public-suffix-list sense).

func NewFailure 

func NewFailure(err error) *string

NewFailure creates a serializable failure from an error. We cannot round trip an error using JSON, so we serialize to this intermediate format that is a sort of Optional<string>.

func NewHTTPClientWithRedirects 

func NewHTTPClientWithRedirects(
	db WritableDB, jar http.CookieJar, txp model.HTTPTransport) model.HTTPClient

NewHTTPClientWithRedirects creates a new HTTPClient instance that automatically perform redirects.

func NewHTTPClientWithoutRedirects 

func NewHTTPClientWithoutRedirects(
	db WritableDB, jar http.CookieJar, txp model.HTTPTransport) model.HTTPClient

NewHTTPClient creates a new HTTPClient instance that does not automatically perform redirects.

func NewHTTPGetRequest 

func NewHTTPGetRequest(ctx context.Context, URL string) (*http.Request, error)

NewHTTPGetRequest is a convenience factory for creating a new http.Request using the GET method and the given URL.

func NewHTTPRequestHeaderForMeasuring 

func NewHTTPRequestHeaderForMeasuring() http.Header

NewHTTPRequestHeaderForMeasuring returns an http.Header where the headers are the ones we use for measuring.

func NewHTTPRequestWithContext 

func NewHTTPRequestWithContext(ctx context.Context,
	method, URL string, body io.Reader) (*http.Request, error)

NewHTTPRequestWithContext is a convenience factory for creating a new HTTP request with the typical headers we use when performing measurements already set inside of req.Header.

func PortFromURL 

func PortFromURL(URL *url.URL) (string, error)

PortFromURL returns the port determined from the URL or an error.

func WrapDialer 

func WrapDialer(begin time.Time, db WritableDB, dialer model.Dialer) model.Dialer

WrapDialer wraps a dialer.

func WrapResolver 

func WrapResolver(begin time.Time, db WritableDB, r model.Resolver) model.Resolver

WrapResolver wraps a resolver.

Types

type ArchivalBinaryData 

type ArchivalBinaryData struct {
	Data   []byte `json:"data"`
	Format string `json:"format"`
}

ArchivalBinaryData is the archival format for binary data.

func NewArchivalBinaryData 

func NewArchivalBinaryData(data []byte) (out *ArchivalBinaryData)

NewArchivalBinaryData builds a new ArchivalBinaryData from an array of bytes. If the array is nil, we return nil.

func NewArchivalTLSCerts 

func NewArchivalTLSCerts(in [][]byte) (out []*ArchivalBinaryData)

NewArchivalTLSCertList builds a new []ArchivalBinaryData from a list of raw x509 certificates data.

type ArchivalDNSLookupAnswer 

type ArchivalDNSLookupAnswer struct {
	// JSON names compatible with df-002-dnst's spec
	Type string `json:"answer_type"`
	IPv4 string `json:"ipv4,omitempty"`
	IPv6 string `json:"ivp6,omitempty"`

	// Names not part of the spec.
	ALPN string `json:"alpn,omitempty"`
}

ArchivalDNSLookupAnswer is the archival format of a DNS lookup answer according to df-002-dnst.

func NewArchivalDNSLookupAnswers 

func NewArchivalDNSLookupAnswers(in *DNSLookupEvent) (out []ArchivalDNSLookupAnswer)

NewArchivalDNSLookupAnswers creates a list of ArchivalDNSLookupAnswer.

type ArchivalDNSLookupEvent 

type ArchivalDNSLookupEvent struct {
	// fields inside df-002-dnst
	Answers   []ArchivalDNSLookupAnswer `json:"answers"`
	Network   string                    `json:"engine"`
	Failure   *string                   `json:"failure"`
	Domain    string                    `json:"hostname"`
	QueryType string                    `json:"query_type"`
	Address   string                    `json:"resolver_address"`
	Finished  float64                   `json:"t"`

	// Names not part of the spec.
	Started float64 `json:"started"`
	Oddity  Oddity  `json:"oddity"`
}

ArchivalDNSLookupEvent is the archival data format of a DNS lookup according to df-002-dnst.

func NewArchivalDNSLookupEvent 

func NewArchivalDNSLookupEvent(in *DNSLookupEvent) *ArchivalDNSLookupEvent

NewArchivalDNSLookupEvent converts a DNSLookupEvent to its archival representation.

func NewArchivalDNSLookupEventList 

func NewArchivalDNSLookupEventList(in []*DNSLookupEvent) (out []*ArchivalDNSLookupEvent)

NewArchivalDNSLookupEventList converts a list of DNSLookupEvent to a list of ArchivalDNSLookupEvent.

type ArchivalDNSMeasurement 

type ArchivalDNSMeasurement struct {
	Domain string `json:"domain"`
	*ArchivalMeasurement
}

ArchivalDNSMeasurement is the archival representation of DNSMeasurement.

func NewArchivalDNSMeasurement 

func NewArchivalDNSMeasurement(in *DNSMeasurement) *ArchivalDNSMeasurement

NewArchivalDNSMeasurement converts a DNSMeasurement to an ArchivalDNSMeasurement.

func NewArchivalDNSMeasurementList 

func NewArchivalDNSMeasurementList(in []*DNSMeasurement) (out []*ArchivalDNSMeasurement)

NewArchivalDNSMeasurementList converts a list of DNSMeasurement to a list of ArchivalDNSMeasurement.

type ArchivalDNSRoundTripEvent 

type ArchivalDNSRoundTripEvent struct {
	Network  string              `json:"engine"`
	Address  string              `json:"resolver_address"`
	Query    *ArchivalBinaryData `json:"raw_query"`
	Started  float64             `json:"started"`
	Finished float64             `json:"t"`
	Failure  *string             `json:"failure"`
	Reply    *ArchivalBinaryData `json:"raw_reply"`
}

ArchivalDNSRoundTripEvent is the OONI data format representation of a DNS round trip, which is currently not specified.

We are trying to use names compatible with the names currently used by other specifications we currently use.

func NewArchivalDNSRoundTripEvent 

func NewArchivalDNSRoundTripEvent(in *DNSRoundTripEvent) *ArchivalDNSRoundTripEvent

NewArchivalDNSRoundTripEvent converts a DNSRoundTripEvent into is archival format.

func NewArchivalDNSRoundTripEventList 

func NewArchivalDNSRoundTripEventList(in []*DNSRoundTripEvent) (out []*ArchivalDNSRoundTripEvent)

NewArchivalDNSRoundTripEventList converts a DNSRoundTripEvent list to the corresponding archival format.

type ArchivalEndpointMeasurement 

type ArchivalEndpointMeasurement struct {
	// Network is the network of this endpoint.
	Network EndpointNetwork `json:"network"`

	// Address is the address of this endpoint.
	Address string `json:"address"`

	// An EndpointMeasurement is a Measurement.
	*ArchivalMeasurement
}

ArchivalEndpointMeasurement is the archival representation of EndpointMeasurement.

func NewArchivalEndpointMeasurement 

func NewArchivalEndpointMeasurement(in *EndpointMeasurement) *ArchivalEndpointMeasurement

NewArchivalEndpointMeasurement converts an EndpointMeasurement to the corresponding archival data format.

type ArchivalHTTPEndpointMeasurement 

type ArchivalHTTPEndpointMeasurement struct {
	URL     string          `json:"url"`
	Network EndpointNetwork `json:"network"`
	Address string          `json:"address"`
	*ArchivalMeasurement
}

ArchivalHTTPEndpointMeasurement is the archival representation of an HTTPEndpointMeasurement.

func NewArchivalHTTPEndpointMeasurement 

func NewArchivalHTTPEndpointMeasurement(in *HTTPEndpointMeasurement) *ArchivalHTTPEndpointMeasurement

NewArchivalHTTPEndpointMeasurement converts an HTTPEndpointMeasurement to an ArchivalHTTPEndpointMeasurement.

func NewArchivalHTTPEndpointMeasurementList 

func NewArchivalHTTPEndpointMeasurementList(in []*HTTPEndpointMeasurement) (out []*ArchivalHTTPEndpointMeasurement)

NewArchivalHTTPEndpointMeasurementList converts a list of HTTPEndpointMeasurement to a list of ArchivalHTTPEndpointMeasurement.

type ArchivalHTTPRequest 

type ArchivalHTTPRequest struct {
	Method  string          `json:"method"`
	URL     string          `json:"url"`
	Headers ArchivalHeaders `json:"headers"`
}

ArchivalHTTPRequest is the archival format of an HTTP request according to df-001-http.md.

type ArchivalHTTPResponse 

type ArchivalHTTPResponse struct {
	// Names consistent with df-001-http.md
	Code            int64               `json:"code"`
	Headers         ArchivalHeaders     `json:"headers"`
	Body            *ArchivalBinaryData `json:"body"`
	BodyIsTruncated bool                `json:"body_is_truncated"`

	// Fields not part of the spec
	BodyLength int64 `json:"x_body_length"`
	BodyIsUTF8 bool  `json:"x_body_is_utf8"`
}

ArchivalHTTPResponse is the archival format of an HTTP response according to df-001-http.md.

type ArchivalHTTPRoundTripEvent 

type ArchivalHTTPRoundTripEvent struct {
	// JSON names following the df-001-httpt data format.
	Failure  *string       `json:"failure"`
	Request  *HTTPRequest  `json:"request"`
	Response *HTTPResponse `json:"response"`
	Finished float64       `json:"t"`
	Started  float64       `json:"started"`

	// Names not in the specification
	Oddity Oddity `json:"oddity"`
}

ArchivalHTTPRoundTripEvent is the archival format of an HTTP response according to df-001-http.md.

func NewArchivalHTTPRoundTripEvent 

func NewArchivalHTTPRoundTripEvent(in *HTTPRoundTripEvent) *ArchivalHTTPRoundTripEvent

NewArchivalHTTPRoundTripEvent converts an HTTPRoundTrip to its archival format.

func NewArchivalHTTPRoundTripEventList 

func NewArchivalHTTPRoundTripEventList(in []*HTTPRoundTripEvent) (out []*ArchivalHTTPRoundTripEvent)

NewArchivalHTTPRoundTripEventList converts a list of HTTPRoundTripEvent to a list of ArchivalRoundTripEvent.

type ArchivalHeaders 

type ArchivalHeaders map[string]string

ArchivalHeaders is a list of HTTP headers.

func NewArchivalHeaders 

func NewArchivalHeaders(in http.Header) (out ArchivalHeaders)

NewArchivalHeaders builds a new HeadersList from http.Header.

func (ArchivalHeaders) Get 

func (headers ArchivalHeaders) Get(key string) string

Get searches for the first header with the named key and returns it. If not found, returns an empty string.

type ArchivalMeasurement 

type ArchivalMeasurement struct {
	NetworkEvents  []*ArchivalNetworkEvent          `json:"network_events,omitempty"`
	DNSEvents      []*ArchivalDNSRoundTripEvent     `json:"dns_events,omitempty"`
	Queries        []*ArchivalDNSLookupEvent        `json:"queries,omitempty"`
	TCPConnect     []*ArchivalTCPConnect            `json:"tcp_connect,omitempty"`
	TLSHandshakes  []*ArchivalQUICTLSHandshakeEvent `json:"tls_handshakes,omitempty"`
	QUICHandshakes []*ArchivalQUICTLSHandshakeEvent `json:"quic_handshakes,omitempty"`
	Requests       []*ArchivalHTTPRoundTripEvent    `json:"requests,omitempty"`
}

ArchivalMeasurement is the archival representation of a Measurement.

func NewArchivalMeasurement 

func NewArchivalMeasurement(in *Measurement) *ArchivalMeasurement

NewArchivalMeasurement converts a Measurement to ArchivalMeasurement.

type ArchivalNetworkEvent 

type ArchivalNetworkEvent struct {
	// JSON names compatible with df-008-netevents
	RemoteAddr string  `json:"address"`
	Failure    *string `json:"failure"`
	Count      int     `json:"num_bytes,omitempty"`
	Operation  string  `json:"operation"`
	Network    string  `json:"proto"`
	Finished   float64 `json:"t"`
	Started    float64 `json:"started"`

	// Names that are not part of the spec.
	Oddity Oddity `json:"oddity"`
}

ArchivalNetworkEvent is the OONI data format representation of a network event according to df-008-netevents.

func NewArchivalNetworkEvent 

func NewArchivalNetworkEvent(in *NetworkEvent) *ArchivalNetworkEvent

NewArchivalNetworkEvent converts a network event to its archival format.

func NewArchivalNetworkEventList 

func NewArchivalNetworkEventList(in []*NetworkEvent) (out []*ArchivalNetworkEvent)

NewArchivalNetworkEventList converts a list of NetworkEvent to a list of ArchivalNetworkEvent.

type ArchivalQUICTLSHandshakeEvent 

type ArchivalQUICTLSHandshakeEvent struct {
	// JSON names compatible with df-006-tlshandshake
	CipherSuite     string                `json:"cipher_suite"`
	Failure         *string               `json:"failure"`
	NegotiatedProto string                `json:"negotiated_proto"`
	TLSVersion      string                `json:"tls_version"`
	PeerCerts       []*ArchivalBinaryData `json:"peer_certificates"`
	Finished        float64               `json:"t"`

	// JSON names that are consistent with the
	// spirit of the spec but are not in it
	RemoteAddr string   `json:"address"`
	SNI        string   `json:"server_name"` // used in prod
	ALPN       []string `json:"alpn"`
	SkipVerify bool     `json:"no_tls_verify"` // used in prod
	Oddity     Oddity   `json:"oddity"`
	Network    string   `json:"proto"`
	Started    float64  `json:"started"`
}

ArchivalQUICTLSHandshakeEvent is the archival data format for a QUIC or TLS handshake event according to df-006-tlshandshake.

func NewArchivalQUICTLSHandshakeEvent 

func NewArchivalQUICTLSHandshakeEvent(in *QUICTLSHandshakeEvent) *ArchivalQUICTLSHandshakeEvent

NewArchivalQUICTLSHandshakeEvent converts a QUICTLSHandshakeEvent to its archival data format.

func NewArchivalQUICTLSHandshakeEventList 

func NewArchivalQUICTLSHandshakeEventList(in []*QUICTLSHandshakeEvent) (out []*ArchivalQUICTLSHandshakeEvent)

NewArchivalQUICTLSHandshakeEventList converts a list of QUICTLSHandshakeEvent to a list of ArchivalQUICTLSHandshakeEvent.

type ArchivalTCPConnect 

type ArchivalTCPConnect struct {
	// Names part of the spec.
	IP       string                    `json:"ip"`
	Port     int64                     `json:"port"`
	Finished float64                   `json:"t"`
	Status   *ArchivalTCPConnectStatus `json:"status"`

	// Names not part of the spec.
	Started float64 `json:"started"`
	Oddity  Oddity  `json:"oddity"`
}

ArchivalTCPConnect is the archival form of TCP connect events in compliance with df-005-tcpconnect.

func NewArchivalTCPConnect 

func NewArchivalTCPConnect(in *NetworkEvent) *ArchivalTCPConnect

NewArchivalTCPConnect converts a NetworkEvent to an ArchivalTCPConnect.

func NewArchivalTCPConnectList 

func NewArchivalTCPConnectList(in []*NetworkEvent) (out []*ArchivalTCPConnect)

NewArchivalTCPConnectList converts a list of NetworkEvent to a list of ArchivalTCPConnect. In doing that, the code only considers "connect" events using the TCP protocol.

type ArchivalTCPConnectStatus 

type ArchivalTCPConnectStatus struct {
	Blocked bool    `json:"blocked"`
	Failure *string `json:"failure"`
	Success bool    `json:"success"`
}

ArchivalTCPConnectStatus contains the status of a TCP connect.

type ArchivalTHMeasurement 

type ArchivalTHMeasurement struct {
	DNS       []*ArchivalDNSMeasurement          `json:"dns"`
	Endpoints []*ArchivalHTTPEndpointMeasurement `json:"endpoints"`
}

ArchivalTHMeasurement is the archival representation of THMeasurement.

func NewArchivalTHMeasurement 

func NewArchivalTHMeasurement(in *THMeasurement) (out *ArchivalTHMeasurement)

NewArchivalTHMeasurement creates the archival representation of THMeasurement.

type ArchivalURLMeasurement 

type ArchivalURLMeasurement struct {
	URL          string                             `json:"url"`
	DNS          []*ArchivalDNSMeasurement          `json:"dns"`
	Endpoints    []*ArchivalHTTPEndpointMeasurement `json:"endpoints"`
	TH           *ArchivalTHMeasurement             `json:"th"`
	TotalRuntime time.Duration                      `json:"x_total_runtime"`
	DNSRuntime   time.Duration                      `json:"x_dns_runtime"`
	THRuntime    time.Duration                      `json:"x_th_runtime"`
	EpntsRuntime time.Duration                      `json:"x_epnts_runtime"`
}

ArchivalURLMeasurement is the archival representation of URLMeasurement

func NewArchivalURLMeasurement 

func NewArchivalURLMeasurement(in *URLMeasurement) *ArchivalURLMeasurement

NewArchivalURLMeasurement creates the archival representation of an URLMeasurement data structure.

type Conn 

type Conn = net.Conn

Conn is a network connection.

type DNSLookupEvent 

type DNSLookupEvent struct {
	Network   string
	Failure   *string
	Domain    string
	QueryType string
	Address   string
	Finished  float64
	Started   float64
	Oddity    Oddity
	A         []string
	AAAA      []string
	ALPN      []string
}

DNSLookupEvent contains the results of a DNS lookup.

func (*DNSLookupEvent) Addrs 

func (ev *DNSLookupEvent) Addrs() (out []string)

Addrs returns all the IPv4/IPv6 addresses

func (*DNSLookupEvent) SupportsHTTP3 

func (ev *DNSLookupEvent) SupportsHTTP3() bool

SupportsHTTP3 returns true if this query is for HTTPS and the answer contains an ALPN for "h3"

type DNSMeasurement 

type DNSMeasurement struct {
	// Domain is the domain this measurement refers to.
	Domain string

	// A DNSMeasurement is a Measurement.
	*Measurement
}

DNSMeasurement is a DNS measurement.

type DNSRoundTripEvent 

type DNSRoundTripEvent struct {
	Network  string
	Address  string
	Query    []byte
	Started  float64
	Finished float64
	Failure  *string
	Reply    []byte
}

DNSRoundTripEvent contains the result of a DNS round trip.

type EasyTLSConfig 

type EasyTLSConfig struct {
	// contains filtered or unexported fields
}

EasyTLSConfig helps you to generate a *tls.Config.

func NewEasyTLSConfig 

func NewEasyTLSConfig() *EasyTLSConfig

NewEasyTLSConfig creates a new EasyTLSConfig instance.

func NewEasyTLSConfigWithServerName 

func NewEasyTLSConfigWithServerName(serverName string) *EasyTLSConfig

NewEasyTLSConfigWithServerName creates a new EasyTLSConfig with an already configured value for ServerName.

func (*EasyTLSConfig) InsecureSkipVerify 

func (easy *EasyTLSConfig) InsecureSkipVerify(v bool) *EasyTLSConfig

InsecureSkipVerify disables TLS verification.

func (*EasyTLSConfig) RootCAs 

func (easy *EasyTLSConfig) RootCAs(v *x509.CertPool) *EasyTLSConfig

RootCAs allows the set the CA pool.

func (*EasyTLSConfig) ServerName 

func (easy *EasyTLSConfig) ServerName(v string) *EasyTLSConfig

ServerName sets the SNI value.

type Endpoint 

type Endpoint struct {
	// Network is the network (e.g., "tcp", "quic")
	Network EndpointNetwork

	// Address is the endpoint address (e.g., "8.8.8.8:443")
	Address string
}

Endpoint is an endpoint for a domain.

func AllEndpointsForDomain 

func AllEndpointsForDomain(domain, port string, meas ...*DNSMeasurement) ([]*Endpoint, error)

AllEndpointsForDomain gathers all the endpoints for a given domain from a list of DNSMeasurements, removes duplicates and returns the result.

func AllEndpointsForURL 

func AllEndpointsForURL(URL *url.URL, meas ...*DNSMeasurement) ([]*Endpoint, error)

AllEndpointsForURL is like AllHTTPEndpointsForURL but return simple Endpoints rather than HTTPEndpoints.

func HTTPEndpointsToEndpoints 

func HTTPEndpointsToEndpoints(in []*HTTPEndpoint) (out []*Endpoint)

HTTPEndpointsToEndpoints convers HTTPEndpoints to Endpoints

func (*Endpoint) String 

func (e *Endpoint) String() string

String converts an endpoint to a string (e.g., "8.8.8.8:443/tcp")

type EndpointMeasurement 

type EndpointMeasurement struct {
	// Network is the network of this endpoint.
	Network EndpointNetwork

	// Address is the address of this endpoint.
	Address string

	// An EndpointMeasurement is a Measurement.
	*Measurement
}

EndpointMeasurement is an endpoint measurement.

type EndpointNetwork 

type EndpointNetwork string

EndpointNetwork is the network of an endpoint.

type HTTPEndpoint 

type HTTPEndpoint struct {
	// Domain is the endpoint domain (e.g., "dns.google").
	Domain string

	// Network is the network (e.g., "tcp" or "quic").
	Network EndpointNetwork

	// Address is the endpoint address (e.g., "8.8.8.8:443").
	Address string

	// SNI is the SNI to use (only used with URL.scheme == "https").
	SNI string

	// ALPN is the ALPN to use (only used with URL.scheme == "https").
	ALPN []string

	// URL is the endpoint URL.
	URL *url.URL

	// Header contains request headers.
	Header http.Header
}

HTTPEndpoint is an HTTP/HTTPS/HTTP3 endpoint.

func AllHTTPEndpointsForURL 

func AllHTTPEndpointsForURL(URL *url.URL,
	headers http.Header, meas ...*DNSMeasurement) ([]*HTTPEndpoint, error)

AllHTTPEndpointsForURL gathers all the HTTP endpoints for a given URL from a list of DNSMeasurements, removes duplicates and returns the result. This call may fail if we cannot determine the port from the URL, in which case we return an error. You MUST supply the headers you want to use for measuring.

func UnmeasuredHTTPEndpoints 

func UnmeasuredHTTPEndpoints(db *MeasurementDB, URL string,
	headers http.Header) ([]*HTTPEndpoint, error)

UnmeasuredHTTPEndpoints returns the endpoints whose IP address has been resolved but for which we don't have any measurement inside of the given database. The returned list will be empty if there is no such endpoint in the DB. This function will return an error if the URL is not valid or not HTTP/HTTPS.

func (*HTTPEndpoint) String 

func (e *HTTPEndpoint) String() string

String converts an HTTP endpoint to a string (e.g., "8.8.8.8:443/tcp")

type HTTPEndpointMeasurement 

type HTTPEndpointMeasurement struct {
	// URL is the URL this measurement refers to.
	URL string

	// Network is the network of this endpoint.
	Network EndpointNetwork

	// Address is the address of this endpoint.
	Address string

	// An HTTPEndpointMeasurement is a Measurement.
	*Measurement
}

HTTPEndpointMeasurement is an HTTP endpoint measurement.

type HTTPRedirectEvent 

type HTTPRedirectEvent struct {
	// URL is the URL triggering the redirect.
	URL *url.URL

	// Location is the URL to which we're redirected.
	Location *url.URL

	// Cookies contains the cookies for Location.
	Cookies []*http.Cookie

	// The Error field can have three values:
	//
	// - nil if the redirect occurred;
	//
	// - ErrHTTPTooManyRedirects when we see too many redirections;
	//
	// - http.ErrUseLastResponse if redirections are disabled.
	Error error
}

HTTPRedirectEvent records an HTTP redirect.

type HTTPRequest 

type HTTPRequest struct {
	// Names consistent with df-001-http.md
	Method  string          `json:"method"`
	URL     string          `json:"url"`
	Headers ArchivalHeaders `json:"headers"`
}

HTTPRequest is the HTTP request.

type HTTPResponse 

type HTTPResponse struct {
	// Names consistent with df-001-http.md
	Code            int64               `json:"code"`
	Headers         ArchivalHeaders     `json:"headers"`
	Body            *ArchivalBinaryData `json:"body"`
	BodyIsTruncated bool                `json:"body_is_truncated"`

	// Fields not part of the spec
	BodyLength int64 `json:"x_body_length"`
	BodyIsUTF8 bool  `json:"x_body_is_utf8"`
}

HTTPResponse is the HTTP response.

type HTTPRoundTripEvent 

type HTTPRoundTripEvent struct {
	Failure                 *string
	Method                  string
	URL                     string
	RequestHeaders          http.Header
	StatusCode              int64
	ResponseHeaders         http.Header
	ResponseBody            []byte
	ResponseBodyLength      int64
	ResponseBodyIsTruncated bool
	ResponseBodyIsUTF8      bool
	Finished                float64
	Started                 float64
	Oddity                  Oddity
}

HTTPRoundTripEvent contains information about an HTTP round trip.

type HTTPTransportDB 

type HTTPTransportDB struct {
	model.HTTPTransport

	// Begin is when we started measuring.
	Begin time.Time

	// DB is where to write events.
	DB WritableDB

	// MaxBodySnapshotSize is the maximum size of the body
	// snapshot that we take during a round trip.
	MaxBodySnapshotSize int64
}

HTTPTransportDB is an implementation of HTTPTransport that writes measurement events into a WritableDB.

There are many factories to construct this data type. Otherwise, you can construct it manually. In which case, do not modify public fields during usage, since this may cause a data race.

func NewTracingHTTPTransport 

func NewTracingHTTPTransport(logger model.Logger, begin time.Time, db WritableDB,
	resolver model.Resolver, dialer model.Dialer, handshaker model.TLSHandshaker,
	maxBodySnapshotSize int64) *HTTPTransportDB

NewTracingHTTPTransport creates a new HTTPTransport instance with events tracing.

Arguments:

- logger is the logger to use

- begin is the zero time for measurements

- db is the DB in which to write events that will eventually become the measurement

- dialer is the base dialer to establish conns

- resolver is the underlying resolver to use

- handshake is the TLS handshaker to use

- maxBodySnapshotSize is the max size of the response body snapshot to save: we'll truncate bodies larger than that.

func NewTracingHTTPTransportWithDefaultSettings 

func NewTracingHTTPTransportWithDefaultSettings(
	begin time.Time, logger model.Logger, db WritableDB) *HTTPTransportDB

NewTracingHTTPTransportWithDefaultSettings creates a new HTTP transport with tracing capabilities and default settings.

Arguments:

- begin is the zero time for measurements

- logger is the logger to use

- db is the DB in which to write events that will eventually become the measurement

func WrapHTTPTransport 

func WrapHTTPTransport(
	begin time.Time, db WritableDB, txp model.HTTPTransport,
	maxBodySnapshotSize int64) *HTTPTransportDB

WrapHTTPTransport creates a new model.HTTPTransport instance using the following configuration:

- begin is the conventional "zero time" indicating the moment when the measurement begun;

- db is the writable DB into which to write the measurement;

- txp is the underlying transport to use;

- maxBodySnapshotSize is the max size of the response body snapshot to save: we'll truncate bodies larger than that.

func (*HTTPTransportDB) RoundTrip 

func (txp *HTTPTransportDB) RoundTrip(req *http.Request) (*http.Response, error)

type MeasureURLHelper 

type MeasureURLHelper interface {
	// LookupExtraHTTPEndpoints searches for extra HTTP endpoints
	// suitable for the given URL we're measuring.
	//
	// Arguments:
	//
	// - ctx is the context for timeout/cancellation/deadline
	//
	// - URL is the URL we're currently measuring
	//
	// - headers contains the HTTP headers we wish to use
	//
	// - epnts is the current list of endpoints
	//
	// This function SHOULD return a NEW list of extra endpoints
	// it discovered and SHOULD NOT merge the epnts endpoints with
	// extra endpoints it discovered. Therefore:
	//
	// - on any kind of error it MUST return nil, err
	//
	// - on success it MUST return the NEW endpoints it discovered
	// as well as the TH measurement to be added to the measurement
	// that the URL measurer is constructing.
	//
	// It is the caller's responsibility to merge the NEW list of
	// endpoints with the ones it passed as argument.
	//
	// It is also the caller's responsibility to ENSURE that the
	// newly returned endpoints only use the few headers that our
	// test helper protocol allows one to set.
	LookupExtraHTTPEndpoints(ctx context.Context, URL *url.URL,
		headers http.Header, epnts ...*HTTPEndpoint) (
		newEpnts []*HTTPEndpoint, thMeasurement *THMeasurement, err error)
}

MeasureURLHelper is a Test Helper that discovers additional endpoints after MeasureURL has finished discovering endpoints via the usual DNS mechanism. The MeasureURLHelper:

- is used by experiments to call a real test helper, i.e., a remote service providing extra endpoints

- is used by test helpers to augment the set of endpoints discovered so far with the ones provided by a client.

type Measurement 

type Measurement struct {
	// Connect contains all the connect operations.
	Connect []*NetworkEvent

	// ReadWrite contains all the read and write operations.
	ReadWrite []*NetworkEvent

	// Close contains all the close operations.
	Close []*NetworkEvent

	// TLSHandshake contains all the TLS handshakes.
	TLSHandshake []*QUICTLSHandshakeEvent

	// QUICHandshake contains all the QUIC handshakes.
	QUICHandshake []*QUICTLSHandshakeEvent

	// LookupHost contains all the host lookups.
	LookupHost []*DNSLookupEvent

	// LookupHTTPSSvc contains all the HTTPSSvc lookups.
	LookupHTTPSSvc []*DNSLookupEvent

	// DNSRoundTrip contains all the DNS round trips.
	DNSRoundTrip []*DNSRoundTripEvent

	// HTTPRoundTrip contains all the HTTP round trips.
	HTTPRoundTrip []*HTTPRoundTripEvent

	// HTTPRedirect contains all the redirections.
	HTTPRedirect []*HTTPRedirectEvent
}

Measurement groups all the events that have the same MeasurementID. This data format is not compatible with the OONI data format.

type MeasurementDB 

type MeasurementDB struct {
	// contains filtered or unexported fields
}

MeasurementDB is a WritableDB that also allows high-level code to generate a Measurement from all the saved events.

func (*MeasurementDB) AsMeasurement 

func (db *MeasurementDB) AsMeasurement() *Measurement

AsMeasurement converts the current state of the database into a finalized Measurement structure. The original events will remain into the database. To start a new measurement cycle, just create a new MeasurementDB instance and use that.

func (*MeasurementDB) DeleteAll 

func (db *MeasurementDB) DeleteAll()

DeleteAll deletes all the content of the DB.

func (*MeasurementDB) InsertIntoClose 

func (db *MeasurementDB) InsertIntoClose(ev *NetworkEvent)

InsertIntoClose implements EventDB.InsertIntoClose.

func (*MeasurementDB) InsertIntoDNSRoundTrip 

func (db *MeasurementDB) InsertIntoDNSRoundTrip(ev *DNSRoundTripEvent)

InsertIntoDNSRoundTrip implements EventDB.InsertIntoDNSRoundTrip.

func (*MeasurementDB) InsertIntoDial 

func (db *MeasurementDB) InsertIntoDial(ev *NetworkEvent)

InsertIntoDial implements EventDB.InsertIntoDial.

func (*MeasurementDB) InsertIntoHTTPRedirect 

func (db *MeasurementDB) InsertIntoHTTPRedirect(ev *HTTPRedirectEvent)

InsertIntoHTTPRedirect implements EventDB.InsertIntoHTTPRedirect.

func (*MeasurementDB) InsertIntoHTTPRoundTrip 

func (db *MeasurementDB) InsertIntoHTTPRoundTrip(ev *HTTPRoundTripEvent)

InsertIntoHTTPRoundTrip implements EventDB.InsertIntoHTTPRoundTrip.

func (*MeasurementDB) InsertIntoLookupHTTPSSvc 

func (db *MeasurementDB) InsertIntoLookupHTTPSSvc(ev *DNSLookupEvent)

InsertIntoHTTPSSvc implements EventDB.InsertIntoHTTPSSvc

func (*MeasurementDB) InsertIntoLookupHost 

func (db *MeasurementDB) InsertIntoLookupHost(ev *DNSLookupEvent)

InsertIntoLookupHost implements EventDB.InsertIntoLookupHost.

func (*MeasurementDB) InsertIntoQUICHandshake 

func (db *MeasurementDB) InsertIntoQUICHandshake(ev *QUICTLSHandshakeEvent)

InsertIntoQUICHandshake implements EventDB.InsertIntoQUICHandshake.

func (*MeasurementDB) InsertIntoReadWrite 

func (db *MeasurementDB) InsertIntoReadWrite(ev *NetworkEvent)

InsertIntoReadWrite implements EventDB.InsertIntoReadWrite.

func (*MeasurementDB) InsertIntoTLSHandshake 

func (db *MeasurementDB) InsertIntoTLSHandshake(ev *QUICTLSHandshakeEvent)

InsertIntoTLSHandshake implements EventDB.InsertIntoTLSHandshake.

type Measurer 

type Measurer struct {
	// Begin is when we started measuring (this field is MANDATORY).
	Begin time.Time

	// DNSLookupTimeout is the OPTIONAL timeout for performing
	// a DNS lookup. If not set, we use a default value.
	//
	// Note that the underlying network implementation MAY use a
	// shorter-than-you-selected watchdog timeout. In such a case,
	// the shorter watchdog timeout will prevail.
	DNSLookupTimeout time.Duration

	// HTTPClient is the MANDATORY HTTP client for the WCTH.
	HTTPClient model.HTTPClient

	// HTTPMaxBodySnapshotSize is the OPTIONAL maximum size,
	// in bytes, of the response body snapshot we save. If this field
	// is zero or negative, we'll use a small default value.
	HTTPMaxBodySnapshotSize int64

	// HTTPRoundTripTimeout is the OPTIONAL timeout for performing
	// an HTTP round trip. If not set, we use a default value.
	//
	// Note that the underlying network implementation MAY use a
	// shorter-than-you-selected watchdog timeout. In such a case,
	// the shorter watchdog timeout will prevail.
	HTTPRoundTripTimeout time.Duration

	// Logger is the MANDATORY logger to use.
	Logger model.Logger

	// MeasureURLHelper is the OPTIONAL test helper to use when
	// we're measuring using the MeasureURL function. If this field
	// is not set, we'll not be using any helper.
	MeasureURLHelper MeasureURLHelper

	// QUICHandshakeTimeout is the OPTIONAL timeout for performing
	// a QUIC handshake. If not set, we use a default value.
	//
	// Note that the underlying network implementation MAY use a
	// shorter-than-you-selected watchdog timeout. In such a case,
	// the shorter watchdog timeout will prevail.
	QUICHandshakeTimeout time.Duration

	// Resolvers is the MANDATORY list of resolvers.
	Resolvers []*ResolverInfo

	// TCPConnectTimeout is the OPTIONAL timeout for performing
	// a tcp connect. If not set, we use a default value.
	//
	// Note that the underlying network implementation MAY use a
	// shorter-than-you-selected watchdog timeout. In such a case,
	// the shorter watchdog timeout will prevail.
	TCPconnectTimeout time.Duration

	// TLSHandshakeTimeout is the OPTIONAL timeout for performing
	// a tls handshake. If not set, we use a default value.
	//
	// Note that the underlying network implementation MAY use a
	// shorter-than-you-selected watchdog timeout. In such a case,
	// the shorter watchdog timeout will prevail.
	TLSHandshakeTimeout time.Duration

	// TLSHandshaker is the MANDATORY TLS handshaker.
	TLSHandshaker model.TLSHandshaker
}

Measurer performs measurements. If you don't use a factory for creating this type, make sure you set all the MANDATORY fields.

func NewMeasurerWithDefaultSettings 

func NewMeasurerWithDefaultSettings() *Measurer

NewMeasurerWithDefaultSettings creates a new Measurer instance using the most default settings.

func (*Measurer) EasyHTTPRoundTripGET 

func (mx *Measurer) EasyHTTPRoundTripGET(ctx context.Context, timeout time.Duration,
	URL string) (meas *ArchivalMeasurement, failure *string)

EasyHTTPRoundTripGET performs a GET with the given URL and default headers. This function will perform just a single HTTP round trip (i.e., no redirections).

Arguments:

- ctx is the context for deadline/timeout/cancellation;

- timeout is the timeout for the whole operation;

- URL is the URL to GET;

Returns:

- meas is a JSON serializable OONI measurement (this field will never be a nil pointer);

- failure is either nil or a pointer to a OONI failure.

func (*Measurer) EasyOBFS4ConnectAndHandshake 

func (mx *Measurer) EasyOBFS4ConnectAndHandshake(ctx context.Context,
	timeout time.Duration, endpoint string, dataDir string,
	rawParams map[string][]string) (meas *ArchivalMeasurement, failure *string)

EasyOBFS4ConnectAndHandshake performs a TCP connect to a TCP endpoint followed by an OBFS4 handshake. This function is designed to receive in input the Tor bridges from the OONI API.

Arguments:

- ctx is the context for deadline/timeout/cancellation;

- timeout is the timeout for the whole operation;

- endpoint is the TCP endpoint to connect to (e.g., 8.8.8.8:443 where the address part of the endpoint MUST be an IPv4 or IPv6 address and MUST NOT be a domain);

- dataDir is the data directory to use for obfs4;

- rawParams contains raw obfs4 params from the OONI API.

Returns:

- meas is a JSON serializable OONI measurement (this field will never be a nil pointer);

- failure is either nil or a pointer to a OONI failure.

func (*Measurer) EasyTCPConnect 

func (mx *Measurer) EasyTCPConnect(ctx context.Context,
	endpoint string) (meas *ArchivalMeasurement, failure *string)

EasyTCPConnect performs a TCP connect to a TCP endpoint.

Arguments:

- ctx is the context for deadline/timeout/cancellation;

- endpoint is the TCP endpoint to connect to (e.g., 8.8.8.8:443 where the address part of the endpoint MUST be an IPv4 or IPv6 address and MUST NOT be a domain).

Returns:

- meas is a JSON serializable OONI measurement (this field will never be a nil pointer);

- failure is either nil or a pointer to a OONI failure.

Note:

- we use the Measurer's TCPConnectTimeout.

func (*Measurer) EasyTLSConnectAndHandshake 

func (mx *Measurer) EasyTLSConnectAndHandshake(ctx context.Context, endpoint string,
	tlsConfig *EasyTLSConfig) (meas *ArchivalMeasurement, failure *string)

EasyTLSConnectAndHandshake performs a TCP connect to a TCP endpoint followed by a TLS handshake using the given config.

Arguments:

- ctx is the context for deadline/timeout/cancellation;

- endpoint is the TCP endpoint to connect to (e.g., 8.8.8.8:443 where the address part of the endpoint MUST be an IPv4 or IPv6 address and MUST NOT be a domain);

- tlsConfig is the EasyTLSConfig to use (MUST NOT be nil).

Returns:

- meas is a JSON serializable OONI measurement (this field will never be a nil pointer);

- failure is either nil or a pointer to a OONI failure.

Note:

- we use the Measurer's TCPConnectTimeout and TLSHandshakeTimeout.

func (*Measurer) HTTPClientGET 

func (mx *Measurer) HTTPClientGET(
	ctx context.Context, clnt model.HTTPClient, URL *url.URL) (*http.Response, error)

HTTPClientGET performs a GET operation of the given URL using the given HTTP client instance.

func (*Measurer) HTTPEndpointGet 

func (mx *Measurer) HTTPEndpointGet(
	ctx context.Context, epnt *HTTPEndpoint, jar http.CookieJar) *HTTPEndpointMeasurement

HTTPEndpointGet performs a GET request for an HTTP endpoint.

This function WILL NOT follow redirects. If there is a redirect you will see it inside the specific database table.

Arguments:

- ctx is the context allowing to timeout the operation;

- epnt is the HTTP endpoint;

- jar is the cookie jar to use.

Returns a measurement. The returned measurement is empty if the endpoint is misconfigured or the URL has an unknown scheme.

func (*Measurer) HTTPEndpointGetParallel 

func (mx *Measurer) HTTPEndpointGetParallel(ctx context.Context, parallelism int,
	jar http.CookieJar, epnts ...*HTTPEndpoint) <-chan *HTTPEndpointMeasurement

HTTPEndpointGetParallel performs an HTTPEndpointGet for each input endpoint using a pool of background goroutines.

You can choose the parallelism with the parallelism argument. If this argument is zero, or negative, we use a small default value.

This function returns to the caller a channel where to read measurements from. The channel is closed when done.

func (*Measurer) HTTPEndpointGetWithDB 

func (mx *Measurer) HTTPEndpointGetWithDB(ctx context.Context, epnt *HTTPEndpoint,
	db WritableDB, jar http.CookieJar) (err error)

HTTPEndpointGetWithDB is an HTTPEndpointGet that stores the events into the given WritableDB.

func (*Measurer) HTTPEndpointGetWithoutCookies 

func (mx *Measurer) HTTPEndpointGetWithoutCookies(
	ctx context.Context, epnt *HTTPEndpoint) *HTTPEndpointMeasurement

HTTPEndpointGetWithoutCookies is like HTTPEndpointGet but does not require you to provide a CookieJar.

func (*Measurer) LookupHTTPSSvcUDP 

func (mx *Measurer) LookupHTTPSSvcUDP(
	ctx context.Context, domain, address string) *DNSMeasurement

LookupHTTPSSvcUDP issues an HTTPSSvc query for the given domain.

Arguments:

- ctx is the context allowing to timeout the operation;

- domain is the domain to resolve (e.g., "x.org");

- address is the UDP resolver address (e.g., "dns.google:53").

Returns a DNSMeasurement.

func (*Measurer) LookupHostParallel 

func (mx *Measurer) LookupHostParallel(ctx context.Context,
	parallelism int, hostname, port string) <-chan *DNSMeasurement

LookupHostParallel is like LookupURLHostParallel but we only have in input an hostname rather than a URL. As such, we cannot determine whether to perform HTTPSSvc lookups and so we aren't going to perform this kind of lookups in this case.

You can choose the parallelism with the parallelism argument. If this argument is zero, or negative, we use a small default value.

func (*Measurer) LookupHostSystem 

func (mx *Measurer) LookupHostSystem(ctx context.Context, domain string) *DNSMeasurement

LookupHostSystem performs a LookupHost using the system resolver.

func (*Measurer) LookupHostUDP 

func (mx *Measurer) LookupHostUDP(
	ctx context.Context, domain, address string) *DNSMeasurement

LookupHostUDP is like LookupHostSystem but uses an UDP resolver.

Arguments:

- ctx is the context allowing to timeout the operation;

- domain is the domain to resolve (e.g., "x.org");

- address is the UDP resolver address (e.g., "dns.google:53").

Returns a DNSMeasurement.

func (*Measurer) LookupURLHostParallel 

func (mx *Measurer) LookupURLHostParallel(ctx context.Context, parallelism int,
	URL *url.URL, resos ...*ResolverInfo) <-chan *DNSMeasurement

LookupURLHostParallel performs an LookupHost-like operation for each resolver that you provide as argument using a pool of goroutines.

You can choose the parallelism with the parallelism argument. If this argument is zero, or negative, we use a small default value.

func (*Measurer) MeasureURL 

func (mx *Measurer) MeasureURL(
	ctx context.Context, parallelism int, URL string, headers http.Header,
	cookies http.CookieJar) (*URLMeasurement, error)

MeasureURL measures an HTTP or HTTPS URL. The DNS resolvers and the Test Helpers we use in this measurement are the ones configured into the database. The default is to use the system resolver and to use not Test Helper. Use RegisterWCTH and RegisterUDPResolvers (and other similar functions that have not been written at the moment of writing this note) to augment the set of resolvers and Test Helpers we use here.

Arguments:

- ctx is the context for timeout/cancellation.

- parallelism is the number of parallel background goroutines to use to perform parallelizable operations (i.e., operations for which `measurex` defines an `OpParallel` API where `Op` is the name of an operation implemented by `measurex`). If parallel's value is zero or negative, we use a reasonably small default.

- URL is the URL to measure.

- header contains the HTTP headers for the request.

- cookies contains the cookies we should use for measuring this URL and possibly future redirections.

To create an empty set of cookies, use NewCookieJar. It's normal to have empty cookies at the beginning. If we follow extra redirections after this run then the cookie jar will contain the cookies for following the next redirection.

We need cookies because a small amount of URLs does not redirect properly without cookies. This has been documented at https://github.com/ooni/probe/issues/1727.

func (*Measurer) MeasureURLAndFollowRedirections 

func (mx *Measurer) MeasureURLAndFollowRedirections(ctx context.Context, parallelism int,
	URL string, headers http.Header, cookies http.CookieJar) <-chan *URLMeasurement

MeasureURLAndFollowRedirections is like MeasureURL except that it _also_ follows all the HTTP redirections.

func (*Measurer) NewDialerWithSystemResolver 

func (mx *Measurer) NewDialerWithSystemResolver(db WritableDB, logger model.Logger) model.Dialer

NewDialerWithSystemResolver creates a

func (*Measurer) NewDialerWithoutResolver 

func (mx *Measurer) NewDialerWithoutResolver(db WritableDB, logger model.Logger) model.Dialer

NewDialerWithoutResolver is a convenience factory for creating a dialer that saves measurements into the DB and that is not attached to any resolver (hence only works when passed IP addresses).

func (*Measurer) NewHTTPTransportWithConn 

func (mx *Measurer) NewHTTPTransportWithConn(
	logger model.Logger, db WritableDB, conn Conn) *HTTPTransportDB

NewHTTPTransportWithConn creates and wraps an HTTPTransport that does not dial and only uses the given conn.

func (*Measurer) NewHTTPTransportWithQUICConn added in v3.15.0 

func (mx *Measurer) NewHTTPTransportWithQUICConn(
	logger model.Logger, db WritableDB, qconn quic.EarlyConnection) *HTTPTransportDB

NewHTTPTransportWithQUICConn creates and wraps an HTTPTransport that does not dial and only uses the given QUIC connection.

func (*Measurer) NewHTTPTransportWithTLSConn 

func (mx *Measurer) NewHTTPTransportWithTLSConn(
	logger model.Logger, db WritableDB, conn netxlite.TLSConn) *HTTPTransportDB

NewHTTPTransportWithTLSConn creates and wraps an HTTPTransport that does not dial and only uses the given conn.

func (*Measurer) NewQUICDialerWithoutResolver 

func (mx *Measurer) NewQUICDialerWithoutResolver(db WritableDB, logger model.Logger) model.QUICDialer

NewQUICDialerWithoutResolver creates a new QUICDialer that is not attached to any resolver. This means that every attempt to dial any address containing a domain name will fail. This QUICDialer will save any event into the WritableDB. Any QUICConn created by it will likewise save any event into the WritableDB.

func (*Measurer) NewResolverSystem 

func (mx *Measurer) NewResolverSystem(db WritableDB, logger model.Logger) model.Resolver

NewResolverSystem creates a system resolver and then wraps it using the WrapResolver function.

func (*Measurer) NewResolverUDP 

func (mx *Measurer) NewResolverUDP(db WritableDB, logger model.Logger, address string) model.Resolver

NewResolverUDP is a convenience factory for creating a Resolver using UDP that saves measurements into the DB.

Arguments:

- db is where to save events;

- logger is the logger;

- address is the resolver address (e.g., "1.1.1.1:53").

func (*Measurer) NewTLSHandshakerStdlib 

func (mx *Measurer) NewTLSHandshakerStdlib(db WritableDB, logger model.Logger) model.TLSHandshaker

NewTLSHandshakerStdlib creates a new TLS handshaker that saves results into the DB and uses the stdlib for TLS.

func (*Measurer) NewTracingHTTPTransportWithDefaultSettings 

func (mx *Measurer) NewTracingHTTPTransportWithDefaultSettings(db WritableDB) *HTTPTransportDB

NewTracingHTTPTransportWithDefaultSettings creates a new HTTP transport with tracing capabilities and default settings.

Arguments:

- db is the DB in which to write events that will eventually become the measurement

func (*Measurer) QUICHandshake 

func (mx *Measurer) QUICHandshake(ctx context.Context, address string,
	config *tls.Config) *EndpointMeasurement

QUICHandshake connects and TLS handshakes with a QUIC endpoint.

Arguments:

- ctx is the context allowing to timeout the whole operation;

- address is the endpoint address (e.g., "1.1.1.1:443");

- config contains the TLS config (see below).

You MUST set the following config fields:

- ServerName to the desired SNI or InsecureSkipVerify to skip the certificate name verification;

- RootCAs to nextlite.NewDefaultCertPool() output;

- NextProtos to the desired ALPN ([]string{"h2", "http/1.1"} for HTTPS and []string{"dot"} for DNS-over-TLS).

Returns an EndpointMeasurement.

func (*Measurer) QUICHandshakeWithDB 

func (mx *Measurer) QUICHandshakeWithDB(ctx context.Context, db WritableDB,
	address string, config *tls.Config) (quic.EarlyConnection, error)

QUICHandshakeWithDB is like QUICHandshake but uses the given db to store events rather than creating a temporary one and use it to generate a new Measurement.

func (*Measurer) TCPConnect 

func (mx *Measurer) TCPConnect(ctx context.Context, address string) *EndpointMeasurement

TCPConnect establishes a connection with a TCP endpoint.

Arguments:

- ctx is the context allowing to timeout the connect;

- address is the TCP endpoint address (e.g., "8.8.4.4:443").

Returns an EndpointMeasurement.

func (*Measurer) TCPConnectWithDB 

func (mx *Measurer) TCPConnectWithDB(ctx context.Context, db WritableDB, address string) (Conn, error)

TCPConnectWithDB is like TCPConnect but does not create a new measurement, rather it just stores the events inside of the given DB.

func (*Measurer) TLSConnectAndHandshake 

func (mx *Measurer) TLSConnectAndHandshake(ctx context.Context,
	address string, config *tls.Config) *EndpointMeasurement

TLSConnectAndHandshake connects and TLS handshakes with a TCP endpoint.

Arguments:

- ctx is the context allowing to timeout the whole operation;

- address is the endpoint address (e.g., "1.1.1.1:443");

- config contains the TLS config (see below).

You MUST set the following config fields:

- ServerName to the desired SNI or InsecureSkipVerify to skip the certificate name verification;

- RootCAs to nextlite.NewDefaultCertPool() output;

- NextProtos to the desired ALPN ([]string{"h2", "http/1.1"} for HTTPS and []string{"dot"} for DNS-over-TLS).

Caveats:

The mx.TLSHandshaker field could point to a TLS handshaker using the Go stdlib or one using gitlab.com/yawning/utls.git.

In the latter case, the content of the ClientHello message will not only depend on the config field but also on the utls.ClientHelloID thay you're using.

Returns an EndpointMeasurement.

func (*Measurer) TLSConnectAndHandshakeWithDB 

func (mx *Measurer) TLSConnectAndHandshakeWithDB(ctx context.Context,
	db WritableDB, address string, config *tls.Config) (netxlite.TLSConn, error)

TLSConnectAndHandshakeWithDB is like TLSConnectAndHandshake but uses the given DB instead of creating a new Measurement.

func (*Measurer) WrapDNSXRoundTripper 

func (mx *Measurer) WrapDNSXRoundTripper(db WritableDB, rtx model.DNSTransport) model.DNSTransport

WrapDNSXRoundTripper creates a new DNSXRoundTripper that saves events into the given WritableDB.

func (*Measurer) WrapDialer 

func (mx *Measurer) WrapDialer(db WritableDB, dialer model.Dialer) model.Dialer

WrapDialer creates a new dialer that writes events into the given WritableDB. The net.Conns created by a wrapped dialer also write into the WritableDB.

func (*Measurer) WrapHTTPTransport 

func (mx *Measurer) WrapHTTPTransport(
	db WritableDB, txp model.HTTPTransport) *HTTPTransportDB

WrapHTTPTransport creates a new transport that saves HTTP events into the WritableDB.

func (*Measurer) WrapResolver 

func (mx *Measurer) WrapResolver(db WritableDB, r model.Resolver) model.Resolver

WrapResolver creates a new Resolver that saves events into the WritableDB.

func (*Measurer) WrapTLSHandshaker 

func (mx *Measurer) WrapTLSHandshaker(db WritableDB, thx model.TLSHandshaker) model.TLSHandshaker

WrapTLSHandshaker wraps a netxlite.TLSHandshaker to return a new instance of TLSHandshaker that saves events into the DB.

type NetworkEvent 

type NetworkEvent struct {
	RemoteAddr string
	Failure    *string
	Count      int
	Operation  string
	Network    string
	Oddity     Oddity
	Finished   float64
	Started    float64
}

NetworkEvent contains a network event. This kind of events are generated by Dialer, QUICDialer, Conn, QUICConn.

type Oddity 

type Oddity string

Oddity is an unexpected result on the probe or or test helper side during a measurement. We will promote the oddity to anomaly if the probe and the test helper see different results.

type OperationLogger 

type OperationLogger struct {
	// contains filtered or unexported fields
}

OperationLogger logs about an in-progress operation

func NewOperationLogger 

func NewOperationLogger(logger model.Logger, format string, v ...interface{}) *OperationLogger

NewOperationLogger creates a new logger that logs about an in-progress operation.

func (*OperationLogger) Stop 

func (ol *OperationLogger) Stop(err error)

type QUICTLSHandshakeEvent 

type QUICTLSHandshakeEvent struct {
	CipherSuite     string
	Failure         *string
	NegotiatedProto string
	TLSVersion      string
	PeerCerts       [][]byte
	Finished        float64
	RemoteAddr      string
	SNI             string
	ALPN            []string
	SkipVerify      bool
	Oddity          Oddity
	Network         string
	Started         float64
}

QUICTLSHandshakeEvent contains a QUIC or TLS handshake event.

type ResolverInfo 

type ResolverInfo struct {
	// Network is the resolver's network (e.g., "doh", "udp")
	Network ResolverNetwork

	// Address is the address (e.g., "1.1.1.1:53", "https://1.1.1.1/dns-query")
	Address string

	// ForeignResolver is only used when Network's
	// value equals the ResolverForeign constant.
	ForeignResolver model.Resolver
}

ResolverInfo contains info about a DNS resolver.

type ResolverNetwork 

type ResolverNetwork string

ResolverNetwork identifies the network of a resolver.

type THMeasurement 

type THMeasurement struct {
	// DNS contains all the DNS related measurements.
	DNS []*DNSMeasurement

	// Endpoints contains a measurement for each endpoint
	// that was discovered by the probe or the TH.
	Endpoints []*HTTPEndpointMeasurement
}

THMeasurement is the measurement performed by the TH.

type URLMeasurement 

type URLMeasurement struct {
	// URL is the URL we're measuring.
	URL string

	// DNS contains all the DNS related measurements.
	DNS []*DNSMeasurement

	// Endpoints contains a measurement for each endpoint
	// that we discovered via DNS or TH.
	Endpoints []*HTTPEndpointMeasurement

	// RedirectURLs contain the URLs to which we should fetch
	// if we choose to follow redirections.
	RedirectURLs []string

	// TH is the measurement collected by the TH. This field
	// will be nil if we cannot contact the TH.
	TH *THMeasurement

	// TotalRuntime is the total time to measure this URL.
	TotalRuntime time.Duration

	// DNSRuntime is the time to run all DNS checks.
	DNSRuntime time.Duration

	// THRuntime is the total time to invoke all test helpers.
	THRuntime time.Duration

	// EpntsRuntime is the total time to check all the endpoints.
	EpntsRuntime time.Duration
}

URLMeasurement is the measurement of a whole URL. It contains a bunch of measurements detailing each measurement step.

type WritableDB 

type WritableDB interface {
	// InsertIntoDial saves a Dial event.
	InsertIntoDial(ev *NetworkEvent)

	// InsertIntoReadWrite saves an I/O event.
	InsertIntoReadWrite(ev *NetworkEvent)

	// InsertIntoClose saves a close event.
	InsertIntoClose(ev *NetworkEvent)

	// InsertIntoTLSHandshake saves a TLS handshake event.
	InsertIntoTLSHandshake(ev *QUICTLSHandshakeEvent)

	// InsertIntoLookupHost saves a lookup host event.
	InsertIntoLookupHost(ev *DNSLookupEvent)

	// InsertIntoLookupHTTPSvc saves an HTTPSvc lookup event.
	InsertIntoLookupHTTPSSvc(ev *DNSLookupEvent)

	// InsertIntoDNSRoundTrip saves a DNS round trip event.
	InsertIntoDNSRoundTrip(ev *DNSRoundTripEvent)

	// InsertIntoHTTPRoundTrip saves an HTTP round trip event.
	InsertIntoHTTPRoundTrip(ev *HTTPRoundTripEvent)

	// InsertIntoHTTPRedirect saves an HTTP redirect event.
	InsertIntoHTTPRedirect(ev *HTTPRedirectEvent)

	// InsertIntoQUICHandshake saves a QUIC handshake event.
	InsertIntoQUICHandshake(ev *QUICTLSHandshakeEvent)
}

WritableDB is an events "database" in which networking code (e.g., Dialer) can save measurement events (e.g., the result of a connect, a TLS handshake, a read).

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.