webconnectivity

package
v3.20.0-alpha Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 13, 2023 License: GPL-3.0 Imports: 20 Imported by: 0

Documentation

Overview

Package webconnectivity implements OONI's Web Connectivity experiment.

See https://github.com/ooni/spec/blob/master/nettests/ts-017-web-connectivity.md

Index

Constants

View Source 
const (
	StatusSuccessSecure    = 1 << iota // success when using HTTPS
	StatusSuccessCleartext             // success when using HTTP
	StatusSuccessNXDOMAIN              // probe and control agree on NXDOMAIN

	StatusAnomalyControlUnreachable // cannot access the control
	StatusAnomalyControlFailure     // control failed for HTTP
	StatusAnomalyDNS                // probe seems blocked by their DNS
	StatusAnomalyHTTPDiff           // probe and control do not agree on HTTP features
	StatusAnomalyConnect            // we saw an error when connecting
	StatusAnomalyReadWrite          // we saw an error when doing I/O
	StatusAnomalyUnknown            // we don't know when the error happened
	StatusAnomalyTLSHandshake       // we think error was during TLS handshake

	StatusExperimentDNS     // we noticed something in the DNS experiment
	StatusExperimentConnect // ... in the connect experiment
	StatusExperimentHTTP    // ... in the HTTP experiment

	StatusBugNoRequests // this should never happen
)

The following set of status flags identifies in a more nuanced way the reason why we say something is blocked, accessible, etc.

This is an experimental implementation. The objective is to start using it and learning from it, to eventually understand in which direction the Web Connectivity experiment should evolve. For example, there are a bunch of flags where our understandind is fuzzy or unclear.

It also helps to write more precise unit and integration tests.

View Source 
const (
	// DNSExperimentTag is a tag indicating the DNS experiment.
	DNSExperimentTag = "dns_experiment"

	// TCPTLSExperimentTag is a tag indicating the connect experiment.
	TCPTLSExperimentTag = "tcptls_experiment"

	// HTTPExperimentTag is a tag indicating the HTTP experiment.
	HTTPExperimentTag = "http_experiment"
)

Tags describing the section of this experiment in which the data has been collected.

View Source 
const DNSNameError = model.THDNSNameError

DNSNameError is the error returned by the control on NXDOMAIN

Variables

View Source 
var (
	// DNSConsistent indicates that the measurement and the
	// control have consistent DNS results.
	DNSConsistent = "consistent"

	// DNSInconsistent indicates that the measurement and the
	// control have inconsistent DNS results.
	DNSInconsistent = "inconsistent"
)
View Source 
var (
	// ErrNoAvailableTestHelpers is emitted when there are no available test helpers.
	ErrNoAvailableTestHelpers = errors.New("no available helpers")

	// ErrNoInput indicates that no input was provided
	ErrNoInput = errors.New("no input provided")

	// ErrInputIsNotAnURL indicates that the input is not an URL.
	ErrInputIsNotAnURL = errors.New("input is not an URL")

	// ErrUnsupportedInput indicates that the input URL scheme is unsupported.
	ErrUnsupportedInput = errors.New("unsupported input scheme")
)

Functions

func ComputeTCPBlocking added in v3.17.0 

func ComputeTCPBlocking(measurement []tracex.TCPConnectEntry,
	control map[string]ControlTCPConnectResult) (out []tracex.TCPConnectEntry)

ComputeTCPBlocking will return a copy of the input TCPConnect structure where we set the Blocking value depending on the control results.

func DetermineBlocking added in v3.17.0 

func DetermineBlocking(s Summary) interface{}

DetermineBlocking returns the value of Summary.Blocking according to the expectations of OONI data consumers (nil|false|string).

Measurement Kit sets blocking to false when accessible is true. The spec doesn't mention this possibility, as of 2019-08-20-001. Yet we implemented it back in 2016, with little explanation <https://git.io/JJHOl>.

We eventually managed to link such a change with the 0.3.4 release of Measurement Kit <https://git.io/JJHOS>. This led us to find out the related issue #867 <https://git.io/JJHOH>. From this issue it become clear that the change on Measurement Kit was applied to mirror a change implemented in OONI Probe Legacy <https://git.io/JJH3T>. In such a change, determine_blocking() was modified to return False in case no blocking was detected, to distinguish this case from the case where there was an early failure in the experiment.

Indeed, the OONI Android app uses the case where `blocking` is `null` to flag failed tests. Instead, success is identified by `blocking` being false and all other cases indicate anomaly <https://git.io/JJH3C>.

Because of that, we must preserve the original behaviour.

func HTTPBodyLengthChecks added in v3.17.0 

func HTTPBodyLengthChecks(
	tk urlgetter.TestKeys, ctrl ControlResponse) (match *bool, proportion float64)

HTTPBodyLengthChecks returns whether the measured body is reasonably long as much as the control body as well as the proportion between the two bodies. This check may return nil, nil when such a comparison would actually not be applicable.

func HTTPGetMakeDNSCache added in v3.17.0 

func HTTPGetMakeDNSCache(domain, addresses string) string

HTTPGetMakeDNSCache constructs the DNSCache option for HTTPGet by combining domain and addresses into a single string. As a corner case, if the domain is an IP address, we return an empty string. This corner case corresponds to Web Connectivity inputs like https://1.1.1.1.

func HTTPHeadersMatch added in v3.17.0 

func HTTPHeadersMatch(tk urlgetter.TestKeys, ctrl ControlResponse) *bool

HTTPHeadersMatch returns whether uncommon headers match between control and measurement, or nil if check is not applicable.

func HTTPStatusCodeMatch added in v3.17.0 

func HTTPStatusCodeMatch(tk urlgetter.TestKeys, ctrl ControlResponse) (out *bool)

HTTPStatusCodeMatch returns whether the status code of the measurement matches the status code of the control, or nil if such comparison is actually not applicable.

func HTTPTitleMatch added in v3.17.0 

func HTTPTitleMatch(tk urlgetter.TestKeys, ctrl ControlResponse) (out *bool)

HTTPTitleMatch returns whether the measurement and the control titles reasonably match, or nil if not applicable.

func NewExperimentMeasurer 

func NewExperimentMeasurer(config Config) model.ExperimentMeasurer

NewExperimentMeasurer creates a new ExperimentMeasurer.

Types

type Config 

type Config struct{}

Config contains the experiment config.

type ConnectsConfig added in v3.17.0 

type ConnectsConfig struct {
	Begin         time.Time
	Session       model.ExperimentSession
	TargetURL     *url.URL
	URLGetterURLs []string
}

ConnectsConfig contains the config for Connects

type ConnectsNoCallbacks added in v3.17.0 

type ConnectsNoCallbacks struct{}

ConnectsNoCallbacks suppresses the callbacks

func (ConnectsNoCallbacks) OnProgress added in v3.17.0 

func (ConnectsNoCallbacks) OnProgress(percentage float64, message string)

OnProgress implements ExperimentCallbacks.OnProgress

type ConnectsResult added in v3.17.0 

type ConnectsResult struct {
	AllKeys   []urlgetter.TestKeys
	Successes int
	Total     int
}

ConnectsResult contains the results of Connects

func Connects added in v3.17.0 

func Connects(ctx context.Context, config ConnectsConfig) (out ConnectsResult)

Connects performs 0..N connects (either using TCP or TLS) to check whether the resolved endpoints are reachable.

type ControlDNSResult added in v3.17.0 

type ControlDNSResult = model.THDNSResult

Redirect to types defined inside the model package

type ControlHTTPRequestResult added in v3.17.0 

type ControlHTTPRequestResult = model.THHTTPRequestResult

Redirect to types defined inside the model package

type ControlRequest added in v3.17.0 

type ControlRequest = model.THRequest

Redirect to types defined inside the model package

type ControlResponse added in v3.17.0 

type ControlResponse = model.THResponse

Redirect to types defined inside the model package

func Control 

func Control(
	ctx context.Context, sess model.ExperimentSession,
	testhelpers []model.OOAPIService, creq ControlRequest) (ControlResponse, *model.OOAPIService, error)

Control performs the control request and returns the response.

type ControlTCPConnectResult added in v3.17.0 

type ControlTCPConnectResult = model.THTCPConnectResult

Redirect to types defined inside the model package

type DNSAnalysisResult added in v3.17.0 

type DNSAnalysisResult struct {
	DNSConsistency *string `json:"dns_consistency"`
}

DNSAnalysisResult contains the results of analysing comparing the measurement and the control DNS results.

func DNSAnalysis added in v3.17.0 

func DNSAnalysis(URL *url.URL, measurement DNSLookupResult,
	control ControlResponse) (out DNSAnalysisResult)

DNSAnalysis compares the measurement and the control DNS results. This implementation is a simplified version of the implementation of the same check implemented in Measurement Kit v0.10.11.

type DNSLookupConfig added in v3.17.0 

type DNSLookupConfig struct {
	Begin   time.Time
	Session model.ExperimentSession
	URL     *url.URL
}

DNSLookupConfig contains settings for the DNS lookup.

type DNSLookupResult added in v3.17.0 

type DNSLookupResult struct {
	Addrs    map[string]int64
	Failure  *string
	TestKeys urlgetter.TestKeys
}

DNSLookupResult contains the result of the DNS lookup.

func DNSLookup added in v3.17.0 

func DNSLookup(ctx context.Context, config DNSLookupConfig) (out DNSLookupResult)

DNSLookup performs the DNS lookup part of Web Connectivity.

func (DNSLookupResult) Addresses added in v3.17.0 

func (r DNSLookupResult) Addresses() (out []string)

Addresses returns the IP addresses in the DNSLookupResult

type EndpointInfo added in v3.17.0 

type EndpointInfo struct {
	String       string // String representation
	URLGetterURL string // URL for urlgetter
}

EndpointInfo describes a TCP/TLS endpoint.

type EndpointPort added in v3.17.0 

type EndpointPort struct {
	URLGetterScheme string
	Port            string
}

EndpointPort is the port to be used by a TCP/TLS endpoint.

func NewEndpointPort added in v3.17.0 

func NewEndpointPort(URL *url.URL) (out EndpointPort)

NewEndpointPort creates an EndpointPort from the given URL. This function panic if the scheme is not `http` or `https` as well as if the host is not valid. The latter should not happen if you used url.Parse.

type EndpointsList added in v3.17.0 

type EndpointsList []EndpointInfo

EndpointsList is a list of EndpointInfo

func NewEndpoints added in v3.17.0 

func NewEndpoints(URL *url.URL, addrs []string) (out EndpointsList)

NewEndpoints creates a list of TCP/TLS endpoints to test from the target URL and the list of resolved IP addresses.

func (EndpointsList) Endpoints added in v3.17.0 

func (el EndpointsList) Endpoints() (out []string)

Endpoints returns a list of endpoints for TCP connect

func (EndpointsList) URLs added in v3.17.0 

func (el EndpointsList) URLs() (out []string)

URLs returns a list of URLs for TCP urlgetter

type HTTPAnalysisResult added in v3.17.0 

type HTTPAnalysisResult struct {
	BodyLengthMatch *bool   `json:"body_length_match"`
	BodyProportion  float64 `json:"body_proportion"`
	StatusCodeMatch *bool   `json:"status_code_match"`
	HeadersMatch    *bool   `json:"headers_match"`
	TitleMatch      *bool   `json:"title_match"`
}

HTTPAnalysisResult contains the results of the analysis performed on the client. We obtain it by comparing the measurement and the control.

func HTTPAnalysis added in v3.17.0 

func HTTPAnalysis(tk urlgetter.TestKeys, ctrl ControlResponse) (out HTTPAnalysisResult)

HTTPAnalysis performs follow-up analysis on the webconnectivity measurement by comparing the measurement test keys and the control.

func (HTTPAnalysisResult) Log added in v3.17.0 

func (har HTTPAnalysisResult) Log(logger model.Logger)

Log logs the results of the analysis

type HTTPGetConfig added in v3.17.0 

type HTTPGetConfig struct {
	Addresses []string
	Begin     time.Time
	Session   model.ExperimentSession
	TargetURL *url.URL
}

HTTPGetConfig contains the config for HTTPGet

type HTTPGetResult added in v3.17.0 

type HTTPGetResult struct {
	TestKeys urlgetter.TestKeys
	Failure  *string
}

HTTPGetResult contains the results of HTTPGet

func HTTPGet added in v3.17.0 

func HTTPGet(ctx context.Context, config HTTPGetConfig) (out HTTPGetResult)

HTTPGet performs the HTTP/HTTPS part of Web Connectivity.

type Measurer 

type Measurer struct {
	Config Config
}

Measurer performs the measurement.

func (Measurer) ExperimentName 

func (m Measurer) ExperimentName() string

ExperimentName implements ExperimentMeasurer.ExperExperimentName.

func (Measurer) ExperimentVersion 

func (m Measurer) ExperimentVersion() string

ExperimentVersion implements ExperimentMeasurer.ExperExperimentVersion.

func (Measurer) GetSummaryKeys 

func (m Measurer) GetSummaryKeys(measurement *model.Measurement) (interface{}, error)

GetSummaryKeys implements model.ExperimentMeasurer.GetSummaryKeys.

func (Measurer) Run 

func (m Measurer) Run(ctx context.Context, args *model.ExperimentArgs) error

Run implements ExperimentMeasurer.Run.

type Summary added in v3.17.0 

type Summary struct {
	// Accessible is nil when the measurement failed, true if we do
	// not think there was blocking, false in case of blocking.
	Accessible *bool `json:"accessible"`

	// BlockingReason indicates the cause of blocking when the Accessible
	// variable is false. BlockingReason is meaningless otherwise.
	//
	// This is an intermediate variable used to compute Blocking, which
	// is what OONI data consumers expect to see.
	BlockingReason *string `json:"-"`

	// Blocking implements the blocking variable as expected by OONI
	// data consumers. See DetermineBlocking's docs.
	Blocking interface{} `json:"blocking"`

	// Status contains zero or more status flags. This is currently
	// an experimental interface subject to change at any time.
	Status int64 `json:"x_status"`
}

Summary contains the Web Connectivity summary.

func Summarize added in v3.17.0 

func Summarize(tk *TestKeys) (out Summary)

Summarize computes the summary from the TestKeys.

func (Summary) Log added in v3.17.0 

func (s Summary) Log(logger model.Logger)

Log logs the summary using the provided logger.

type SummaryKeys 

type SummaryKeys struct {
	Accessible bool   `json:"accessible"`
	Blocking   string `json:"blocking"`
	IsAnomaly  bool   `json:"-"`
}

SummaryKeys contains summary keys for this experiment.

Note that this structure is part of the ABI contract with ooniprobe therefore we should be careful when changing it.

type TestKeys 

type TestKeys struct {
	Agent          string  `json:"agent"`
	ClientResolver string  `json:"client_resolver"`
	Retries        *int64  `json:"retries"`    // unused
	SOCKSProxy     *string `json:"socksproxy"` // unused

	// For now mostly TCP/TLS "connect" experiment but we are
	// considering adding more events. An open question is
	// currently how to properly tag these events so that it
	// is rather obvious where they come from.
	//
	// See https://github.com/ooni/probe/issues/1413.
	NetworkEvents []tracex.NetworkEvent `json:"network_events"`
	TLSHandshakes []tracex.TLSHandshake `json:"tls_handshakes"`

	// DNS experiment
	Queries              []tracex.DNSQueryEntry `json:"queries"`
	DNSExperimentFailure *string                `json:"dns_experiment_failure"`
	DNSAnalysisResult

	// Control experiment
	ControlFailure *string         `json:"control_failure"`
	ControlRequest ControlRequest  `json:"-"`
	Control        ControlResponse `json:"control"`

	// TCP/TLS "connect" experiment
	TCPConnect          []tracex.TCPConnectEntry `json:"tcp_connect"`
	TCPConnectSuccesses int                      `json:"-"`
	TCPConnectAttempts  int                      `json:"-"`

	// HTTP experiment
	Requests              []tracex.RequestEntry `json:"requests"`
	HTTPExperimentFailure *string               `json:"http_experiment_failure"`
	HTTPAnalysisResult

	// Top-level analysis
	Summary

	// DNSRuntime is the time to run all DNS checks.
	DNSRuntime time.Duration `json:"x_dns_runtime"`

	// THRuntime is the total time to invoke all test helpers.
	THRuntime time.Duration `json:"x_th_runtime"`

	// TCPTLSRuntime is the total time to perform TCP/TLS "connects".
	TCPTLSRuntime time.Duration `json:"x_tcptls_runtime"`

	// HTTPRuntime is the total time to perform the HTTP GET.
	HTTPRuntime time.Duration `json:"x_http_runtime"`
}

TestKeys contains webconnectivity test keys.

Source Files

View all Source files

Directories

Path Synopsis
Package internal contains internal code.
 Package internal contains internal code.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.