webconnectivity

package
v3.10.0-beta.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 7, 2021 License: BSD-3-Clause Imports: 20 Imported by: 0

Documentation

Overview

Package webconnectivity implements OONI's Web Connectivity experiment.

See https://github.com/ooni/spec/blob/master/nettests/ts-017-web-connectivity.md

Index

Constants

View Source 
const (
	StatusSuccessSecure    = 1 << iota // success when using HTTPS
	StatusSuccessCleartext             // success when using HTTP
	StatusSuccessNXDOMAIN              // probe and control agree on NXDOMAIN

	StatusAnomalyControlUnreachable // cannot access the control
	StatusAnomalyControlFailure     // control failed for HTTP
	StatusAnomalyDNS                // probe seems blocked by their DNS
	StatusAnomalyHTTPDiff           // probe and control do not agree on HTTP features
	StatusAnomalyConnect            // we saw an error when connecting
	StatusAnomalyReadWrite          // we saw an error when doing I/O
	StatusAnomalyUnknown            // we don't know when the error happened
	StatusAnomalyTLSHandshake       // we think error was during TLS handshake

	StatusExperimentDNS     // we noticed something in the DNS experiment
	StatusExperimentConnect // ... in the connect experiment
	StatusExperimentHTTP    // ... in the HTTP experiment

	StatusBugNoRequests // this should never happen
)

The following set of status flags identifies in a more nuanced way the reason why we say something is blocked, accessible, etc.

This is an experimental implementation. The objective is to start using it and learning from it, to eventually understand in which direction the Web Connectivity experiment should evolve. For example, there are a bunch of flags where our understandind is fuzzy or unclear.

It also helps to write more precise unit and integration tests.

View Source 
const (
	// DNSExperimentTag is a tag indicating the DNS experiment.
	DNSExperimentTag = "dns_experiment"

	// TCPTLSExperimentTag is a tag indicating the connect experiment.
	TCPTLSExperimentTag = "tcptls_experiment"

	// HTTPExperimentTag is a tag indicating the HTTP experiment.
	HTTPExperimentTag = "http_experiment"
)

Tags describing the section of this experiment in which the data has been collected.

View Source 
const DNSNameError = "dns_name_error"

DNSNameError is the error returned by the control on NXDOMAIN

Variables

View Source 
var (
	// DNSConsistent indicates that the measurement and the
	// control have consistent DNS results.
	DNSConsistent = "consistent"

	// DNSInconsistent indicates that the measurement and the
	// control have inconsistent DNS results.
	DNSInconsistent = "inconsistent"
)
View Source 
var (
	// ErrNoAvailableTestHelpers is emitted when there are no available test helpers.
	ErrNoAvailableTestHelpers = errors.New("no available helpers")

	// ErrNoInput indicates that no input was provided
	ErrNoInput = errors.New("no input provided")

	// ErrInputIsNotAnURL indicates that the input is not an URL.
	ErrInputIsNotAnURL = errors.New("input is not an URL")

	// ErrUnsupportedInput indicates that the input URL scheme is unsupported.
	ErrUnsupportedInput = errors.New("unsupported input scheme")
)

Functions

func ComputeTCPBlocking 

func ComputeTCPBlocking(measurement []archival.TCPConnectEntry,
	control map[string]ControlTCPConnectResult) (out []archival.TCPConnectEntry)

ComputeTCPBlocking will return a copy of the input TCPConnect structure where we set the Blocking value depending on the control results.

func DetermineBlocking 

func DetermineBlocking(s Summary) interface{}

DetermineBlocking returns the value of Summary.Blocking according to the expectations of OONI data consumers (nil|false|string).

Measurement Kit sets blocking to false when accessible is true. The spec doesn't mention this possibility, as of 2019-08-20-001. Yet we implemented it back in 2016, with little explanation <https://git.io/JJHOl>.

We eventually managed to link such a change with the 0.3.4 release of Measurement Kit <https://git.io/JJHOS>. This led us to find out the related issue #867 <https://git.io/JJHOH>. From this issue it become clear that the change on Measurement Kit was applied to mirror a change implemented in OONI Probe Legacy <https://git.io/JJH3T>. In such a change, determine_blocking() was modified to return False in case no blocking was detected, to distinguish this case from the case where there was an early failure in the experiment.

Indeed, the OONI Android app uses the case where `blocking` is `null` to flag failed tests. Instead, success is identified by `blocking` being false and all other cases indicate anomaly <https://git.io/JJH3C>.

Because of that, we must preserve the original behaviour.

func GetTitle 

func GetTitle(measurementBody string) string

GetTitle returns the title or an empty string.

func HTTPBodyLengthChecks 

func HTTPBodyLengthChecks(
	tk urlgetter.TestKeys, ctrl ControlResponse) (match *bool, proportion float64)

HTTPBodyLengthChecks returns whether the measured body is reasonably long as much as the control body as well as the proportion between the two bodies. This check may return nil, nil when such a comparison would actually not be applicable.

func HTTPGetMakeDNSCache added in v3.7.0 

func HTTPGetMakeDNSCache(domain, addresses string) string

HTTPGetMakeDNSCache constructs the DNSCache option for HTTPGet by combining domain and addresses into a single string. As a corner case, if the domain is an IP address, we return an empty string. This corner case corresponds to Web Connectivity inputs like https://1.1.1.1.

func HTTPHeadersMatch 

func HTTPHeadersMatch(tk urlgetter.TestKeys, ctrl ControlResponse) *bool

HTTPHeadersMatch returns whether uncommon headers match between control and measurement, or nil if check is not applicable.

func HTTPStatusCodeMatch 

func HTTPStatusCodeMatch(tk urlgetter.TestKeys, ctrl ControlResponse) (out *bool)

HTTPStatusCodeMatch returns whether the status code of the measurement matches the status code of the control, or nil if such comparison is actually not applicable.

func HTTPTitleMatch 

func HTTPTitleMatch(tk urlgetter.TestKeys, ctrl ControlResponse) (out *bool)

HTTPTitleMatch returns whether the measurement and the control titles reasonably match, or nil if not applicable.

func NewExperimentMeasurer 

func NewExperimentMeasurer(config Config) model.ExperimentMeasurer

NewExperimentMeasurer creates a new ExperimentMeasurer.

Types

type Config 

type Config struct{}

Config contains the experiment config.

type ConnectsConfig 

type ConnectsConfig struct {
	Begin         time.Time
	Session       model.ExperimentSession
	TargetURL     *url.URL
	URLGetterURLs []string
}

ConnectsConfig contains the config for Connects

type ConnectsNoCallbacks 

type ConnectsNoCallbacks struct{}

ConnectsNoCallbacks suppresses the callbacks

func (ConnectsNoCallbacks) OnProgress 

func (ConnectsNoCallbacks) OnProgress(percentage float64, message string)

OnProgress implements ExperimentCallbacks.OnProgress

type ConnectsResult 

type ConnectsResult struct {
	AllKeys   []urlgetter.TestKeys
	Successes int
	Total     int
}

ConnectsResult contains the results of Connects

func Connects 

func Connects(ctx context.Context, config ConnectsConfig) (out ConnectsResult)

Connects performs 0..N connects (either using TCP or TLS) to check whether the resolved endpoints are reachable.

type ControlDNSResult 

type ControlDNSResult struct {
	Failure *string  `json:"failure"`
	Addrs   []string `json:"addrs"`
	ASNs    []int64  `json:"-"` // not visible from the JSON
}

ControlDNSResult is the result of the DNS lookup performed by the control vantage point.

func (*ControlDNSResult) FillASNs 

func (dns *ControlDNSResult) FillASNs(sess model.ExperimentSession)

FillASNs fills the ASNs array of ControlDNSResult. For each Addr inside of the ControlDNSResult structure, we obtain the corresponding ASN.

This is very useful to know what ASNs were the IP addresses returned by the control according to the probe's ASN database.

type ControlHTTPRequestResult 

type ControlHTTPRequestResult struct {
	BodyLength int64             `json:"body_length"`
	Failure    *string           `json:"failure"`
	Title      string            `json:"title"`
	Headers    map[string]string `json:"headers"`
	StatusCode int64             `json:"status_code"`
}

ControlHTTPRequestResult is the result of the HTTP request performed by the control vantage point.

type ControlRequest 

type ControlRequest struct {
	HTTPRequest        string              `json:"http_request"`
	HTTPRequestHeaders map[string][]string `json:"http_request_headers"`
	TCPConnect         []string            `json:"tcp_connect"`
}

ControlRequest is the request that we send to the control

type ControlResponse 

type ControlResponse struct {
	TCPConnect  map[string]ControlTCPConnectResult `json:"tcp_connect"`
	HTTPRequest ControlHTTPRequestResult           `json:"http_request"`
	DNS         ControlDNSResult                   `json:"dns"`
}

ControlResponse is the response from the control service.

func Control 

func Control(
	ctx context.Context, sess model.ExperimentSession,
	thAddr string, creq ControlRequest) (out ControlResponse, err error)

Control performs the control request and returns the response.

type ControlTCPConnectResult 

type ControlTCPConnectResult struct {
	Status  bool    `json:"status"`
	Failure *string `json:"failure"`
}

ControlTCPConnectResult is the result of the TCP connect attempt performed by the control vantage point.

type DNSAnalysisResult 

type DNSAnalysisResult struct {
	DNSConsistency *string `json:"dns_consistency"`
}

DNSAnalysisResult contains the results of analysing comparing the measurement and the control DNS results.

func DNSAnalysis 

func DNSAnalysis(URL *url.URL, measurement DNSLookupResult,
	control ControlResponse) (out DNSAnalysisResult)

DNSAnalysis compares the measurement and the control DNS results. This implementation is a simplified version of the implementation of the same check implemented in Measurement Kit v0.10.11.

type DNSLookupConfig 

type DNSLookupConfig struct {
	Begin   time.Time
	Session model.ExperimentSession
	URL     *url.URL
}

DNSLookupConfig contains settings for the DNS lookup.

type DNSLookupResult 

type DNSLookupResult struct {
	Addrs    map[string]int64
	Failure  *string
	TestKeys urlgetter.TestKeys
}

DNSLookupResult contains the result of the DNS lookup.

func DNSLookup 

func DNSLookup(ctx context.Context, config DNSLookupConfig) (out DNSLookupResult)

DNSLookup performs the DNS lookup part of Web Connectivity.

func (DNSLookupResult) Addresses 

func (r DNSLookupResult) Addresses() (out []string)

Addresses returns the IP addresses in the DNSLookupResult

type EndpointInfo 

type EndpointInfo struct {
	String       string // String representation
	URLGetterURL string // URL for urlgetter
}

EndpointInfo describes a TCP/TLS endpoint.

type EndpointPort 

type EndpointPort struct {
	URLGetterScheme string
	Port            string
}

EndpointPort is the port to be used by a TCP/TLS endpoint.

func NewEndpointPort 

func NewEndpointPort(URL *url.URL) (out EndpointPort)

NewEndpointPort creates an EndpointPort from the given URL. This function panic if the scheme is not `http` or `https` as well as if the host is not valid. The latter should not happen if you used url.Parse.

type EndpointsList 

type EndpointsList []EndpointInfo

EndpointsList is a list of EndpointInfo

func NewEndpoints 

func NewEndpoints(URL *url.URL, addrs []string) (out EndpointsList)

NewEndpoints creates a list of TCP/TLS endpoints to test from the target URL and the list of resolved IP addresses.

func (EndpointsList) Endpoints 

func (el EndpointsList) Endpoints() (out []string)

Endpoints returns a list of endpoints for TCP connect

func (EndpointsList) URLs 

func (el EndpointsList) URLs() (out []string)

URLs returns a list of URLs for TCP urlgetter

type HTTPAnalysisResult 

type HTTPAnalysisResult struct {
	BodyLengthMatch *bool   `json:"body_length_match"`
	BodyProportion  float64 `json:"body_proportion"`
	StatusCodeMatch *bool   `json:"status_code_match"`
	HeadersMatch    *bool   `json:"headers_match"`
	TitleMatch      *bool   `json:"title_match"`
}

HTTPAnalysisResult contains the results of the analysis performed on the client. We obtain it by comparing the measurement and the control.

func HTTPAnalysis 

func HTTPAnalysis(tk urlgetter.TestKeys, ctrl ControlResponse) (out HTTPAnalysisResult)

HTTPAnalysis performs follow-up analysis on the webconnectivity measurement by comparing the measurement test keys and the control.

func (HTTPAnalysisResult) Log 

func (har HTTPAnalysisResult) Log(logger model.Logger)

Log logs the results of the analysis

type HTTPGetConfig 

type HTTPGetConfig struct {
	Addresses []string
	Begin     time.Time
	Session   model.ExperimentSession
	TargetURL *url.URL
}

HTTPGetConfig contains the config for HTTPGet

type HTTPGetResult 

type HTTPGetResult struct {
	TestKeys urlgetter.TestKeys
	Failure  *string
}

HTTPGetResult contains the results of HTTPGet

func HTTPGet 

func HTTPGet(ctx context.Context, config HTTPGetConfig) (out HTTPGetResult)

HTTPGet performs the HTTP/HTTPS part of Web Connectivity.

type Measurer 

type Measurer struct {
	Config Config
}

Measurer performs the measurement.

func (Measurer) ExperimentName 

func (m Measurer) ExperimentName() string

ExperimentName implements ExperimentMeasurer.ExperExperimentName.

func (Measurer) ExperimentVersion 

func (m Measurer) ExperimentVersion() string

ExperimentVersion implements ExperimentMeasurer.ExperExperimentVersion.

func (Measurer) GetSummaryKeys 

func (m Measurer) GetSummaryKeys(measurement *model.Measurement) (interface{}, error)

GetSummaryKeys implements model.ExperimentMeasurer.GetSummaryKeys.

func (Measurer) Run 

func (m Measurer) Run(
	ctx context.Context,
	sess model.ExperimentSession,
	measurement *model.Measurement,
	callbacks model.ExperimentCallbacks,
) error

Run implements ExperimentMeasurer.Run.

type Summary 

type Summary struct {
	// Accessible is nil when the measurement failed, true if we do
	// not think there was blocking, false in case of blocking.
	Accessible *bool `json:"accessible"`

	// BlockingReason indicates the cause of blocking when the Accessible
	// variable is false. BlockingReason is meaningless otherwise.
	//
	// This is an intermediate variable used to compute Blocking, which
	// is what OONI data consumers expect to see.
	BlockingReason *string `json:"-"`

	// Blocking implements the blocking variable as expected by OONI
	// data consumers. See DetermineBlocking's docs.
	Blocking interface{} `json:"blocking"`

	// Status contains zero or more status flags. This is currently
	// an experimental interface subject to change at any time.
	Status int64 `json:"x_status"`
}

Summary contains the Web Connectivity summary.

func Summarize 

func Summarize(tk *TestKeys) (out Summary)

Summarize computes the summary from the TestKeys.

func (Summary) Log 

func (s Summary) Log(logger model.Logger)

Log logs the summary using the provided logger.

type SummaryKeys 

type SummaryKeys struct {
	Accessible bool   `json:"accessible"`
	Blocking   string `json:"blocking"`
	IsAnomaly  bool   `json:"-"`
}

SummaryKeys contains summary keys for this experiment.

Note that this structure is part of the ABI contract with probe-cli therefore we should be careful when changing it.

type TestKeys 

type TestKeys struct {
	Agent          string  `json:"agent"`
	ClientResolver string  `json:"client_resolver"`
	Retries        *int64  `json:"retries"`    // unused
	SOCKSProxy     *string `json:"socksproxy"` // unused

	// For now mostly TCP/TLS "connect" experiment but we are
	// considering adding more events. An open question is
	// currently how to properly tag these events so that it
	// is rather obvious where they come from.
	//
	// See https://github.com/ooni/probe/issues/1413.
	NetworkEvents []archival.NetworkEvent `json:"network_events"`
	TLSHandshakes []archival.TLSHandshake `json:"tls_handshakes"`

	// DNS experiment
	Queries              []archival.DNSQueryEntry `json:"queries"`
	DNSExperimentFailure *string                  `json:"dns_experiment_failure"`
	DNSAnalysisResult

	// Control experiment
	ControlFailure *string         `json:"control_failure"`
	ControlRequest ControlRequest  `json:"-"`
	Control        ControlResponse `json:"control"`

	// TCP/TLS "connect" experiment
	TCPConnect          []archival.TCPConnectEntry `json:"tcp_connect"`
	TCPConnectSuccesses int                        `json:"-"`
	TCPConnectAttempts  int                        `json:"-"`

	// HTTP experiment
	Requests              []archival.RequestEntry `json:"requests"`
	HTTPExperimentFailure *string                 `json:"http_experiment_failure"`
	HTTPAnalysisResult

	// Top-level analysis
	Summary
}

TestKeys contains webconnectivity test keys.

Source Files

View all Source files

Directories

Path Synopsis
Package internal contains internal code.
 Package internal contains internal code.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.