websteps

package
v0.0.0-...-41ba115 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 30, 2023 License: GPL-3.0 Imports: 24 Imported by: 0

Documentation

Overview

Package websteps implements the websteps experiment.

Index

Constants

View Source 
const (
	//
	// Failure
	//
	AnalysisNXDOMAIN    = 1 << 0
	AnalysisDNSTimeout  = 1 << 1
	AnalysisBogon       = 1 << 2
	AnalysisDNSNoAnswer = 1 << 3
	AnalysisDNSRefused  = 1 << 4
	AnalysisDNSDiff     = 1 << 5
	AnalysisDNSServfail = 1 << 6
	AnalysisTCPTimeout  = 1 << 7
	AnalysisTCPRefused  = 1 << 8
	AnalysisQUICTimeout = 1 << 9
	AnalysisTLSTimeout  = 1 << 10
	AnalysisTLSEOF      = 1 << 11
	AnalysisTLSReset    = 1 << 12
	AnalysisCertificate = 1 << 13
	AnalysisHTTPDiff    = 1 << 14
	AnalysisHTTPTimeout = 1 << 15
	AnalysisHTTPReset   = 1 << 16
	AnalysisHTTPEOF     = 1 << 17

	//
	// Reserved
	//
	AnalysisInconclusive               = 1 << 32
	AnalysisProbeBug                   = 1 << 33
	AnalysisHTTPDiffStatusCode         = 1 << 34
	AnalysisHTTPDiffTitle              = 1 << 35
	AnalysisHTTPDiffHeaders            = 1 << 36
	AnalysisHTTPDiffBodyLength         = 1 << 37
	AnalysisHTTPDiffLegitimateRedirect = 1 << 38
	AnalysisHTTPDiffTransparentProxy   = 1 << 39
)

We represent analysis results using an int64 bitmask. We define the following groups of bits within the bitmask: 

0    4    8   12   16   20   24   28   32
+----+----+----+----+----+----+----+----+
|                Failure                |
+----+----+----+----+----+----+----+----+
|                Reserved               |
+----+----+----+----+----+----+----+----+
32  36   40   44   48   52   56   60   64

The failure flags indicate censorship conditions we detected. The HTTP flags provide further details regarding #httpDiff like results.

All the other flags are reserved for future. Consumers of the data format should completely ignore all the reserved flags.

View Source 
const (
	// LoopFlagGreedy makes websteps stop following redirects as soon
	// as it has found signs of censorship. This flag allows a user to
	// control whether to prioritize depth or breadth.
	LoopFlagGreedy = 1 << iota
)
View Source 
const THHMaxAcceptableMessageSize = 1 << 20

THHMaxAcceptableMessageSize is the maximum websocket/http message size.

View Source 
const THHMaxResponseBodySnapshotSize = 1 << 22

THHMaxResponseBodySnapshotSize is the maximum snapshot size accepted by the THHandle from client options.

Variables

View Source 
var ErrInvalidTHHOptions = errors.New("THHandle: invalid measurex.Options")

ErrInvalidTHHOptions indicates that some options have invalid values.

Functions

func AnalysisFlagsContainAnomalies 

func AnalysisFlagsContainAnomalies(f int64) bool

AnalysisFlagsContainAnomalies returns true if the flags contain an anomaly and false otherwise.

func ExplainFlagsUsingTagsAndSeverity 

func ExplainFlagsUsingTagsAndSeverity(flags int64) (tags []string, severity int64)

ExplainFlagsUsingTagsAndSeverity provides an explanation of a given set of flags in terms of a list of hashtags and a severity level.

func ExplainFlagsWithLogging 

func ExplainFlagsWithLogging(ei Explainable, flags int64)

ExplainFlagsWithLogging logs an explanation of the given flags.

func PredictableDNSResolvers 

func PredictableDNSResolvers() []*measurex.DNSResolverInfo

PredictableDNSResolvers always returns the same list of DNS resolvers. It's not recommended to use this function when running websteps in production. However, having predictable resolvers reduces the effort required to build a probe cache. Without forcing predictable resolvers, every websteps run possibly picks different random resolvers. Running websteps once does not therefore guarantee that you can reuse the cache produced by such a single run from another location to replicate the same measurement that generated the cache. On the contrary, forcing predictable resolvers to be Client.Resolvers gives you that guarantee.

Types

type Analysis 

type Analysis struct {
	// DNS contains the DNS results analysis.
	DNS []*AnalysisDNS `json:"dns"`

	// Endpoint contains the endpoint results analysis.
	Endpoint []*AnalysisEndpoint `json:"endpoint"`

	// TH contains the TH results analysis.
	TH []*AnalysisEndpoint `json:"th"`
}

Analysis contains the results of the analysis.

type AnalysisDNS 

type AnalysisDNS struct {
	// ID is the unique ID of this analysis.
	ID int64 `json:"id"`

	// URLMeasurementID is the related URL measurement ID.
	URLMeasurementID int64 `json:"-"`

	// Ref references the measurements we used. The first ref is the
	// measurement we're describing and the other refs instead are
	// the measurement(s) we use as the "control".
	Refs []int64 `json:"refs"`

	// Flags contains the analysis flags.
	Flags int64 `json:"flags"`
}

AnalysisDNS is the analysis of an individual lookup.

func (*AnalysisDNS) Describe 

func (ad *AnalysisDNS) Describe() string

Describes this analysis.

type AnalysisDescription 

type AnalysisDescription struct {
	// Flag is the flag value.
	Flag int64

	// Hashtag is the related hashtag.
	Hashtag string

	// Severity is the related emoji.
	Severity int64
}

AnalysisDescription maps an analysis flag to information useful to describe the same flag in a human readable way.

type AnalysisEndpoint 

type AnalysisEndpoint struct {
	// ID is the unique ID of this analysis.
	ID int64 `json:"id"`

	// URLMeasurementID is the related URL measurement ID.
	URLMeasurementID int64 `json:"-"`

	// Ref is the ID of the lookup.
	Refs []int64 `json:"refs"`

	// Flags contains the analysis flags.
	Flags int64 `json:"flags"`
}

AnalysisEndpoint is the analysis of an individual endpoint.

func (*AnalysisEndpoint) Describe 

func (ad *AnalysisEndpoint) Describe() string

Describes this analysis.

type ArchivalSingleStepMeasurement 

type ArchivalSingleStepMeasurement struct {
	// Initial measurement by the probe
	ID          int64                                   `json:"id"`
	EndpointIDs []int64                                 `json:"endpoint_ids"`
	URL         string                                  `json:"url"`
	Cookies     []string                                `json:"cookies"`
	DNS         []measurex.ArchivalDNSLookupMeasurement `json:"dns"`
	Endpoint    []measurex.ArchivalEndpointMeasurement  `json:"endpoint"`

	// Data gathered by the TH or follow-up experiments
	TH              *ArchivalTHResponse                    `json:"th"`
	DNSPing         *dnsping.ArchivalResult                `json:"dnsping"`
	ProbeAdditional []measurex.ArchivalEndpointMeasurement `json:"probe_additional"`

	// Overall analysis of this step
	Analysis *Analysis `json:"analysis"`
	Flags    int64     `json:"flags"`
}

ArchivalSingleStepMeasurement is the archival data format for a SingleStepMeasurement.

type ArchivalTHResponse 

type ArchivalTHResponse struct {
	DNS      []measurex.ArchivalDNSLookupMeasurement `json:"dns"`
	Endpoint []measurex.ArchivalEndpointMeasurement  `json:"endpoint"`
}

ArchivalTHResponse is the archival format of a TH response.

type ArchivalTestKeys 

type ArchivalTestKeys struct {
	URL   string                           `json:"url"`
	Steps []*ArchivalSingleStepMeasurement `json:"steps"`
	Flags int64                            `json:"flags"`
}

ArchivalTestKeys contains the archival test keys.

type Client 

type Client struct {
	// Input is the MANDATORY channel for receiving Input.
	Input chan string

	// MeasurerFactory is the OPTIONAL factory for creating
	// new measurer instances. If you set this field, you MUST
	// set it before starting any background worker.
	MeasurerFactory MeasurerFactory

	// NewDNSPingEngine is the MANDATORY factory for creating
	// new instances of the dnsping engine. You should set this
	// field before starting any background worker. You may
	// want to use this field for caching dnsping results.
	NewDNSPingEngine func(
		idgen dnsping.IDGenerator, queryTimeout time.Duration) dnsping.AbstractEngine

	// Output is the MANDATORY channel for emitting measurements.
	Output chan *TestKeysOrError

	// Resolvers contains the MANDATORY Resolvers to use.
	Resolvers []*measurex.DNSResolverInfo

	// THMeasurementObserver is an OPTIONAL hook allowing
	// the user to view/store the response from the TH.
	//
	// You MUST NOT modify the measurement. Doing that is
	// most likely going to result in a data race.
	THMeasurementObserver func(m *THResponse)
	// contains filtered or unexported fields
}

Client is the websteps client. You cannot create an instance of this struct manually, because the zero type does not work out of the box. You MUST use the NewClient constructor to construct a valid Client and then you can modify the public fields. You MUST do that before starting the client loop.

func NewClient 

func NewClient(dialer model.Dialer, tlsDialer model.TLSDialer, thURL string,
	clientOptions *measurex.Options) *Client

NewClient creates a new Client instance.

func (*Client) Loop 

func (c *Client) Loop(ctx context.Context, flags int64)

Loop is the client Loop.

func (*Client) THRequest 

func (c *Client) THRequest(ctx context.Context, req *THRequest) (*THResponse, error)

THRequest sends a THRequest to the TH and waits for a response.

func (*Client) THRequestAsync 

func (c *Client) THRequestAsync(
	ctx context.Context, thReq *THRequest, out chan<- *THResponseOrError)

THRequestAsync performs an async TH request posting the result on the out channel. The output channel MUST be buffered with one place in the buffer.

type Explainable 

type Explainable interface {
	// Describe returns a description of the explainable.
	Describe() string
}

Explainable is something for which we can explain a set of flags.

type MeasurerFactory 

type MeasurerFactory func(options *measurex.Options) (measurex.AbstractMeasurer, error)

MeasurerFactory is a factory for creating a measurer.

type SingleStepMeasurement 

type SingleStepMeasurement struct {
	// ProbeInitial contains the initial probe measurement.
	ProbeInitial *measurex.URLMeasurement

	// TH contains the response from the test helper.
	TH *THResponse `json:",omitempty"`

	// DNSPing contains the optional result of
	// the dnsping follow-up experiment.
	DNSPing *dnsping.Result `json:",omitempty"`

	// ProbeAdditional contains additional measurements performed
	// by the probe using extra info from the TH.
	ProbeAdditional []*measurex.EndpointMeasurement `json:",omitempty"`

	// Analysis contains the results analysis.
	Analysis *Analysis

	// Flags contains aggregate flags for this single step.
	Flags int64
}

SingleStepMeasurement contains a a single-step measurement.

func (*SingleStepMeasurement) ProbeInitialDomain 

func (ssm *SingleStepMeasurement) ProbeInitialDomain() string

ProbeInitialDomain returns the domain of ProbeInitial.Domain() or zero.

func (*SingleStepMeasurement) ProbeInitialURLMeasurementID 

func (ssm *SingleStepMeasurement) ProbeInitialURLMeasurementID() int64

ProbeInitialURLMeasurementID returns the ProbeInitial.ID value or zero.

func (*SingleStepMeasurement) ToArchival 

func (ssm *SingleStepMeasurement) ToArchival(begin time.Time) *ArchivalSingleStepMeasurement

ToArchival converts test keys to the OONI archival data format.

type THClient 

type THClient interface {
	// THRequestAsync performs an async TH request posting the result on the out channel.
	THRequestAsync(ctx context.Context, thReq *THRequest, out chan<- *THResponseOrError)

	// THRequest performs a sync TH request.
	THRequest(ctx context.Context, req *THRequest) (*THResponse, error)
}

THClient is a client for communicating with the test helper.

func NewTHClient 

func NewTHClient(dialer model.Dialer, tlsDialer model.TLSDialer, thURL string) THClient

NewTHClient creates a new client for communicating with the TH.

func NewTHClientWithDefaultSettings 

func NewTHClientWithDefaultSettings(thURL string) THClient

NewTHClient with default settings creates a new THClient using default settings.

type THHandler 

type THHandler struct {
	// Options contains the TH handler options.
	Options *THHandlerOptions

	// IDGenerator generates the next ID.
	IDGenerator *measurex.IDGenerator
}

THHandler handles TH requests.

func NewTHHandler 

func NewTHHandler(options *THHandlerOptions) *THHandler

NewTHHandler creates a new TH handler with default settings.

func (*THHandler) ServeWithHTTP 

func (thh *THHandler) ServeWithHTTP(w http.ResponseWriter, req *http.Request)

ServeWithHTTP serves clients that choose to use the HTTP API.

func (*THHandler) ServeWithWebsocket 

func (thh *THHandler) ServeWithWebsocket(w http.ResponseWriter, req *http.Request)

ServeWithWebsocket serves clients that choose to use the websocket API.

type THHandlerOptions 

type THHandlerOptions struct {
	// MeasurerFactory is the OPTIONAL factory used
	// to construct a measurer. By changing this
	// factory, you can force the THHandler to use
	// a different measurer (e.g., a caching measurer).
	MeasurerFactory MeasurerFactory

	// Resolvers contains the resolvers to use.
	Resolvers []*measurex.DNSResolverInfo

	// Saver saves measurements.
	Saver THHandlerSaver
}

THHandlerOptions contains options for the THHandler.

type THHandlerSaver 

type THHandlerSaver interface {
	// Save saves this measurement somewhere.
	Save(um *measurex.URLMeasurement)
}

THHandlerSaver allows to save THHandler results.

type THRequest 

type THRequest struct {
	// URL is the current URL.
	URL string

	// Options contains the options. Nil means using defaults.
	Options *measurex.Options `json:",omitempty"`

	// Cookies is the list of cookies to use.
	Cookies []string

	// Plan is the endpoint measurement plan.
	Plan []THRequestEndpointPlan `json:",omitempty"`
}

THRequest is a request for the TH service.

type THRequestEndpointPlan 

type THRequestEndpointPlan struct {
	// Network is the endpoint network.
	Network string

	// Address it the endpoint addr.
	Address string

	// URL is the endpoint URL.
	URL string
}

THRequestEndpointPlan is the plan for measuring an endpoint.

type THRequestHandler 

type THRequestHandler struct {
	// Options contains the options.
	Options *THHandlerOptions

	// ID is the unique ID of this request.
	ID int64
}

THRequestHandler handles a single request from a client.

type THResponse 

type THResponse struct {
	// StillRunning is a boolean flag that tells the client that
	// the test helper is still alive and running.
	StillRunning bool `json:",omitempty"`

	// URLMeasurementID is the URL measurement ID. We do not
	// serialize this field to JSON since that would be redundant.
	URLMeasurementID int64 `json:"-"`

	// DNS contains DNS measurements.
	DNS []*measurex.DNSLookupMeasurement

	// Endpoint contains the endpoints.
	Endpoint []*measurex.EndpointMeasurement
}

THResponse is the response from the TH.

func (*THResponse) ToArchival 

func (r *THResponse) ToArchival(begin time.Time) ArchivalTHResponse

ToArchival converts THResponse to its archival data format.

func (*THResponse) URLAddressList 

func (thm *THResponse) URLAddressList(domain string) (o []*measurex.URLAddress, v bool)

URLAddressList builds a []*URLAddress from the TH measurement. The domain argument is the domain of the URL we're measuring. In case the TH measurement is nil or there are no suitable addresses, the return value is a nil list and false. Otherwise, a valid list and true.

type THResponseOrError 

type THResponseOrError struct {
	// Resp is the response
	Resp *THResponse

	// Err is the error
	Err error
}

THResponseOrError is a thResponse or an error.

type TestKeys 

type TestKeys struct {
	// URL is the URL this measurement refers to.
	URL string

	// Steps contains all the steps.
	Steps []*SingleStepMeasurement

	// Flags contains the analysis flags.
	Flags int64
}

TestKeys contains the experiment test keys.

func (*TestKeys) ToArchival 

func (tk *TestKeys) ToArchival(begin time.Time) (out *ArchivalTestKeys)

ToArchival converts TestKeys to the archival data format.

type TestKeysOrError 

type TestKeysOrError struct {
	// Err is the error that occurred.
	Err error

	// TestKeys are the test keys.
	TestKeys *TestKeys
}

TestKeysOrError contains either test keys or an error.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.