urlgetter

package
v3.18.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 27, 2023 License: GPL-3.0 Imports: 20 Imported by: 0

Documentation

Overview

Package urlgetter implements a nettest that fetches a URL.

See https://github.com/ooni/spec/blob/master/nettests/ts-027-urlgetter.md.

This package is now frozen. Please, use measurexlite for new code. New network experiments should not depend on this package. Please see https://github.com/ooni/probe-cli/blob/master/docs/design/dd-003-step-by-step.md for details about this.

Index

Constants

This section is empty.

Variables

View Source 
var ErrHTTPRequestFailed = &netxlite.ErrWrapper{
	Failure:    httpRequestFailed,
	Operation:  netxlite.TopLevelOperation,
	WrappedErr: errors.New(httpRequestFailed),
}

ErrHTTPRequestFailed indicates that the HTTP request failed.

Functions

func MaybeUserAgent 

func MaybeUserAgent(ua string) string

MaybeUserAgent returns ua if ua is not empty. Otherwise it returns httpheader.RandomUserAgent().

func NewExperimentMeasurer 

func NewExperimentMeasurer(config Config) model.ExperimentMeasurer

NewExperimentMeasurer creates a new ExperimentMeasurer.

func RegisterExtensions 

func RegisterExtensions(m *model.Measurement)

RegisterExtensions registers the extensions used by the urlgetter experiment into the provided measurement.

Types

type Config 

type Config struct {
	// not settable from command line
	CertPool *x509.CertPool
	Timeout  time.Duration

	// settable from command line
	DNSCache          string `ooni:"Add 'DOMAIN IP...' to cache"`
	DNSHTTPHost       string `ooni:"Force using specific HTTP Host header for DNS requests"`
	DNSTLSServerName  string `ooni:"Force TLS to using a specific SNI for encrypted DNS requests"`
	DNSTLSVersion     string `ooni:"Force specific TLS version used for DoT/DoH (e.g. 'TLSv1.3')"`
	FailOnHTTPError   bool   `ooni:"Fail HTTP request if status code is 400 or above"`
	HTTP3Enabled      bool   `ooni:"use http3 instead of http/1.1 or http2"`
	HTTPHost          string `ooni:"Force using specific HTTP Host header"`
	Method            string `ooni:"Force HTTP method different than GET"`
	NoFollowRedirects bool   `ooni:"Disable following redirects"`
	NoTLSVerify       bool   `ooni:"Disable TLS verification"`
	RejectDNSBogons   bool   `ooni:"Fail DNS lookup if response contains bogons"`
	ResolverURL       string `ooni:"URL describing the resolver to use"`
	TLSServerName     string `ooni:"Force TLS to using a specific SNI in Client Hello"`
	TLSVersion        string `ooni:"Force specific TLS version (e.g. 'TLSv1.3')"`
	Tunnel            string `ooni:"Run experiment over a tunnel, e.g. psiphon"`
	UserAgent         string `ooni:"Use the specified User-Agent"`
}

Config contains the experiment's configuration.

type Configuration 

type Configuration struct {
	HTTPConfig netx.Config
	DNSClient  model.Resolver
}

The Configuration is the configuration for running a measurement.

func (Configuration) CloseIdleConnections 

func (c Configuration) CloseIdleConnections()

CloseIdleConnections will close idle connections, if needed.

type Configurer 

type Configurer struct {
	Config   Config
	Logger   model.Logger
	ProxyURL *url.URL
	Saver    *tracex.Saver
}

The Configurer job is to construct a Configuration that can later be used by the measurer to perform measurements.

func (Configurer) NewConfiguration 

func (c Configurer) NewConfiguration() (Configuration, error)

NewConfiguration builds a new measurement configuration.

type Getter 

type Getter struct {
	// Begin is the time when the experiment begun. If you do not
	// set this field, every target is measured independently.
	Begin time.Time

	// Config contains settings for this run. If not set, then
	// we will use the default config.
	Config Config

	// Session is the session for this run. This field must
	// be set otherwise the code will panic.
	Session model.ExperimentSession

	// Target is the thing to measure in this run. This field must
	// be set otherwise the code won't know what to do.
	Target string
	// contains filtered or unexported fields
}

The Getter gets the specified target in the context of the given session and with the specified config.

Other OONI experiment should use the Getter to factor code when the Getter implements the operations they wanna perform.

func (Getter) Get 

func (g Getter) Get(ctx context.Context) (TestKeys, error)

Get performs the action described by g using the given context and returning the test keys and eventually an error

type Measurer 

type Measurer struct {
	Config
}

Measurer performs the measurement.

func (Measurer) ExperimentName 

func (m Measurer) ExperimentName() string

ExperimentName implements model.ExperimentSession.ExperimentName

func (Measurer) ExperimentVersion 

func (m Measurer) ExperimentVersion() string

ExperimentVersion implements model.ExperimentSession.ExperimentVersion

func (Measurer) GetSummaryKeys 

func (m Measurer) GetSummaryKeys(measurement *model.Measurement) (interface{}, error)

GetSummaryKeys implements model.ExperimentMeasurer.GetSummaryKeys.

func (Measurer) Run 

func (m Measurer) Run(ctx context.Context, args *model.ExperimentArgs) error

Run implements model.ExperimentSession.Run

type Multi 

type Multi struct {
	// Begin is the time when the experiment begun. If you do not
	// set this field, every target is measured independently.
	Begin time.Time

	// Getter is the Getter func to be used. If this is nil we use
	// the default getter, which is what you typically want.
	Getter MultiGetter

	// Parallelism is the optional parallelism to be used. If this is
	// zero, or negative, we use a reasonable default.
	Parallelism int

	// Session is the session to be used. If this is nil, the Run
	// method will panic with a nil pointer error.
	Session model.ExperimentSession
}

Multi allows to run several urlgetters in paraller.

func (Multi) Collect 

func (m Multi) Collect(ctx context.Context, inputs []MultiInput,
	prefix string, callbacks model.ExperimentCallbacks) <-chan MultiOutput

Collect prints on the output channel the result of running urlgetter on every provided input. It closes the output channel when done.

func (Multi) CollectOverall 

func (m Multi) CollectOverall(ctx context.Context, inputChunk []MultiInput, overallStartIndex int, overallCount int,
	prefix string, callbacks model.ExperimentCallbacks) <-chan MultiOutput

CollectOverall prints on the output channel the result of running urlgetter on every provided input. You can use this method if you perform multiple collection tasks within one experiment as it allows to calculate the overall progress correctly

func (Multi) Run 

func (m Multi) Run(ctx context.Context, inputs []MultiInput) <-chan MultiOutput

Run performs several urlgetters in parallel. This function returns a channel where each result is posted. This function will always perform all the requested measurements: if the ctx is canceled or its deadline expires, then you will see a bunch of failed measurements. Since all measurements are always performed, you know you're done when you've read len(inputs) results in output.

type MultiGetter 

type MultiGetter func(ctx context.Context, g Getter) (TestKeys, error)

MultiGetter allows to override the behaviour of Multi for testing purposes.

type MultiInput 

type MultiInput struct {
	// Config contains the configuration for this target.
	Config Config

	// Target contains the target URL to measure.
	Target string
}

MultiInput is the input for Multi.Run().

type MultiOutput 

type MultiOutput struct {
	// Input is the input for which we measured.
	Input MultiInput

	// Err contains the measurement error.
	Err error

	// TestKeys contains the measured test keys.
	TestKeys TestKeys
}

MultiOutput is the output returned by Multi.Run()

type Runner 

type Runner struct {
	Config     Config
	HTTPConfig netx.Config
	Target     string
}

The Runner job is to run a single measurement

func (Runner) Run 

func (r Runner) Run(ctx context.Context) error

Run runs a measurement and returns the measurement result

type SummaryKeys 

type SummaryKeys struct {
	IsAnomaly bool `json:"-"`
}

SummaryKeys contains summary keys for this experiment.

Note that this structure is part of the ABI contract with ooniprobe therefore we should be careful when changing it.

type TestKeys 

type TestKeys struct {
	// The following fields are part of the typical JSON emitted by OONI.
	Agent           string                   `json:"agent"`
	BootstrapTime   float64                  `json:"bootstrap_time,omitempty"`
	DNSCache        []string                 `json:"dns_cache,omitempty"`
	FailedOperation *string                  `json:"failed_operation"`
	Failure         *string                  `json:"failure"`
	NetworkEvents   []tracex.NetworkEvent    `json:"network_events"`
	Queries         []tracex.DNSQueryEntry   `json:"queries"`
	Requests        []tracex.RequestEntry    `json:"requests"`
	SOCKSProxy      string                   `json:"socksproxy,omitempty"`
	TCPConnect      []tracex.TCPConnectEntry `json:"tcp_connect"`
	TLSHandshakes   []tracex.TLSHandshake    `json:"tls_handshakes"`
	Tunnel          string                   `json:"tunnel,omitempty"`

	// The following fields are not serialised but are useful to simplify
	// analysing the measurements in telegram, whatsapp, etc.
	HTTPResponseStatus    int64    `json:"-"`
	HTTPResponseBody      string   `json:"-"`
	HTTPResponseLocations []string `json:"-"`
}

TestKeys contains the experiment's result.

func DefaultMultiGetter 

func DefaultMultiGetter(ctx context.Context, g Getter) (TestKeys, error)

DefaultMultiGetter is the default MultiGetter

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.