fetch

Documentation

Overview

Package fetch defines interface to retrieve contents to package.

Index

• Variables
• func NeverRedirect(req *http.Request, via []*http.Request) error
• type FetchClient
  • func WithSelector(client FetchClient, selector urlmatcher.Matcher) FetchClient
• type RequestTweaker
  • func CopyParentHeaders(keys []string) RequestTweaker
  • func SetCustomHeaders(header http.Header) RequestTweaker
  • func SetReferer() RequestTweaker
• type RequestTweakerSequence
  • func (seq RequestTweakerSequence) Tweak(req, parent *http.Request) error
• type Selector
  • func (s *Selector) Match(u *url.URL) bool

Variables

var DefaultFetchClient = &http.Client{CheckRedirect: NeverRedirect}

DefaultFetchClient is a drop-in FetchClient to fetch content via HTTP in a usual manner.

var ErrURLMismatch = errors.New("fetch: URL doesn't match the fetch targets")

ErrURLMismatch is returned by WithSelector clients when the request URL does not match their selector.

Functions

func NeverRedirect

func NeverRedirect(req *http.Request, via []*http.Request) error

NeverRedirect instructs http.Client to stop handling the redirect and just return the last response instead, when set to the CheckRedirect field.

Technically, NeverRedirect is a function just returning ErrUseLastResponse.

Types

type FetchClient

type FetchClient interface {
	// Do handles an HTTP request and returns an HTTP response. It is like
	// http.Client.Do but may not send an HTTP request for real.
	//
	// Do should *not* handle redirects. See the document above.
	Do(req *http.Request) (*http.Response, error)
}

FetchClient retrieves contents from the server or other data source.

FetchClient should not handle redirects. webpackager.Runner handles redirects in its own manner, hence FetchClient should pass any 30x responses through.

An http.Client set up with NeverRedirect, such as DefaultFetchClient, meets the contracts and is the most natural choice. Other implementations may retrieve contents from other sources such as database or filesystem.

func WithSelector

func WithSelector(client FetchClient, selector urlmatcher.Matcher) FetchClient

WithSelector wraps client to issue a fetch only if the request URL matches selector.

type RequestTweaker

type RequestTweaker interface {
	// Tweak mutates req. parent is the request which spawned req. If req is
	// a request to fetch a subresource, parent is typically the one used to
	// fetch the main resource. parent can be nil (e.g. when req is a request
	// to fetch a main resource) and should not be mutated.
	Tweak(req *http.Request, parent *http.Request) error
}

RequestTweaker mutates an http.Request to set the refer(r)er, add custom HTTP headers, and so on.

var DefaultRequestTweaker RequestTweaker = SetReferer()

DefaultRequestTweaker is a RequestTweaker used by default.

func CopyParentHeaders

func CopyParentHeaders(keys []string) RequestTweaker

CopyParentHeaders copies the header fields of the provided keys from the parent request. When the tweaked request already has those header fields, their values will be overwritten by the values from the parent request. CopyParentHeaders, however, only mutates the header fields present in the parent request and keeps all other header fields untouched.

keys are case insensitive: they are canonicalized by http.CanonicalHeaderKey.

func SetCustomHeaders

func SetCustomHeaders(header http.Header) RequestTweaker

SetCustomHeaders populates the provided HTTP header fields to the request. When the request already has the same header fields, their values will be overwritten with the values from the provided http.Header.

func SetReferer

func SetReferer() RequestTweaker

SetReferer sets the Referer HTTP header with the parent request URL.

type RequestTweakerSequence

type RequestTweakerSequence []RequestTweaker

RequestTweakerSequence consists of a series of RequestTweakers.

func (RequestTweakerSequence) Tweak

func (seq RequestTweakerSequence) Tweak(req, parent *http.Request) error

Tweak invokes all tweakers in order. It fails immediately when some tweaker returns an error, in which case the subsequent tweakers will not run.

type Selector

type Selector struct {
	Allow []urlmatcher.Matcher
	Deny  []urlmatcher.Matcher
}

Selector is a urlmatcher.Matcher designed for use with WithSelector. It is almost equivalent to

AllOf(AnyOf(Allow...), Not(AnyOf(Deny...)))

but empty Allow is interpreted as "allow any." Empty Deny is interpreted as "deny none," like AnyOf(empty). Using Selector with WithSelector makes the selector easier to understand, especially for those familiar with the Allow/Deny style. Using Selector is not mandatory though: WithSelector can use any urlmatcher.Matcher as the selector.

func (*Selector) Match

func (s *Selector) Match(u *url.URL) bool

Match implements the urlmatcher.Matcher interface.