http

Documentation

Index

• func NewClient(opts ...ClientOpts) (*http.Client, error)
• type ClientOpts
  • func WithProxy(pr string) ClientOpts
  • func WithSocks(s string) ClientOpts
• type CookieTransport
  • func (t *CookieTransport) RoundTrip(req *http.Request) (*http.Response, error)
• type Fetcher
• type Headers
• type Transport
  • func NewTransport(c *http.Client, h Headers) *Transport
  • func (t *Transport) FetchData(url string) (io.ReadCloser, error)

Functions

func NewClient

func NewClient(opts ...ClientOpts) (*http.Client, error)

NewClient creates a new instance of http.DefaultClient according to provided set of ClientOpts.

Types

type ClientOpts

type ClientOpts func(*http.Client) error

ClientOpts represents an HTTP client option that can be passed to the NewClient function to modify http.Client options.

func WithProxy

func WithProxy(pr string) ClientOpts

WithProxy modifies an HTTP client by adding a proxy value.

func WithSocks

func WithSocks(s string) ClientOpts

WithSocks modifies an HTTP client by setting new transport with provided SOCKS value.

type CookieTransport

type CookieTransport struct {
	Cookie  string
	Wrapped http.RoundTripper
}

CookieTransport defines a http.RoundTripper wrapper for adding cookie header.

func (*CookieTransport) RoundTrip

func (t *CookieTransport) RoundTrip(req *http.Request) (*http.Response, error)

RoundTrip wraps existing http.RoundTripper and adds cookie header.

type Fetcher

type Fetcher interface {
	FetchData(url string) (io.ReadCloser, error)
}

Fetcher is an interface that wraps all network related operations performed by crawler.

type Headers

type Headers map[string]string

Headers defines a simple wrapper for client headers.

type Transport

type Transport struct {
	// contains filtered or unexported fields
}

Transport allows making a network request over HTTP protocol. It is a wrapper over HTTP.Client with a set of default params required by the crawler.

func NewTransport

func NewTransport(c *http.Client, h Headers) *Transport

NewTransport creates a new *HttpTransport with provided client and custom headers. Custom headers can override the default ones.

func (*Transport) FetchData

func (t *Transport) FetchData(url string) (io.ReadCloser, error)

FetchData makes an HTTP request using provided URL and returns the response body as io.ReadCloser interface. Each request will be prepared with provided headers list. The end client is responsible for closing the response body.