http

Documentation

Index

• Constants
• Variables
• func AdvancedTransformExample(key string) *diskv.PathKey
• func InverseTransformExample(pathKey *diskv.PathKey) (key string)
• type Client
  • func NewClient(log *log.Logger, cacheDir string, sd *statsd.Client) (*Client, error)
  • func (c *Client) Get(u string) (*http.Response, error)
  • func (c *Client) Head(u string) (*http.Response, error)
• type Limiter
  • func NewRateLimiter(g, p, cachesize int) (*Limiter, error)
  • func (r *Limiter) Allow() bool
  • func (r *Limiter) AllowHost(a string) bool
  • func (r Limiter) String() string
  • func (r *Limiter) Wait(ctx context.Context) error
  • func (r *Limiter) WaitHost(ctx context.Context, a string) error

Constants

const RetryCount = 2

Variables

var BANNED_DOMAINS = []string{
	"youtube.com",
	"wikipedia.org",
	"github.com",
	"reddit.com",
	"twitter.com",
	"mailchi.mp",
	"youtu.be",
	"news.ycombinator.com",
	"imgur.com",
	"archive.is",
	"archive.ph",
	"docs.google.com",
	"pinterest.com",
	"web.archive.org",
	"phys.org",
	"www.tiktok.com",
	"stackexchange.com",
	"stackoverflow.com",
}

Domains we never even make HTTP requests too

var Err400GreaterError = fmt.Errorf("remote is currently failing")

var ErrBadContentType = fmt.Errorf("url has bad content type")

var ErrBadFormat = fmt.Errorf("url has bad extension")

var ErrBannedUrl = fmt.Errorf("url banned")

var ErrBadHead = fmt.Errorf("url failed head check")

var ErrFailingRemote = fmt.Errorf("remote is known to be failing")

var ErrGlobalLimitHit = fmt.Errorf("global limit hit")

var ErrHostLimitHit = fmt.Errorf("per host limit hit")

var ErrNotFound = fmt.Errorf("url not found")

var ErrNotInCache = fmt.Errorf("url not in cache")

var ErrTeapotFailed = fmt.Errorf("teapot failing")

var ErrTooLarge = fmt.Errorf("url is too large")

var ErrTooManyRedirects = fmt.Errorf("too many redirects")

var ErrTooManyRetries = fmt.Errorf("too many retries here")

Functions

func AdvancedTransformExample

func AdvancedTransformExample(key string) *diskv.PathKey

func InverseTransformExample

func InverseTransformExample(pathKey *diskv.PathKey) (key string)

Types

type Client

type Client struct {
	// contains filtered or unexported fields
}

func NewClient

func NewClient(log *log.Logger, cacheDir string, sd *statsd.Client) (*Client, error)

func (*Client) Get

func (c *Client) Get(u string) (*http.Response, error)

func (*Client) Head

func (c *Client) Head(u string) (*http.Response, error)

type Limiter

type Limiter struct {
	// contains filtered or unexported fields
}

Limiter controls how frequently events are allowed to happen globally or per-host. It uses a token-bucket limiter for the global limit and instantiates a token-bucket limiter for every unique host. The number of per-host limiters is limited to an upper bound ("cache size").

A negative rate limit means "no limit" and a zero rate limit means "Infinite".

func NewRateLimiter

func NewRateLimiter(g, p, cachesize int) (*Limiter, error)

Create a new token bucket rate limiter that limits globally at 'g' requests/sec and per-host at 'p' requests/sec; It remembers the rate of the 'cachesize' most recent hosts (and their limits). The burst rates are pre-configured to be: Global burst limit: 3 * b; Per host burst limit: 2 * p

func (*Limiter) Allow

func (r *Limiter) Allow() bool

Allow returns true if the global rate limit can consume 1 token and false otherwise. Use this if you intend to drop/skip events that exceed a configured global rate limit, otherwise, use Wait().

func (*Limiter) AllowHost

func (r *Limiter) AllowHost(a string) bool

AllowHost returns true if the per-host rate limit for host 'a' can consume 1 token and false otherwise. Use this if you intend to drop/skip events that exceed a configured global rate limit, otherwise, use WaitHost().

func (Limiter) String

func (r Limiter) String() string

String returns a printable representation of the limiter

func (*Limiter) Wait

func (r *Limiter) Wait(ctx context.Context) error

Wait blocks until the ratelimiter permits the configured global rate limit. It returns an error if the burst exceeds the configured limit or the context is cancelled.

func (*Limiter) WaitHost

func (r *Limiter) WaitHost(ctx context.Context, a string) error

WaitHost blocks until the ratelimiter permits the configured per-host rate limit from host 'a'. It returns an error if the burst exceeds the configured limit or the context is cancelled.