providers

README

Providers

This subpackage is what actually finds and downloads the proxies from online proxy providers. This is much lower-level than the encompassing library, and has no implementation of filters or making sure a proxy hasn't been used before.

Usage

A provider is a function with the following signature:

func(proxies *providers.Set, timeout time.Duration) ([]providers.Proxy, error)

Documentation

Index

• func Asset(name string) ([]byte, error)
• func AssetDigest(name string) ([sha256.Size]byte, error)
• func AssetDir(name string) ([]string, error)
• func AssetInfo(name string) (os.FileInfo, error)
• func AssetNames() []string
• func AssetString(name string) (string, error)
• func Digests() (map[string][sha256.Size]byte, error)
• func InitLog(l *logrus.Logger)
• func MustAsset(name string) []byte
• func MustAssetString(name string) string
• func RestoreAsset(dir, name string) error
• func RestoreAssets(dir, name string) error
• type Provider
• type Proxy
  • func DummyProvider(proxies *Set, timeout time.Duration) ([]Proxy, error)
  • func DummyProviderEmpty(proxies *Set, timeout time.Duration) ([]Proxy, error)
  • func DummyProviderError(proxies *Set, timeout time.Duration) ([]Proxy, error)
  • func FreeProxyLists(proxies *Set, timeout time.Duration) ([]Proxy, error)
  • func GetProxyList(proxies *Set, timeout time.Duration) ([]Proxy, error)
  • func ProxyScrape(proxies *Set, timeout time.Duration) ([]Proxy, error)
  • func Static(proxies *Set, timeout time.Duration) ([]Proxy, error)
• type Set
  • func NewSet() *Set
  • func (s *Set) Add(p Proxy)
  • func (s *Set) All() (proxies map[Proxy]bool)
  • func (s *Set) FromCountries(countries []string) (Proxy, error)
  • func (s *Set) In(p Proxy) bool
  • func (s *Set) Length() int
  • func (s *Set) List() (proxies []Proxy)
  • func (s *Set) Random() Proxy
  • func (s *Set) Remove(proxy Proxy)

Functions

func Asset

func Asset(name string) ([]byte, error)

Asset loads and returns the asset for the given name. It returns an error if the asset could not be found or could not be loaded.

func AssetDigest

func AssetDigest(name string) ([sha256.Size]byte, error)

AssetDigest returns the digest of the file with the given name. It returns an error if the asset could not be found or the digest could not be loaded.

func AssetDir

func AssetDir(name string) ([]string, error)

AssetDir returns the file names below a certain directory embedded in the file by go-bindata. For example if you run go-bindata on data/... and data contains the following hierarchy:

data/
  foo.txt
  img/
    a.png
    b.png

then AssetDir("data") would return []string{"foo.txt", "img"}, AssetDir("data/img") would return []string{"a.png", "b.png"}, AssetDir("foo.txt") and AssetDir("notexist") would return an error, and AssetDir("") will return []string{"data"}.

func AssetInfo

func AssetInfo(name string) (os.FileInfo, error)

AssetInfo loads and returns the asset info for the given name. It returns an error if the asset could not be found or could not be loaded.

func AssetNames

func AssetNames() []string

AssetNames returns the names of the assets.

func AssetString

func AssetString(name string) (string, error)

AssetString returns the asset contents as a string (instead of a []byte).

func Digests

func Digests() (map[string][sha256.Size]byte, error)

Digests returns a map of all known files and their checksums.

func InitLog

func InitLog(l *logrus.Logger)

InitLog initialises the logger with options specified.

func MustAsset

func MustAsset(name string) []byte

MustAsset is like Asset but panics when Asset would return an error. It simplifies safe initialization of global variables.

func MustAssetString

func MustAssetString(name string) string

MustAssetString is like AssetString but panics when Asset would return an error. It simplifies safe initialization of global variables.

func RestoreAsset

func RestoreAsset(dir, name string) error

RestoreAsset restores an asset under the given directory.

func RestoreAssets

func RestoreAssets(dir, name string) error

RestoreAssets restores an asset under the given directory recursively.

Types

type Provider

type Provider func(*Set, time.Duration) ([]Proxy, error)

Provider is a type alias representing a proxy provider.

type Proxy

type Proxy struct {
	URL      *url.URL `json:"url"`
	Provider string   `json:"providers"`
	Country  string   `json:"country"`

	Used bool
}

Proxy represents a proxy

func DummyProvider

func DummyProvider(proxies *Set, timeout time.Duration) ([]Proxy, error)

DummyProvider provides a fixed, small provider of proxies. It is used mainly for testing.

func DummyProviderEmpty

func DummyProviderEmpty(proxies *Set, timeout time.Duration) ([]Proxy, error)

DummyProviderEmpty provides an example of a provider that is not working, for the purpose of testing. It does not return an error.

func DummyProviderError

func DummyProviderError(proxies *Set, timeout time.Duration) ([]Proxy, error)

DummyProviderError provides an example of a provider that is not working, for the purpose of testing. Unlike DummyProviderNotWoring, this does return a 'no proxies could be gathered' error.

func FreeProxyLists

func FreeProxyLists(proxies *Set, timeout time.Duration) ([]Proxy, error)

FreeProxyLists returns the proxies that can be found on the site https://freeproxylists.com

func GetProxyList

func GetProxyList(proxies *Set, timeout time.Duration) ([]Proxy, error)

GetProxyList returns the proxies that can be found on the site https://api.getproxylist.com/proxy.

func ProxyScrape

func ProxyScrape(proxies *Set, timeout time.Duration) ([]Proxy, error)

ProxyScrape returns the proxies that can be found on the site https://proxyscrape.com.

func Static

func Static(proxies *Set, timeout time.Duration) ([]Proxy, error)

Static provides access to a static proxy list that can be used offline.

type Set

type Set struct {
	// contains filtered or unexported fields
}

Set is a utility for storing the proxies in a concurrency-safe way

func NewSet

func NewSet() *Set

NewSet creates a new set.

func (*Set) Add

func (s *Set) Add(p Proxy)

Add adds a new proxy to the set.

func (*Set) All

func (s *Set) All() (proxies map[Proxy]bool)

All returns all the proxies in the set as a map.

func (*Set) FromCountries

func (s *Set) FromCountries(countries []string) (Proxy, error)

FromCountries gets a random proxy from the specified countries.

func (*Set) In

func (s *Set) In(p Proxy) bool

In checks wheter a proxy is in the set.

func (*Set) Length

func (s *Set) Length() int

Length gets the amount of proxies being stores.

func (*Set) List

func (s *Set) List() (proxies []Proxy)

List returns all the proxies in the set as a slice.

func (*Set) Random

func (s *Set) Random() Proxy

Random gets a random proxy from the set.

func (*Set) Remove

func (s *Set) Remove(proxy Proxy)

Remove removes a proxy from a set. If the proxy doesn't exist, no change is made.