commoncrawl

Documentation

Index

• Constants
• type Client
  • func New(c *providers.Config, filters providers.Filters) (*Client, error)
  • func (c *Client) Fetch(ctx context.Context, domain string, results chan string) error
  • func (c *Client) Name() string

Constants

const (
	Name = "commoncrawl"
)

Types

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client is the structure that holds the Filters and the Client's configuration

func New

func New(c *providers.Config, filters providers.Filters) (*Client, error)

func (*Client) Fetch

func (c *Client) Fetch(ctx context.Context, domain string, results chan string) error

Fetch fetches all urls for a given domain and sends them to a channel. It returns an error should one occur.

func (*Client) Name

func (c *Client) Name() string