crawler

Documentation

Index

• type Crawler
  • func New(db *db.DB, wiki *wikipedia.Client) *Crawler
  • func (c *Crawler) CrawlOnce(ctx context.Context) error

Types

type Crawler

type Crawler struct {
	// contains filtered or unexported fields
}

func New

func New(db *db.DB, wiki *wikipedia.Client) *Crawler

New creates a new crawler.

func (*Crawler) CrawlOnce

func (c *Crawler) CrawlOnce(ctx context.Context) error

CrawlOnce does one pull of the top list of articles and then fetches them all.