paginate

Documentation

Overview

Package paginate of the Dataflow kit describes Paginator interface to retrieve the next page from the current one.

Next page can be obtained in several ways:

BySelector returns a Paginator that extracts the next page from a document by querying a given CSS selector and extracting the given HTML attribute from the resulting element.

ByQueryParam returns a Paginator that returns the next page from a document by incrementing a given query parameter. Note that this will paginate infinitely - you probably want to specify a maximum number of pages to scrape by using MaxPages parameter of ScrapeOptions.

Index

• type Paginator
  • func ByQueryParam(param string) Paginator
  • func BySelector(sel, attr string) Paginator

Types

type Paginator

type Paginator interface {
	// NextPage controls the progress of the scrape.  It is called for each input
	// page, starting with the origin URL, and is expected to return the URL of
	// the next page to process.  Note that order matters - calling 'NextPage' on
	// page 1 should return page 2, not page 3.  The function should return an
	// empty string when there are no more pages to process.
	NextPage(url string, document *goquery.Selection) (string, error)
}

The Paginator interface should be implemented by things that can retrieve the next page from the current one.

func ByQueryParam

func ByQueryParam(param string) Paginator

ByQueryParam returns a Paginator that returns the next page from a document by incrementing a given query parameter. Note that this will paginate infinitely - you probably want to specify a maximum number of pages to scrape by using MaxPages parameter of ScrapeOptions.

func BySelector

func BySelector(sel, attr string) Paginator

BySelector returns a Paginator that extracts the next page from a document by querying a given CSS selector and extracting the given HTML attribute from the resulting element.