trafilatura

Documentation

Index

• func MustNew(client fetch.Client) fetch.URLFetcher
• type TrafilaturaFetcher
  • func New(client fetch.Client) (*TrafilaturaFetcher, error)
  • func (f *TrafilaturaFetcher) Close() error
  • func (f *TrafilaturaFetcher) Fetch(url *nurl.URL) (*resource.WebPage, error)
  • func (f *TrafilaturaFetcher) Open(ctx context.Context) error

Functions

func MustNew

func MustNew(client fetch.Client) fetch.URLFetcher

Types

type TrafilaturaFetcher

type TrafilaturaFetcher struct {
	// contains filtered or unexported fields
}

func New

func New(client fetch.Client) (*TrafilaturaFetcher, error)

func (*TrafilaturaFetcher) Close

func (f *TrafilaturaFetcher) Close() error

func (*TrafilaturaFetcher) Fetch

func (f *TrafilaturaFetcher) Fetch(url *nurl.URL) (*resource.WebPage, error)

Fetch a URL and return a WebPage resource. The web page will be fetched and parsed using the Trafilatura library. The returned resource will contain the metadata and content text. The request's StatusCode will be set to the HTTP status code returned. If there's an error fetching the page, in addition to the returned error, the *resource.WebPage will contain partial data pertaining to the request.

func (*TrafilaturaFetcher) Open

func (f *TrafilaturaFetcher) Open(ctx context.Context) error