internal

Documentation

Overview

Package scrape provides a simple interface for fetching and storing web pages' metadata and text content. The `scrape` and `scrape-server` commands provide a command-line interface and a REST API, respectively.

Index

• type StorageBackedFetcher
  • func NewStorageBackedFetcher(fetcher fetch.URLFetcher, storage store.URLDataStore) *StorageBackedFetcher
  • func (f StorageBackedFetcher) Batch(urls []string, options fetch.BatchOptions) <-chan *resource.WebPage
  • func (f StorageBackedFetcher) Delete(url *nurl.URL) (bool, error)
  • func (f *StorageBackedFetcher) Fetch(url *nurl.URL) (*resource.WebPage, error)
  • func (f *StorageBackedFetcher) Wait() error
  • func (f *StorageBackedFetcher) WithAlternateURLFetcher(ctx context.Context, uf fetch.URLFetcher) (*StorageBackedFetcher, error)

Types

type StorageBackedFetcher

type StorageBackedFetcher struct {
	Fetcher fetch.URLFetcher
	Storage store.URLDataStore
	// contains filtered or unexported fields
}

func NewStorageBackedFetcher

func NewStorageBackedFetcher(
	fetcher fetch.URLFetcher,
	storage store.URLDataStore,
) *StorageBackedFetcher

func (StorageBackedFetcher) Batch

func (f StorageBackedFetcher) Batch(urls []string, options fetch.BatchOptions) <-chan *resource.WebPage

func (StorageBackedFetcher) Delete

func (f StorageBackedFetcher) Delete(url *nurl.URL) (bool, error)

func (*StorageBackedFetcher) Fetch

func (f *StorageBackedFetcher) Fetch(url *nurl.URL) (*resource.WebPage, error)

func (*StorageBackedFetcher) Wait

func (f *StorageBackedFetcher) Wait() error

Wait() will block on pending saves.

func (*StorageBackedFetcher) WithAlternateURLFetcher

func (f *StorageBackedFetcher) WithAlternateURLFetcher(ctx context.Context, uf fetch.URLFetcher) (*StorageBackedFetcher, error)

WithAlternateURLFetcher returns new SBF using the same storage but a different url fetcher. This is to support headless fetching, where we want to use a different underlying http client but the same storage. Call this _after_ Open() has been called. On the source fetcher. This function will Open() the passed URLFetcher with passed context, which should be the same context that was passed to Open() on the source fetcher. Do not call Open() on the returned fetcher.