Documentation ¶
Overview ¶
Package scrape provides a simple interface for fetching and storing web pages' metadata and text content. The `scrape` and `scrape-server` commands provide a command-line interface and a REST API, respectively.
Index ¶
- type StorageBackedFetcher
- func (f StorageBackedFetcher) Batch(urls []string, options fetch.BatchOptions) <-chan *resource.WebPage
- func (f StorageBackedFetcher) Delete(url *nurl.URL) (bool, error)
- func (f *StorageBackedFetcher) Fetch(url *nurl.URL) (*resource.WebPage, error)
- func (f *StorageBackedFetcher) Wait() error
- func (f *StorageBackedFetcher) WithAlternateURLFetcher(ctx context.Context, uf fetch.URLFetcher) (*StorageBackedFetcher, error)
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type StorageBackedFetcher ¶
type StorageBackedFetcher struct { Fetcher fetch.URLFetcher Storage store.URLDataStore // contains filtered or unexported fields }
func NewStorageBackedFetcher ¶
func NewStorageBackedFetcher( fetcher fetch.URLFetcher, storage store.URLDataStore, ) *StorageBackedFetcher
func (StorageBackedFetcher) Batch ¶
func (f StorageBackedFetcher) Batch(urls []string, options fetch.BatchOptions) <-chan *resource.WebPage
func (StorageBackedFetcher) Delete ¶
func (f StorageBackedFetcher) Delete(url *nurl.URL) (bool, error)
func (*StorageBackedFetcher) Wait ¶
func (f *StorageBackedFetcher) Wait() error
Wait() will block on pending saves.
func (*StorageBackedFetcher) WithAlternateURLFetcher ¶
func (f *StorageBackedFetcher) WithAlternateURLFetcher(ctx context.Context, uf fetch.URLFetcher) (*StorageBackedFetcher, error)
WithAlternateURLFetcher returns new SBF using the same storage but a different url fetcher. This is to support headless fetching, where we want to use a different underlying http client but the same storage. Call this _after_ Open() has been called. On the source fetcher. This function will Open() the passed URLFetcher with passed context, which should be the same context that was passed to Open() on the source fetcher. Do not call Open() on the returned fetcher.
Directories ¶
Path | Synopsis |
---|---|
JWT token generation and verification logic.
|
JWT token generation and verification logic. |
Key generation and related methods relevant to any storage backend.
|
Key generation and related methods relevant to any storage backend. |