internal

package
v0.8.6 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 25, 2024 License: GPL-3.0 Imports: 8 Imported by: 0

Documentation

Overview

Package scrape provides a simple interface for fetching and storing web pages' metadata and text content. The `scrape` and `scrape-server` commands provide a command-line interface and a REST API, respectively.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type StorageBackedFetcher

type StorageBackedFetcher struct {
	Fetcher fetch.URLFetcher
	Storage store.URLDataStore
	// contains filtered or unexported fields
}

func NewStorageBackedFetcher

func NewStorageBackedFetcher(
	fetcher fetch.URLFetcher,
	storage store.URLDataStore,
) *StorageBackedFetcher

func (StorageBackedFetcher) Batch

func (f StorageBackedFetcher) Batch(urls []string, options fetch.BatchOptions) <-chan *resource.WebPage

func (StorageBackedFetcher) Delete

func (f StorageBackedFetcher) Delete(url *nurl.URL) (bool, error)

func (*StorageBackedFetcher) Fetch

func (f *StorageBackedFetcher) Fetch(url *nurl.URL) (*resource.WebPage, error)

func (*StorageBackedFetcher) Wait

func (f *StorageBackedFetcher) Wait() error

Wait() will block on pending saves.

func (*StorageBackedFetcher) WithAlternateURLFetcher

func (f *StorageBackedFetcher) WithAlternateURLFetcher(ctx context.Context, uf fetch.URLFetcher) (*StorageBackedFetcher, error)

WithAlternateURLFetcher returns new SBF using the same storage but a different url fetcher. This is to support headless fetching, where we want to use a different underlying http client but the same storage. Call this _after_ Open() has been called. On the source fetcher. This function will Open() the passed URLFetcher with passed context, which should be the same context that was passed to Open() on the source fetcher. Do not call Open() on the returned fetcher.

Directories

Path Synopsis
JWT token generation and verification logic.
JWT token generation and verification logic.
Key generation and related methods relevant to any storage backend.
Key generation and related methods relevant to any storage backend.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL