scraper

Documentation

Index

• Constants
• type Cache
  • func NewCache(globalConfig GlobalConfig) (*Cache, error)
  • func (c Cache) ListMovieScrapers() []*models.Scraper
  • func (c Cache) ListPerformerScrapers() []*models.Scraper
  • func (c Cache) ListSceneScrapers() []*models.Scraper
  • func (c *Cache) ReloadScrapers() error
  • func (c Cache) ScrapeMovieURL(url string) (*models.ScrapedMovie, error)
  • func (c Cache) ScrapePerformer(scraperID string, scrapedPerformer models.ScrapedPerformerInput) (*models.ScrapedPerformer, error)
  • func (c Cache) ScrapePerformerList(scraperID string, query string) ([]*models.ScrapedPerformer, error)
  • func (c Cache) ScrapePerformerURL(url string) (*models.ScrapedPerformer, error)
  • func (c Cache) ScrapeScene(scraperID string, scene models.SceneUpdateInput) (*models.ScrapedScene, error)
  • func (c Cache) ScrapeSceneURL(url string) (*models.ScrapedScene, error)
  • func (c *Cache) UpdateConfig(globalConfig GlobalConfig)
• type GlobalConfig

Constants

const FreeonesScraperID = "builtin_freeones"

FreeonesScraperID is the scraper ID for the built-in Freeones scraper

Types

type Cache

type Cache struct {
	// contains filtered or unexported fields
}

Cache stores scraper details.

func NewCache

func NewCache(globalConfig GlobalConfig) (*Cache, error)

NewCache returns a new Cache loading scraper configurations from the scraper path provided in the global config object. It returns a new instance and an error if the scraper directory could not be loaded.

Scraper configurations are loaded from yml files in the provided scrapers directory and any subdirectories.

func (Cache) ListMovieScrapers

func (c Cache) ListMovieScrapers() []*models.Scraper

ListMovieScrapers returns a list of scrapers that are capable of scraping scenes.

func (Cache) ListPerformerScrapers

func (c Cache) ListPerformerScrapers() []*models.Scraper

ListPerformerScrapers returns a list of scrapers that are capable of scraping performers.

func (Cache) ListSceneScrapers

func (c Cache) ListSceneScrapers() []*models.Scraper

ListSceneScrapers returns a list of scrapers that are capable of scraping scenes.

func (*Cache) ReloadScrapers

func (c *Cache) ReloadScrapers() error

ReloadScrapers clears the scraper cache and reloads from the scraper path. In the event of an error during loading, the cache will be left empty.

func (Cache) ScrapeMovieURL

func (c Cache) ScrapeMovieURL(url string) (*models.ScrapedMovie, error)

ScrapeMovieURL uses the first scraper it finds that matches the URL provided to scrape a movie. If no scrapers are found that matches the URL, then nil is returned.

func (Cache) ScrapePerformer

func (c Cache) ScrapePerformer(scraperID string, scrapedPerformer models.ScrapedPerformerInput) (*models.ScrapedPerformer, error)

ScrapePerformer uses the scraper with the provided ID to scrape a performer using the provided performer fragment.

func (Cache) ScrapePerformerList

func (c Cache) ScrapePerformerList(scraperID string, query string) ([]*models.ScrapedPerformer, error)

ScrapePerformerList uses the scraper with the provided ID to query for performers using the provided query string. It returns a list of scraped performer data.

func (Cache) ScrapePerformerURL

func (c Cache) ScrapePerformerURL(url string) (*models.ScrapedPerformer, error)

ScrapePerformerURL uses the first scraper it finds that matches the URL provided to scrape a performer. If no scrapers are found that matches the URL, then nil is returned.

func (Cache) ScrapeScene

func (c Cache) ScrapeScene(scraperID string, scene models.SceneUpdateInput) (*models.ScrapedScene, error)

ScrapeScene uses the scraper with the provided ID to scrape a scene.

func (Cache) ScrapeSceneURL

func (c Cache) ScrapeSceneURL(url string) (*models.ScrapedScene, error)

ScrapeSceneURL uses the first scraper it finds that matches the URL provided to scrape a scene. If no scrapers are found that matches the URL, then nil is returned.

func (*Cache) UpdateConfig

func (c *Cache) UpdateConfig(globalConfig GlobalConfig)

UpdateConfig updates the global config for the cache. If the scraper path has changed, ReloadScrapers will need to be called separately.

type GlobalConfig

type GlobalConfig struct {
	// User Agent used when scraping using http.
	UserAgent string

	// Path (file or remote address) to a Chrome CDP instance.
	CDPPath string
	Path    string
}

GlobalConfig contains the global scraper options.