scraper

package
v0.10.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 11, 2021 License: AGPL-3.0 Imports: 38 Imported by: 0

Documentation

Index

Constants

View Source
const FreeonesScraperID = "builtin_freeones"

FreeonesScraperID is the scraper ID for the built-in Freeones scraper

Variables

This section is empty.

Functions

func MatchScrapedMovie added in v0.10.0

func MatchScrapedMovie(qb models.MovieReader, m *models.ScrapedMovie) error

MatchScrapedMovie matches the provided movie with the movies in the database and sets the ID field if one is found.

func MatchScrapedPerformer added in v0.10.0

func MatchScrapedPerformer(qb models.PerformerReader, p *models.ScrapedPerformer) error

MatchScrapedPerformer matches the provided performer with the performers in the database and sets the ID field if one is found.

func MatchScrapedStudio added in v0.10.0

func MatchScrapedStudio(qb models.StudioReader, s *models.ScrapedStudio) error

MatchScrapedStudio matches the provided studio with the studios in the database and sets the ID field if one is found.

func MatchScrapedTag added in v0.10.0

func MatchScrapedTag(qb models.TagReader, s *models.ScrapedTag) error

MatchScrapedTag matches the provided tag with the tags in the database and sets the ID field if one is found.

Types

type Cache added in v0.3.0

type Cache struct {
	// contains filtered or unexported fields
}

Cache stores scraper details.

func NewCache added in v0.3.0

func NewCache(globalConfig GlobalConfig, txnManager models.TransactionManager) (*Cache, error)

NewCache returns a new Cache loading scraper configurations from the scraper path provided in the global config object. It returns a new instance and an error if the scraper directory could not be loaded.

Scraper configurations are loaded from yml files in the provided scrapers directory and any subdirectories.

func (Cache) ListGalleryScrapers added in v0.4.0

func (c Cache) ListGalleryScrapers() []*models.Scraper

ListGalleryScrapers returns a list of scrapers that are capable of scraping galleries.

func (Cache) ListMovieScrapers added in v0.3.0

func (c Cache) ListMovieScrapers() []*models.Scraper

ListMovieScrapers returns a list of scrapers that are capable of scraping scenes.

func (Cache) ListPerformerScrapers added in v0.3.0

func (c Cache) ListPerformerScrapers() []*models.Scraper

ListPerformerScrapers returns a list of scrapers that are capable of scraping performers.

func (Cache) ListSceneScrapers added in v0.3.0

func (c Cache) ListSceneScrapers() []*models.Scraper

ListSceneScrapers returns a list of scrapers that are capable of scraping scenes.

func (*Cache) ReloadScrapers added in v0.3.0

func (c *Cache) ReloadScrapers() error

ReloadScrapers clears the scraper cache and reloads from the scraper path. In the event of an error during loading, the cache will be left empty.

func (Cache) ScrapeGallery added in v0.4.0

func (c Cache) ScrapeGallery(scraperID string, galleryID int) (*models.ScrapedGallery, error)

ScrapeGallery uses the scraper with the provided ID to scrape a gallery using existing data.

func (Cache) ScrapeGalleryFragment added in v0.10.0

func (c Cache) ScrapeGalleryFragment(scraperID string, gallery models.ScrapedGalleryInput) (*models.ScrapedGallery, error)

ScrapeGalleryFragment uses the scraper with the provided ID to scrape a gallery.

func (Cache) ScrapeGalleryURL added in v0.4.0

func (c Cache) ScrapeGalleryURL(url string) (*models.ScrapedGallery, error)

ScrapeGalleryURL uses the first scraper it finds that matches the URL provided to scrape a scene. If no scrapers are found that matches the URL, then nil is returned.

func (Cache) ScrapeMovieURL added in v0.3.0

func (c Cache) ScrapeMovieURL(url string) (*models.ScrapedMovie, error)

ScrapeMovieURL uses the first scraper it finds that matches the URL provided to scrape a movie. If no scrapers are found that matches the URL, then nil is returned.

func (Cache) ScrapePerformer added in v0.3.0

func (c Cache) ScrapePerformer(scraperID string, scrapedPerformer models.ScrapedPerformerInput) (*models.ScrapedPerformer, error)

ScrapePerformer uses the scraper with the provided ID to scrape a performer using the provided performer fragment.

func (Cache) ScrapePerformerList added in v0.3.0

func (c Cache) ScrapePerformerList(scraperID string, query string) ([]*models.ScrapedPerformer, error)

ScrapePerformerList uses the scraper with the provided ID to query for performers using the provided query string. It returns a list of scraped performer data.

func (Cache) ScrapePerformerURL added in v0.3.0

func (c Cache) ScrapePerformerURL(url string) (*models.ScrapedPerformer, error)

ScrapePerformerURL uses the first scraper it finds that matches the URL provided to scrape a performer. If no scrapers are found that matches the URL, then nil is returned.

func (Cache) ScrapeScene added in v0.3.0

func (c Cache) ScrapeScene(scraperID string, sceneID int) (*models.ScrapedScene, error)

ScrapeScene uses the scraper with the provided ID to scrape a scene using existing data.

func (Cache) ScrapeSceneFragment added in v0.10.0

func (c Cache) ScrapeSceneFragment(scraperID string, scene models.ScrapedSceneInput) (*models.ScrapedScene, error)

ScrapeSceneFragment uses the scraper with the provided ID to scrape a scene.

func (Cache) ScrapeSceneQuery added in v0.10.0

func (c Cache) ScrapeSceneQuery(scraperID string, query string) ([]*models.ScrapedScene, error)

ScrapeSceneQuery uses the scraper with the provided ID to query for scenes using the provided query string. It returns a list of scraped scene data.

func (Cache) ScrapeSceneURL added in v0.3.0

func (c Cache) ScrapeSceneURL(url string) (*models.ScrapedScene, error)

ScrapeSceneURL uses the first scraper it finds that matches the URL provided to scrape a scene. If no scrapers are found that matches the URL, then nil is returned.

func (*Cache) UpdateConfig added in v0.3.0

func (c *Cache) UpdateConfig(globalConfig GlobalConfig)

TODO - don't think this is needed UpdateConfig updates the global config for the cache. If the scraper path has changed, ReloadScrapers will need to be called separately.

type GlobalConfig added in v0.3.0

type GlobalConfig interface {
	GetScraperUserAgent() string
	GetScrapersPath() string
	GetScraperCDPPath() string
	GetScraperCertCheck() bool
}

GlobalConfig contains the global scraper options.

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL