lib

Documentation

Index

• type Config
• type GeneralConfig
• type JSON
• type Results
• type ScrapeConfig
• type Scraper
  • func NewScraper(config Config) (Scraper, error)
  • func (s *Scraper) Scrape() (Results, error)
• type Value
• type ValueMap

Types

type Config

type Config struct {
	GeneralConfig GeneralConfig   `json:"config" mapstructure:"config"`
	CollyConfig   colly.Collector `json:"colly" mapstructure:"colly"`
	ScrapeConfig  ScrapeConfig    `json:"scraper" mapstructure:"scraper"`
}

Config represents the JSON file specification of the scraper. It includes general settings, colly specific settings, and the raw scraper spec

type GeneralConfig

type GeneralConfig struct {
	Sites []string
	// This logger enables logging
	Logger *log.Logger `json:"-"`
}

GeneralConfig contains the general configuration for jsonscrape

type JSON

type JSON []byte

JSON represents a JSON input

type Results

type Results map[string][]ValueMap

Results is a map of values that are the results of the scraper Uses a pointer to allow nil base types

type ScrapeConfig

type ScrapeConfig map[string]datum

ScrapeConfig represents the values that the scraper will retrieve from the various sites

type Scraper

type Scraper struct {
	Config    Config
	Collector *colly.Collector
	Results   Results
}

Scraper represents a web scraper

func NewScraper

func NewScraper(config Config) (Scraper, error)

NewScraper returns a new JSONscraper

func (*Scraper) Scrape

func (s *Scraper) Scrape() (Results, error)

Scrape runs the scraper as specified. It returns the data retrieved and/or an error from the scraping process It also will block until the scraper has stopped entirely

type Value

type Value string

Value represents the available values to access from the selected elements It corresponds to the attributes on a given element

type ValueMap

type ValueMap map[string]Value

A value map is a map from strings or keys to values which will be extracted from the given element