scraper

Documentation

Index

• func JSONMarshalUnescaped(v interface{}) ([]byte, error)
• type Config
• type Scraper
  • func New(configPath string) (*Scraper, error)
  • func (s *Scraper) Start()

Functions

func JSONMarshalUnescaped

func JSONMarshalUnescaped(v interface{}) ([]byte, error)

JSONMarshalUnescaped does what json.Marshal does without escaping &, <, >.

Types

type Config

type Config struct {
	URLs    []string  `json:"urls"`
	Targets []*target `json:"targets"`
	Output  *output   `json:"output"`
	// contains filtered or unexported fields
}

Config holds information from config file.

type Scraper

type Scraper struct {
	Config  *Config
	Results results
}

Scraper struct used for scraping activity.

func New

func New(configPath string) (*Scraper, error)

New creates new Scraper and returns pointer to it, with error (if occurred).

func (*Scraper) Start

func (s *Scraper) Start()

Start will start scraping in separate goroutine and save results when it is done in file that is defined in Config.