scraper

Documentation

Index

• type ScrapedData
• type Scraper
  • func NewScraper() *Scraper
  • func (s *Scraper) AllowedDomains(domains ...string)
  • func (s *Scraper) Scrape(url string) (*ScrapedData, error)
  • func (s *Scraper) SetCustomHeaders(headers map[string]string)

Types

type ScrapedData

type ScrapedData struct {
	URL         string
	Title       string
	Text        []string
	Links       []string
	StatusCode  int
	ContentType string
}

ScrapedData represents the structured data extracted from a webpage

type Scraper

type Scraper struct {
	// contains filtered or unexported fields
}

Scraper provides web scraping functionality

func NewScraper

func NewScraper() *Scraper

NewScraper creates a new scraper instance with default configuration

func (*Scraper) AllowedDomains

func (s *Scraper) AllowedDomains(domains ...string)

AllowedDomains sets the allowed domains for scraping

func (*Scraper) Scrape

func (s *Scraper) Scrape(url string) (*ScrapedData, error)

Scrape performs web scraping on the given URL and returns structured data

func (*Scraper) SetCustomHeaders

func (s *Scraper) SetCustomHeaders(headers map[string]string)

SetCustomHeaders allows setting custom headers for the scraper