gogetter

README

gogetter

Fetches open graph data for URLs

If deploying this via docker, you need to remember to mount your host certs to /etc/ssl/certs

Documentation

Index

• Constants
• type Scraper
  • func NewScraper(ua string, shouldCheckRobotsTxt bool) (*Scraper, error)
  • func (s *Scraper) ParseTags(r io.Reader, webUrl string) (wildcard.Wildcard, error)
  • func (s *Scraper) ScrapeTags(url string) (interface{}, error)

Constants

const DEFAULT_UA = "Gogetter (https://github.com/JustinTulloss/gogetter) (like GoogleBot and facebookexternalhit/1.1 and Twitterbot/1.0)"

Types

type Scraper

type Scraper struct {
	// contains filtered or unexported fields
}

A Scraper instance can be used to scrape webpages for metadata.

func NewScraper

func NewScraper(ua string, shouldCheckRobotsTxt bool) (*Scraper, error)

Creates a new scraper. If no user agent is provided, DEFAULT_UA is used.

func (*Scraper) ParseTags

func (s *Scraper) ParseTags(r io.Reader, webUrl string) (wildcard.Wildcard, error)

func (*Scraper) ScrapeTags

func (s *Scraper) ScrapeTags(url string) (interface{}, error)