crawler

Documentation

Index

• func Compact(words []string) []string
• func Parse(content string) ([]string, []string)
• type Crawler
  • func NewCrawler(userAgent, rawurl string) *Crawler
  • func (c *Crawler) Crawl() (*Metadata, error)
  • func (c *Crawler) GetHTML() (string, error)
• type Metadata
  • func (m *Metadata) Save() error

Functions

func Compact

func Compact(words []string) []string

func Parse

func Parse(content string) ([]string, []string)

Types

type Crawler

type Crawler struct {
	// contains filtered or unexported fields
}

func NewCrawler

func NewCrawler(userAgent, rawurl string) *Crawler

func (*Crawler) Crawl

func (c *Crawler) Crawl() (*Metadata, error)

Get the gatekeeper (robots.txt) from the site. Get the HTML and parse it.

func (*Crawler) GetHTML

func (c *Crawler) GetHTML() (string, error)

Attach our user agent to the headers and make the request.

type Metadata

type Metadata struct {
	Site            string
	Links, Keywords []string
}

func (*Metadata) Save

func (m *Metadata) Save() error