web

Documentation

Overview

Package web interacts with web pages.

Index

• func GetText(n *html.Node, addSpaces bool) string
• func SetUserAgent(ua string)
• type Page
  • func FetchPage(ctx context.Context, url string) (*Page, error)
  • func (p *Page) Query(query string) QueryResult
  • func (p *Page) QueryAll(query string) QueryAllResult
• type QueryAllResult
  • func (res QueryAllResult) Text(addSpaces bool) ([]string, error)
• type QueryResult
  • func QueryNode(root *html.Node, query string) QueryResult
  • func (res QueryResult) Attr(attr string) (string, error)
  • func (res QueryResult) Text(addSpaces bool) (string, error)

Functions

func GetText

func GetText(n *html.Node, addSpaces bool) string

GetText concatenates all text content in and under n.

func SetUserAgent

func SetUserAgent(ua string)

SetUserAgent sets a value for the "User-Agent" header to be sent in all future HTTP requests.

Types

type Page

type Page struct {
	Root *html.Node
}

Page represents a parsed HTML page.

func FetchPage

func FetchPage(ctx context.Context, url string) (*Page, error)

FetchPage fetches and parses the HTML page at the supplied URL.

func (*Page) Query

func (p *Page) Query(query string) QueryResult

Query calls QueryNode using p.Root.

func (*Page) QueryAll

func (p *Page) QueryAll(query string) QueryAllResult

QueryAll returns all nodes matched by the supplied CSS selector. Unlike Query/QueryNode, an error is not returned if no nodes are matched.

type QueryAllResult

type QueryAllResult struct {
	Nodes []*html.Node
	Err   error
}

QueryAllResult contains the result of a call to QueryAll.

func (QueryAllResult) Text

func (res QueryAllResult) Text(addSpaces bool) ([]string, error)

Text returns the contents of all text nodes under res.Nodes.

type QueryResult

type QueryResult struct {
	Node *html.Node
	Err  error
}

QueryResult contains the result of a call to Query or QueryNode.

func QueryNode

func QueryNode(root *html.Node, query string) QueryResult

QueryNode returns the first node matched by the supplied CSS selector. The returned result has a non-nil Err field if no node was matched.

func (QueryResult) Attr

func (res QueryResult) Attr(attr string) (string, error)

Attr returns the first occurrence of the named attribute. An error is returned if the attribute isn't present.

func (QueryResult) Text

func (res QueryResult) Text(addSpaces bool) (string, error)

Text recursively concatenates the contents of all child text nodes. TODO: Add more control over formatting, e.g. trimming.