web

package

v0.1.6 Latest Latest Go to latest Published: Mar 5, 2023 License: BSD-3-Clause Imports: 8 Imported by: 0

Details

Package web interacts with web pages.

This section is empty.

This section is empty.

func GetText(n *html.Node, addSpaces bool) string

GetText concatenates all text content in and under n.

func SetUserAgent(ua string)

SetUserAgent sets a value for the "User-Agent" header to be sent in all future HTTP requests.

type Page struct {
	Root *html.Node
}

Page represents a parsed HTML page.

func FetchPage(ctx context.Context, url string) (*Page, error)

FetchPage fetches and parses the HTML page at the supplied URL.

func (p *Page) Query(query string) QueryResult

Query calls QueryNode using p.Root.

func (p *Page) QueryAll(query string) QueryAllResult

QueryAll returns all nodes matched by the supplied CSS selector. Unlike Query/QueryNode, an error is not returned if no nodes are matched.

type QueryAllResult struct {
	Nodes []*html.Node
	Err   error
}

QueryAllResult contains the result of a call to QueryAll.

func (res QueryAllResult) Text(addSpaces bool) ([]string, error)

Text returns the contents of all text nodes under res.Nodes.

type QueryResult struct {
	Node *html.Node
	Err  error
}

QueryResult contains the result of a call to Query or QueryNode.

func QueryNode(root *html.Node, query string) QueryResult

QueryNode returns the first node matched by the supplied CSS selector. The returned result has a non-nil Err field if no node was matched.

func (res QueryResult) Attr(attr string) (string, error)

Attr returns the first occurrence of the named attribute. An error is returned if the attribute isn't present.

func (res QueryResult) Text(addSpaces bool) (string, error)

Text recursively concatenates the contents of all child text nodes. TODO: Add more control over formatting, e.g. trimming.