chapter5

Documentation

Overview

Package chapter5 is Functions, covers examples and exercises in the chapter.

Index

• Constants
• func Crawl(site, dir string) (int, error)
• func E51FindLinks(href []string, n *html.Node) []string
• func Expand(s string, f func(string) string) string
• func InitCli()
• func Join(sep string, words ...string) string
• func MaxInt(numbers ...int) int
• func MaxIntOf(n1 int, numbers ...int) int
• func MinInt(numbers ...int) int
• func MinIntOf(n1 int, numbers ...int) int
• func ParseCss(url string) ([]string, error)
• func ParseImages(url string) ([]string, error)
• func ParseLinks(url string) ([]string, error)
• func ParseOutline(url string) ([]string, error)
• func ParseOutlineCount(url string) (map[string]int, error)
• func ParseScripts(url string) ([]string, error)
• func ParseText(url string) ([]string, error)
• func PrettyHTML(url string) ([]string, error)
• func TopologicalSort(m map[string][]string) []string
• func TopologicalSortMap(m map[string]map[string]bool) []string
• type CLI
  • func (m CLI) DisplayHelp()
  • func (m CLI) ExecCmd(args []string)
• type Hyperlinks
  • func (a Hyperlinks) Len() int
  • func (a Hyperlinks) Less(i, j int) bool
  • func (a Hyperlinks) Swap(i, j int)
• type NodeType
• type NodeValue

Constants

const (
	// A link reference node type is <a
	A NodeType = "a"
	// Img <img src=”/> element
	Img = "img"
	// Script tag <script src='filename.js' />
	Script = "script"
	// Link stylesheet <link rel=stylesheet" href="styles.css">
	Link = "link"
	// Style denotes a HTML <style type="text/css"> node
	Style = "style"
)

const (
	MaxCPUs = 4 // MaxCPUs for concurrency
)

Functions

func Crawl

func Crawl(site, dir string) (int, error)

Crawl a webpage and downloads pages in that domain and saves results in destination dir.

Exercise 5.13: Modify crawl to make local copies of the pages it finds, creating directories as necessary. Don’t make copies of pages that come from a different domain. For example, if the original page comes from golang.org, save all files from there, but exclude ones from vimeo.com. Added concurrency as part of chapter 8

func E51FindLinks

func E51FindLinks(href []string, n *html.Node) []string

E51FindLinks Exercise 5.1 make links using traversal non-loop recursive

func Expand

func Expand(s string, f func(string) string) string

Expand exercise 5.9 replaces $var with f(var) and returns output

func InitCli

func InitCli()

InitCli for command: the-gpl parse -site=http://...

  eg: the-gpl parse -type=links     -site=https://www.yahoo.com
		   the-gpl parse -type=outline -site=https://www.yahoo.com
		   the-gpl parse -type=images -site=https://www.yahoo.com
		   the-gpl parse -type=scripts -site=https://www.yahoo.com
		   the-gpl parse -type=scripts -site=https://www.yahoo.com
		   the-gpl parse -type=css -site=https://www.yahoo.com
		   the-gpl parse -type=pretty -site=https://www.yahoo.com
		   the-gpl parse -type=crawl -site=https://www.yahoo.com -dir=dest-dir

func Join

func Join(sep string, words ...string) string

Join words by adding separator among them.

Exercise 5.15: Write a variadic version of strings.Join

func MaxInt

func MaxInt(numbers ...int) int

MaxInt returns maximum of integers, if no int specified return MaxInt

Exercise 5.15: Write variadic functions max and min, analogous to sum. What should these
functions do when called with no arguments?

func MaxIntOf

func MaxIntOf(n1 int, numbers ...int) int

MaxIntOf returns maximum of integers with at least 1 argument

Exercise 5.15: Write variants that require at least one argument.

func MinInt

func MinInt(numbers ...int) int

MinInt finds minimum of numbers in a series, return minimum int number if nothing to compare

func MinIntOf

func MinIntOf(n1 int, numbers ...int) int

MinIntOf finds minimum of integers with at least one int as argument

func ParseCss

func ParseCss(url string) ([]string, error)

ParseCss returns all full CSS unique href links of a website pointed to by url.

func ParseImages

func ParseImages(url string) ([]string, error)

ParseImages returns all full images unique href links of a website pointed to by url.

func ParseLinks

func ParseLinks(url string) ([]string, error)

ParseLinks returns all full path unique href links of a website pointed to by url.

func ParseOutline

func ParseOutline(url string) ([]string, error)

ParseOutline returns outline of a website pointed to by url.

The outline is array of strings

func ParseOutlineCount

func ParseOutlineCount(url string) (map[string]int, error)

ParseOutlineCount returns count of tags website pointed to by url.

The outline is array of strings

func ParseScripts

func ParseScripts(url string) ([]string, error)

ParseScripts returns all full scripts unique href links of a website pointed to by url.

func ParseText

func ParseText(url string) ([]string, error)

ParseText returns text nodes data except script and style elements

func PrettyHTML

func PrettyHTML(url string) ([]string, error)

PrettyHTML prints HTML of an url in pretty fashion

func TopologicalSort

func TopologicalSort(m map[string][]string) []string

TopologicalSort sorts a map into a topological sorted string of dependencies

func TopologicalSortMap

func TopologicalSortMap(m map[string]map[string]bool) []string

TopologicalSortMap topological-sorts using map.

Exercise 5.10: Rewrite TopologicalSort to use maps instead of slices and eliminate the initial sort.
Verify that the results, though nondeterministic, are valid topological orderings.

Types

type CLI

type CLI struct {
	// contains filtered or unexported fields
}

CLI wrapper for *flag.FlagSet

func (CLI) DisplayHelp

func (m CLI) DisplayHelp()

DisplayHelp prints help on command line for bits module

func (CLI) ExecCmd

func (m CLI) ExecCmd(args []string)

ExecCmd run bit-count from CLI

type Hyperlinks

type Hyperlinks []string

Hyperlinks Interface to sort hrefs after ignoring http//: or https://

func (Hyperlinks) Len

func (a Hyperlinks) Len() int

Len of a sort interface

func (Hyperlinks) Less

func (a Hyperlinks) Less(i, j int) bool

Less of https://www.google.com or http://www.google.com

func (Hyperlinks) Swap

func (a Hyperlinks) Swap(i, j int)

Swap two strings

type NodeType

type NodeType string

NodeType defined node of type "a" "img" "script" and css "link"

type NodeValue

type NodeValue string

NodeValue values of link property

const (
	// Href link reference node type is <a
	Href NodeValue = "href"
	// Src <img src=”/> element
	Src = "src"
)