crawler

Documentation

Index

• func Debug(state bool)
• type Fetcher
• type State
  • func New(URL string, f Fetcher) *State
  • func (state *State) Format() string

Functions

func Debug

func Debug(state bool)

Debug turns debug logging on or off.

Types

type Fetcher

type Fetcher interface {
	// Fetch returns the body of URL and
	// a slice of URLs found on that page.
	Fetch(url string) (body string, urls []string, err error)
}

Fetcher defines an interface that can fetch URLs.

type State

type State struct {
	BaseURL string

	Done bool

	Start  controlFunc
	Pause  controlFunc
	Resume controlFunc
	Wait   controlFunc
	Quit   controlFunc

	sync.Mutex
	// contains filtered or unexported fields
}

State is the current state of the crawler.

func New

func New(URL string, f Fetcher) *State

New takes a URL and a Fetcher to fetch URLs. It initializes the crawler's data structures and returns a set of closures that can be used to start, pause, resume, and quit crawling. It also provides a wait() function to ensure that we can wait for the process to complete if we so desire. See https://stackoverflow.com/questions/38798863/golang-pause-a-loop-in-a-goroutine-with-channels holding the crawl state in the State pointer passed in.

func (*State) Format

func (state *State) Format() string

Format formats the crawl tree as it stands and returns it.