sites

Documentation

Index

• type Site
• type Traffic
  • func New() (t *Traffic)
  • func (t *Traffic) Done() (done <-chan struct{})
  • func (t *Traffic) Feed(urls []*url.URL, parent *url.URL, depth int)
  • func (t *Traffic) Processor(crawl func(s Site), parallel int) (done <-chan struct{})

Types

type Site

type Site struct {
	URL    *url.URL
	Parent *url.URL
	Depth  int
}

Site represents what travels: an URL which may have a Parent URL, and a Depth.

type Traffic

type Traffic struct {
	// contains filtered or unexported fields
}

Traffic goes around inside a circular Site pipe network, e. g. a crawling Crawler.

func New

func New() (t *Traffic)

New returns a new and operational Traffic processor.

func (*Traffic) Done

func (t *Traffic) Done() (done <-chan struct{})

Done returns a channel which will be signalled and closed when traffic has subsided and nothing is left to be processed and consequently all goroutines have terminated.

Note: Done() here is a convenience method.
It is well known from the "context" package.
Just: no need to use it as `Processor` returns same.

func (*Traffic) Feed

func (t *Traffic) Feed(urls []*url.URL, parent *url.URL, depth int)

Feed registers new entries.

func (*Traffic) Processor

func (t *Traffic) Processor(crawl func(s Site), parallel int) (done <-chan struct{})

Processor builds the site traffic processing network; it is cirular if crawl uses Feed to provide feedback.

returned is a channel which will be signalled and closed when traffic has subsided and nothing is left to be processed and consequently all goroutines have terminated - as is from `Done()`.