links

Documentation

Index

• func GetListPagesNum(href string, out chan<- string) error
• type Links
  • func NewLinks() *Links
  • func (l *Links) Close()
  • func (l *Links) CrawlerCatLinks(url string, out chan<- string) error
  • func (l *Links) CrawlerCatListFromNode(url string, out chan<- []string) error
  • func (l *Links) CrawlerDetailPageFromNode(urls []string, out chan<- interface{})
  • func (l *Links) GetSeedURLS(u string) ([]string, error)
  • func (l *Links) LinksList(urls []string)
  • func (l *Links) ListURLS(urls []string, out chan<- string)
  • func (l *Links) Storages(in <-chan interface{}, done chan<- struct{})

Functions

func GetListPagesNum

func GetListPagesNum(href string, out chan<- string) error

Types

type Links

type Links struct {
	Cat string

	Wg *sync.WaitGroup
	// contains filtered or unexported fields
}

Links export Links to main Cralwer

func NewLinks

func NewLinks() *Links

NewLinks newlinks

func (*Links) Close

func (l *Links) Close()

Close release

func (*Links) CrawlerCatLinks

func (l *Links) CrawlerCatLinks(url string, out chan<- string) error

CrawlerCatLinks crawler a cat inclue subcat page

func (*Links) CrawlerCatListFromNode

func (l *Links) CrawlerCatListFromNode(url string, out chan<- []string) error

CrawlerCatListFromNode page

func (*Links) CrawlerDetailPageFromNode

func (l *Links) CrawlerDetailPageFromNode(urls []string, out chan<- interface{})

func (*Links) GetSeedURLS

func (l *Links) GetSeedURLS(u string) ([]string, error)

GetSeedURLS get init seeds.

func (*Links) LinksList

func (l *Links) LinksList(urls []string)

LinksList crawler list links from root seed []string array.

urls root seeds.
out chan  put real per ic url into out channal

func (*Links) ListURLS

func (l *Links) ListURLS(urls []string, out chan<- string)

ListURLS out channel for list page. first output channel

func (*Links) Storages

func (l *Links) Storages(in <-chan interface{}, done chan<- struct{})

Storages channel one the last channel close done channal for singal all channal done