Why Go
- Case Studies
  
  Common problems companies solve with Go
- Use Cases
  
  Stories about how and why companies use Go
- Security Policy
  
  How Go can help keep you secure by default
Learn
Docs
- Effective Go
  
  Tips for writing clear, performant, and idiomatic Go code
- Go User Manual
  
  A complete introduction to building software with Go
- Standard library
  
  Reference documentation for Go's standard library
- Release Notes
  
  Learn what's new in each Go release
Packages
Community
- Recorded Talks
  
  Videos from prior events
- Meetups
  
  Meet other local Go developers
- Conferences
  
  Learn and network with Go developers from around the world
- Go blog
  
  The Go project's official blog.
- Go project
  
  Get help and stay informed from Go
- Get connected

go_spider

package module

Go to main page

Versions in this module

v1

v1.0.2

Nov 3, 2019

Changes in this version

type TaskHandler

+ GapLimitMax int

+ GapLimitMin int

type TaskOpt

+ func TaskOptGapLimitRandom(min int, max int) TaskOpt

v1.0.1

Nov 1, 2019

v0

v0.1.0

Oct 31, 2019

Changes in this version

+ type GoSpider struct

+ func NewGoSpider() *GoSpider

+ func (g *GoSpider) AddTask(handler *TaskHandler)

+ func (g *GoSpider) Run()

+ type OnErrorCallback func(msg string, err error)

+ type OnQueryCallback func(url string, selection *goquery.Selection)

+ type OnRequestCallback func(url string, header *req.Header, param *req.Param, err error)

+ type OnResponseCallback func(url string, rsp *req.Resp, err error)

+ type TaskHandler struct

+ CachePath string

+ Domains []string

+ Entry string

+ GapLimit int

+ Headers *req.Header

+ Http *req.Req

+ IdleLimit int

+ Name string

+ Params *req.Param

+ Proxies []string

+ Queue chan string

+ QueueProcessNum int

+ QueueTotalNum int

+ SrcCharset string

+ WorkerLimit int

+ func NewTaskHandler(name string, entry string, opts ...TaskOpt) *TaskHandler

+ func (t *TaskHandler) Clone() *TaskHandler

+ func (t *TaskHandler) Handle()

+ func (t *TaskHandler) OnQuery(selector string, cb OnQueryCallback)

+ func (t *TaskHandler) OnRequest(cb OnRequestCallback)

+ func (t *TaskHandler) OnResponse(cb OnResponseCallback)

+ func (t *TaskHandler) Visit(url string)

+ type TaskOpt func(*TaskHandler)

+ func TaskOptCache(path string) TaskOpt

+ func TaskOptDomains(domains []string) TaskOpt

+ func TaskOptEnableCookie(b bool) TaskOpt

+ func TaskOptGapLimit(num int) TaskOpt

+ func TaskOptProxy(proxy []string) TaskOpt

+ func TaskOptSrcCharset(charset string) TaskOpt