scheduler

Documentation

Index

• func Init()
• func PauseRecover()
• func Stop()
• type Matrix
  • func AddMatrix(spiderName, spiderSubName string, maxPage int64) *Matrix
  • func (matrix *Matrix) CanStop() bool
  • func (matrix *Matrix) DoHistory(req *request.Request, ok bool) bool
  • func (matrix *Matrix) Free()
  • func (matrix *Matrix) Len() int
  • func (matrix *Matrix) Pull() (req *request.Request)
  • func (matrix *Matrix) Push(req *request.Request)
  • func (matrix *Matrix) TryFlushFailure()
  • func (matrix *Matrix) TryFlushSuccess()
  • func (matrix *Matrix) Use()
  • func (matrix *Matrix) Wait()

Functions

func Init

func Init()

func PauseRecover

func PauseRecover()

pause \ resume all crawling tasks

func Stop

func Stop()

Terminate the task

Types

type Matrix

type Matrix struct {
	sync.Mutex
	// contains filtered or unexported fields
}

a Spider instance's request matrix

func AddMatrix

func AddMatrix(spiderName, spiderSubName string, maxPage int64) *Matrix

register the resource queue

func (*Matrix) CanStop

func (matrix *Matrix) CanStop() bool

func (*Matrix) DoHistory

func (matrix *Matrix) DoHistory(req *request.Request, ok bool) bool

Returns whether the request was added as a new failure to the end of the queue

func (*Matrix) Free

func (matrix *Matrix) Free()

func (*Matrix) Len

func (matrix *Matrix) Len() int

func (*Matrix) Pull

func (matrix *Matrix) Pull() (req *request.Request)

From the queue to take out the request, there is no return to nil, concurrent security

func (*Matrix) Push

func (matrix *Matrix) Push(req *request.Request)

Add the request to the queue, concurrent security

func (*Matrix) TryFlushFailure

func (matrix *Matrix) TryFlushFailure()

Non-server mode to save the history of failure records

func (*Matrix) TryFlushSuccess

func (matrix *Matrix) TryFlushSuccess()

Non-server mode to save the history of success records

func (*Matrix) Use

func (matrix *Matrix) Use()

func (*Matrix) Wait

func (matrix *Matrix) Wait()

Wait for the request to be processed