spiders

Documentation

Index

• Constants
• type Spider
  • func (this *Spider) MakeStartRequests() []*http.Request

Constants

const (
	BASE_PARSE_NAME     = "base"
	SPIDERS_STATUS_INIT = iota
	SPIDERS_STATUS_RUNNING
	SPIDERS_STATUS_STOP
	SPIDERS_BASIC_COOKIE
)

Types

type Spider

type Spider struct {
	BeforeMethod  func()
	InitStartUrls func()
	AfterMethod   func()
	Name          string
	StartUrls     []string
	ExtData       map[string]interface{}
	ParseMap      map[string]func(response *http.Response) ([]*http.Request, error) // defined you own parse function to scrapy response,make sure it return []response
}

what a spider do * make start request * define basic parse func

func (*Spider) MakeStartRequests

func (this *Spider) MakeStartRequests() []*http.Request