spiders

Documentation

Index

• Constants
• type Spider
  • func (this *Spider) MakeStartRequests() []*http.Request

Constants

const (
	BASE_PARSE_NAME     = "base"
	SPIDERS_STATUS_INIT = iota
	SPIDERS_STATUS_RUNNING
	SPIDERS_STATUS_STOP
	SPIDERS_BASIC_COOKIE
)

Types

type Spider

type Spider struct {
	Name      string
	StartUrls []string
	ParseMap  map[string]func(response *http.Response) ([]*http.Request, error) // defined you own parse function to scrapy response,make sure it return []response
}

what a spider do * make start request * define basic parse func

func (*Spider) MakeStartRequests

func (this *Spider) MakeStartRequests() []*http.Request