downloader

Documentation

Index

• type HttpDownloader
  • func NewHttpDownloader(ct string) *HttpDownloader
  • func (self *HttpDownloader) Download(req *robot.Request) *robot.Page

Types

type HttpDownloader

type HttpDownloader struct {
	//如果网页中存在不完整的content－type 则默认
	DefaultContentType string
}

HTTP下载器 The HttpDownloader download page by package net/http. The "html" content is contained in dom parser of package goquery. The "json" content is saved. The "jsonp" content is modified to json. The "text" content will save body plain text only. The page result is saved in Page.

func NewHttpDownloader

func NewHttpDownloader(ct string) *HttpDownloader

func (*HttpDownloader) Download

func (self *HttpDownloader) Download(req *robot.Request) *robot.Page