goreq

package module
v0.0.0-...-240cfd7 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 13, 2020 License: Apache-2.0 Imports: 27 Imported by: 0

README

Goreq

goproxy.cn Go Test codecov

中文文档

易用于网页、API 环境下的 Golang HTTP Client 封装库。

让net/http为人类服务。

go get -u github.com/zhshch2002/goreq

Feature

  • Thread-safe | 线程安全
  • Auto Charset Decode | 自动解码
  • Easy to set proxy for each req | 便捷代理设置
  • Chain config request | 链式配置请求
  • Multipart post support
  • Parse HTML,JSON,XML | HTML、JSON、XML 解析
  • Middleware | 中间件
    • Cache | 缓存
    • Retry | 失败重试
    • Log | 日志
    • Random UserAgent | 随机 UA
    • Referer | 填充 Referer
    • Rate Delay and Parallelism limiter | 设置速率、延时、并发限制

Goreq 是线程安全的，意味着您无论在多线程还是单线程下开发，都无需改动代码。

Goreq 会自动处理网页编码，对于下载下来的网页，Goreq 会根据 HTTP 报头、内容推断编码并加以解码。而且您任可以访问原始的未解码内容。

Request

Goreq 是设计为一般情况下访问网页和 API 而设计的“便携”工具。

// `req`是一个请求`*goreq.Request`，是一个包装过的`net/http`下的 `*http.Request`
req := goreq.Get("https://httpbin.org/")

// 请求可以被链式配置，如果配置过程中出现错误，`req.Err`将不再为`nil`
req = goreq.Get("https://httpbin.org/get").AddParam("A","a")
if req.Err!=nil {
    panic(req.Err)
}
  • AddParam(k, v string)
  • AddParams(v map[string]string)
  • AddHeader(key, value string)
  • AddHeaders(v map[string]string)
  • AddCookie(c *http.Cookie)
  • SetUA(ua string)
  • SetBasicAuth(username, password string)
  • SetProxy(urladdr string)
  • SetTimeout(t time.Duration)
  • DisableRedirect()
  • SetCheckRedirect(fn func(req *http.Request, via []*http.Request) error)
  • Set request body data
    • SetBody(b io.Reader) basic setting
    • SetRawBody(b []byte)
    • SetFormBody(v map[string]string)
    • SetJsonBody(v interface{})
    • SetMultipartBody(data ...interface{}) Set a slice of FormField and FormFile struct as body data
  • Callback(fn func(resp *Response) Set a callback func run after req Do()

Response

一个“请求”需要被“执行”来获得响应。

resp:=goreq.Get("https://httpbin.org/get").AddParam("A","a").Do()

// 等效于执行了
resp:=goreq.DefaultClient.Do(goreq.Get("https://httpbin.org/get").AddParam("A","a"))

执行“请求”的是*goreq.Client，可以对其配置来添加中间件以实现扩展功能。

c:=goreq.NewClient()
c.Use(goreq.WithRandomUA()) // 添加一个自动随机浏览器 UA 的内置中间件
resp:=goreq.Get("https://httpbin.org/get").AddParam("A","a").SetClient(c).Do()
fmt.Println(resp.Txt())

*goreq.Response可以通过下述函数来获取响应数据。

  • Resp() (*Response, error) 获取响应本身以及网络请求错误。
  • Txt() (string, error) 自动处理完编码并解析为文本后的内容以及网络请求错误。
  • HTML() (*goquery.Document, error)
  • XML() (*xmlpath.Node, error)
  • BindXML(i interface{}) error
  • JSON() (gjson.Result, error)
  • BindJSON(i interface{}) error
  • Error() error 网络请求错误。（正常情况下为nil）

Middleware

package main

import (
	"fmt"
	"github.com/zhshch2002/goreq"
)

func main() {
	// you can config `goreq.DefaultClient.Use()` to set global middleware
	c := goreq.NewClient() // create a new client
	c.Use(req.WithRandomUA()) // Add a builtin middleware
	c.Use(func(client *goreq.Client, handler goreq.Handler) goreq.Handler { // Add another middleware
		return func(r *goreq.Request) *goreq.Response {
			fmt.Println("this is a middleware")
			r.Header.Set("req", "goreq")
			return handler(r)
		}
	})

	txt, err := goreq.Get("https://httpbin.org/get").SetClient(c).Do().Txt()
	fmt.Println(txt, err)
}
Builtin middleware
  • WithDebug()
  • WithCache(ca *cache.Cache) Cache of *Response by go-cache
  • WithRetry(maxTimes int, isRespOk func(*Response) set nil for isRespOk means no check func
  • WithProxy(p ...string) set a list of proxy or it will follow all_proxy https_proxy and http_proxy env
  • WithRefererFiller()
  • WithRandomUA()
  • Limiter | control Rate Delay and Parallelism
    • WithFilterLimiter(noneMatchAllow bool, opts ...*FilterLimiterOpinion)
    • WithDelayLimiter(eachSite bool, opts ...*DelayLimiterOpinion)
    • WithRateLimiter(eachSite bool, opts ...*RateLimiterOpinion)
    • WithParallelismLimiter(eachSite bool, opts ...*ParallelismLimiterOpinion)
失败重试 | WithRetry
package main

import (
	"fmt"
	"github.com/zhshch2002/goreq"
)

func main() {
	i := 0
	// 配置失败重试中间件，第二个参数函数用来检查是否为可接受的响应，传入 nil 使用默认函数。
	c := goreq.NewClient(goreq.WithRetry(10, func(resp *goreq.Response) bool {
		if i < 3 { // 为了演示模拟几次失败
			i += 1
			return false
		}
		return true
	}))
	fmt.Println(goreq.Get("https://httpbin.org/get").SetDebug(true).SetClient(c).Do().Text)
}

Output:

[Retry 1 times] got error on request https://httpbin.org/get <nil>
[Retry 2 times] got error on request https://httpbin.org/get <nil>
[Retry 3 times] got error on request https://httpbin.org/get <nil>
{
  "args": {},
  "headers": {
    "Accept-Encoding": "gzip",
    "Host": "httpbin.org",
    "User-Agent": "Go-http-client/2.0",
    "X-Amzn-Trace-Id": "Root=1-5efe9c40-bbf2d5a095e0f6d0c3aaf4c0"
  },
  "url": "https://httpbin.org/get"
}

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var Debug = false //TODO
View Source 
var DefaultClient = NewClient()
View Source 
var ReqRejectedErr = errors.New("request is rejected")

Functions 

func ModifyLink(url string) string

Types

type Client 

type Client struct {
	// contains filtered or unexported fields
}

func NewClient 

func NewClient(m ...Middleware) *Client

func (*Client) Do 

func (s *Client) Do(req *Request) *Response

func (*Client) Use 

func (s *Client) Use(m ...Middleware) *Client

type DelayLimiterOpinion 

type DelayLimiterOpinion struct {
	LimiterMatcher
	Delay       time.Duration
	RandomDelay time.Duration
	// contains filtered or unexported fields
}

type FilterLimiterOpinion 

type FilterLimiterOpinion struct {
	LimiterMatcher
	Allow bool
}

type FormField 

type FormField struct {
	Name, Value string
}

type FormFile 

type FormFile struct {
	FieldName, FileName, ContentType string
	File                             io.Reader
}

type Handler 

type Handler func(*Request) *Response

type LimitRuleAllow 

type LimitRuleAllow uint8
const (
	NotSet LimitRuleAllow = iota
	Allow
	Disallow
)

type LimiterMatcher 

type LimiterMatcher struct {
	Regexp, Glob string
	// contains filtered or unexported fields
}

func (*LimiterMatcher) Compile 

func (s *LimiterMatcher) Compile()

func (*LimiterMatcher) Match 

func (s *LimiterMatcher) Match(u *url.URL) bool

type Middleware 

type Middleware func(*Client, Handler) Handler

func WithCache 

func WithCache(ca *cache.Cache) Middleware

func WithDebug 

func WithDebug() Middleware

func WithDelayLimiter 

func WithDelayLimiter(eachSite bool, opts ...*DelayLimiterOpinion) Middleware

func WithFilterLimiter 

func WithFilterLimiter(noneMatchAllow bool, opts ...*FilterLimiterOpinion) Middleware

func WithParallelismLimiter 

func WithParallelismLimiter(eachSite bool, opts ...*ParallelismLimiterOpinion) Middleware

func WithProxy 

func WithProxy(p ...string) Middleware

func WithRandomUA 

func WithRandomUA() Middleware

func WithRateLimiter 

func WithRateLimiter(eachSite bool, opts ...*RateLimiterOpinion) Middleware

func WithRefererFiller 

func WithRefererFiller() Middleware

func WithRetry 

func WithRetry(maxTimes int, isRespOk func(*Response) bool) Middleware

type ParallelismLimiterOpinion 

type ParallelismLimiterOpinion struct {
	LimiterMatcher
	Parallelism int64
	// contains filtered or unexported fields
}

type RateLimiterOpinion 

type RateLimiterOpinion struct {
	LimiterMatcher
	Rate int64
	// contains filtered or unexported fields
}

type ReqError 

type ReqError struct {
	// contains filtered or unexported fields
}

type Request 

type Request struct {
	*http.Request
	// ProxyURL is the proxy address that handles the request
	ProxyURL string

	RespEncode string

	Writer io.Writer

	Debug bool

	Err error
	// contains filtered or unexported fields
}

Request is a object of HTTP request

func Connect 

func Connect(urladdr string) *Request

func Delete 

func Delete(urladdr string) *Request

func Get 

func Get(urladdr string) *Request
func Head(urladdr string) *Request

func NewRequest 

func NewRequest(method, urladdr string) *Request

func Options 

func Options(urladdr string) *Request

func Patch 

func Patch(urladdr string) *Request

func Post 

func Post(urladdr string) *Request

func Put 

func Put(urladdr string) *Request

func Trace 

func Trace(urladdr string) *Request

func (*Request) AddCookie 

func (s *Request) AddCookie(c *http.Cookie) *Request

AddCookie adds a cookie to the request.

func (*Request) AddCookies 

func (s *Request) AddCookies(cs ...*http.Cookie) *Request

AddCookies adds some cookie to the request at once.

func (*Request) AddHeader 

func (s *Request) AddHeader(key, value string) *Request

SetHeader sets the header entries associated with key to the single element value.

func (*Request) AddHeaders 

func (s *Request) AddHeaders(v map[string]string) *Request

func (*Request) AddParam 

func (s *Request) AddParam(k, v string) *Request

AddParam adds a query param of request url.

func (*Request) AddParams 

func (s *Request) AddParams(v map[string]string) *Request

func (*Request) DisableRedirect 

func (s *Request) DisableRedirect() *Request

func (*Request) Do 

func (s *Request) Do() *Response

func (*Request) SetBasicAuth 

func (s *Request) SetBasicAuth(username, password string) *Request

func (*Request) SetBody 

func (s *Request) SetBody(b io.Reader) *Request

func (*Request) SetCallback 

func (s *Request) SetCallback(fn func(resp *Response) *Response) *Request

func (*Request) SetCheckRedirect 

func (s *Request) SetCheckRedirect(fn func(req *http.Request, via []*http.Request) error) *Request

func (*Request) SetClient 

func (s *Request) SetClient(c *Client) *Request

func (*Request) SetDebug 

func (s *Request) SetDebug(d bool) *Request

func (*Request) SetFormBody 

func (s *Request) SetFormBody(v map[string]string) *Request

func (*Request) SetJsonBody 

func (s *Request) SetJsonBody(v interface{}) *Request

func (*Request) SetMultipartBody 

func (s *Request) SetMultipartBody(data ...interface{}) *Request

func (*Request) SetProxy 

func (s *Request) SetProxy(urladdr string) *Request

func (*Request) SetRawBody 

func (s *Request) SetRawBody(b []byte) *Request

func (*Request) SetTimeout 

func (s *Request) SetTimeout(t time.Duration) *Request

func (*Request) SetUA 

func (s *Request) SetUA(ua string) *Request

SetProxy sets user-agent url of request header.

func (*Request) String 

func (s *Request) String() string

type Response 

type Response struct {
	*http.Response
	// Body is the content of the Response
	Body         []byte
	NoDecodeBody []byte
	// Text is the content of the Response parsed as string
	Text string
	// Request is the Req object from goribot of the response.Tip: there is another Request attr come from *http.Response
	Req *Request
	Err error
}

Response is a object of HTTP response

func Do 

func Do(req *Request) *Response

func (*Response) BindJSON 

func (s *Response) BindJSON(i interface{}) error

func (*Response) BindXML 

func (s *Response) BindXML(i interface{}) error

func (*Response) DecodeAndParse 

func (s *Response) DecodeAndParse() error

DecodeAndParas decodes the body to text and try to parse it to html or json.

func (*Response) Error 

func (s *Response) Error() error

func (*Response) HTML 

func (s *Response) HTML() (*goquery.Document, error)

func (*Response) IsHTML 

func (s *Response) IsHTML() bool

func (*Response) IsJSON 

func (s *Response) IsJSON() bool

func (*Response) JSON 

func (s *Response) JSON() (gjson.Result, error)

func (*Response) Resp 

func (s *Response) Resp() (*Response, error)

func (*Response) Txt 

func (s *Response) Txt() (string, error)

func (*Response) XML 

func (s *Response) XML() (*xmlpath.Node, error)

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.