requests

package module
v0.0.0-...-71a69f9 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 17, 2024 License: GPL-3.0 Imports: 37 Imported by: 15

README

Requests - A next-generation HTTP client for Golang.

Innovative Features

gospider007/requests Other Request Libraries
Unlimited chained proxy Not supported
HTTP/3 fingerprint spoofing protection Not supported
Arbitrary closure of underlying connections Not supported
Genuine request-level proxy settings Not supported
Unique transport layer management mechanism, fully unifying HTTP/1, HTTP/2, HTTP/3, WebSocket, and SSE protocol handling Not supported

Features

Benchmark

gospider007/requests > imroc/req > go-resty > wangluozhe/requests > curl_cffi > httpx > psf/requests

Supported Go Versions

Recommended to use go1.21.3 and above. Initially Requests started supporting go modules

Installation

go get github.com/gospider007/requests

Usage

import "github.com/gospider007/requests"

Quickly Send Requests

package main

import (
	"log"
	"time"
	"github.com/gospider007/requests"
)

func main() {
    resp, err := requests.Get(nil, "http://httpbin.org/anything")
    if err != nil {
      log.Panic(err)
    }
    log.Print(resp.Text())    // Get content and parse as string
    log.Print(resp.Content()) // Get content as bytes
    log.Print(resp.Json())    // Get content and parse as gjson JSON
    log.Print(resp.Html())    // Get content and parse as goquery DOM
    log.Print(resp.Cookies()) // Get cookies
}

Contributing

If you have a bug report or feature request, you can open an issue

Contact

If you have questions, feel free to reach out to us in the following ways:

  • QQ Group (Chinese): 939111384 -
  • WeChat (Chinese): gospider007

Sponsors

If you like and it really helps you, feel free to reward me with a cup of coffee, and don't forget to mention your github id.


Wechat
Alipay

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var ErrUseLastResponse = http.ErrUseLastResponse

Functions

func CreateReqCtx 

func CreateReqCtx(ctx context.Context, option *RequestOption) context.Context

func NewDialer 

func NewDialer(option DialOption) *net.Dialer

func NewJar 

func NewJar() *jar

new cookies jar

func NewRequestWithContext 

func NewRequestWithContext(ctx context.Context, method string, u *url.URL, body io.Reader) (*http.Request, error)

Types

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Connection Management

func NewClient 

func NewClient(preCtx context.Context, options ...ClientOption) (*Client, error)

New Connection Management

func (*Client) ClearCookies 

func (obj *Client) ClearCookies()

clear cookies

func (*Client) Close 

func (obj *Client) Close()

Close the client and cannot be used again after shutdown

func (*Client) CloseConns 

func (obj *Client) CloseConns()

Close idle connections. If the connection is in use, wait until it ends before closing

func (*Client) Connect 

func (obj *Client) Connect(ctx context.Context, href string, options ...RequestOption) (*Response, error)

sends a Connect request and returns the response.

func (*Client) Delete 

func (obj *Client) Delete(ctx context.Context, href string, options ...RequestOption) (*Response, error)

sends a Delete request and returns the response.

func (*Client) ForceCloseConns 

func (obj *Client) ForceCloseConns()

Close the connection, even if it is in use, it will be closed

func (*Client) Get 

func (obj *Client) Get(ctx context.Context, href string, options ...RequestOption) (*Response, error)

sends a Get request and returns the response.

func (*Client) GetCookies 

func (obj *Client) GetCookies(href *url.URL) Cookies

get cookies

func (*Client) Head 

func (obj *Client) Head(ctx context.Context, href string, options ...RequestOption) (*Response, error)

sends a Head request and returns the response.

func (*Client) Options 

func (obj *Client) Options(ctx context.Context, href string, options ...RequestOption) (*Response, error)

sends a Options request and returns the response.

func (*Client) Patch 

func (obj *Client) Patch(ctx context.Context, href string, options ...RequestOption) (*Response, error)

sends a Patch request and returns the response.

func (*Client) Post 

func (obj *Client) Post(ctx context.Context, href string, options ...RequestOption) (*Response, error)

sends a Post request and returns the response.

func (*Client) Put 

func (obj *Client) Put(ctx context.Context, href string, options ...RequestOption) (*Response, error)

sends a Put request and returns the response.

func (*Client) Request 

func (obj *Client) Request(ctx context.Context, method string, href string, options ...RequestOption) (response *Response, err error)

Define a function named Request that takes in four parameters:

func (*Client) SetCookies 

func (obj *Client) SetCookies(href *url.URL, cookies ...any) error

set cookies

func (*Client) SetGetProxy 

func (obj *Client) SetGetProxy(getProxy func(ctx context.Context, url *url.URL) (string, error))

Modify the proxy method of the client

func (*Client) SetGetProxys 

func (obj *Client) SetGetProxys(getProxys func(ctx context.Context, url *url.URL) ([]string, error))

Modify the proxy method of the client

func (*Client) SetProxy 

func (obj *Client) SetProxy(proxyUrl string) (err error)

Modifying the client's proxy

func (*Client) SetProxys 

func (obj *Client) SetProxys(proxyUrls []string) (err error)

Modifying the client's proxy

func (*Client) Trace 

func (obj *Client) Trace(ctx context.Context, href string, options ...RequestOption) (*Response, error)

sends a Trace request and returns the response.

type ClientOption 

type ClientOption struct {
	Logger                func(Log)                                                                             //debuggable
	H3                    bool                                                                                  //开启http3
	OrderHeaders          []string                                                                              //order headers
	Ja3Spec               ja3.Ja3Spec                                                                           //custom ja3Spec,use ja3.CreateSpecWithStr or ja3.CreateSpecWithId create
	H2Ja3Spec             ja3.H2Ja3Spec                                                                         //h2 fingerprint
	Proxy                 string                                                                                //proxy,support https,http,socks5
	Proxys                []string                                                                              //proxy list,support https,http,socks5
	ForceHttp1            bool                                                                                  //force  use http1 send requests
	Ja3                   bool                                                                                  //enable ja3 fingerprint
	DisCookie             bool                                                                                  //disable cookies
	DisDecode             bool                                                                                  //disable auto decode
	DisUnZip              bool                                                                                  //disable auto zip decode
	DisAlive              bool                                                                                  //disable  keepalive
	Bar                   bool                                                                                  ////enable bar display
	OptionCallBack        func(ctx context.Context, option *RequestOption) error                                //option callback,if error is returnd, break request
	ResultCallBack        func(ctx context.Context, option *RequestOption, response *Response) error            //result callback,if error is returnd,next errCallback
	ErrCallBack           func(ctx context.Context, option *RequestOption, response *Response, err error) error //error callback,if error is returnd,break request
	RequestCallBack       func(ctx context.Context, request *http.Request, response *http.Response) error       //request and response callback,if error is returnd,reponse is error
	MaxRetries            int                                                                                   //try num
	MaxRedirect           int                                                                                   //redirect num ,<0 no redirect,==0 no limit
	Headers               any                                                                                   //default headers
	Timeout               time.Duration                                                                         //request timeout
	ResponseHeaderTimeout time.Duration                                                                         //ResponseHeaderTimeout ,default:300
	TlsHandshakeTimeout   time.Duration                                                                         //tls timeout,default:15
	UserAgent             string                                                                                //headers User-Agent value
	//other option
	GetProxy    func(ctx context.Context, url *url.URL) (string, error)   //proxy callback:support https,http,socks5 proxy
	GetProxys   func(ctx context.Context, url *url.URL) ([]string, error) //proxys callback:support https,http,socks5 proxy
	GetAddrType func(host string) gtls.AddrType

	//network card ip
	DialTimeout time.Duration //dial tcp timeout,default:15
	KeepAlive   time.Duration //keepalive,default:30
	LocalAddr   *net.TCPAddr
	Dns         *net.UDPAddr  //dns
	AddrType    gtls.AddrType //dns parse addr type
	Jar         Jar           //custom cookies
	TlsConfig   *tls.Config
	UtlsConfig  *utls.Config
}

Connection Management Options

type Conn 

type Conn interface {
	CloseWithError(err error) error
	DoRequest(*http.Request, []string) (*http.Response, error)
}

type Cookies 

type Cookies []*http.Cookie

cookies

func ReadCookies 

func ReadCookies(val any) (Cookies, error)

read cookies or parse cookies,support json,map,[]string,http.Header,string

func (Cookies) Get 

func (obj Cookies) Get(name string) *http.Cookie

get cookie by name

func (Cookies) GetVal 

func (obj Cookies) GetVal(name string) string

get cookie value by name,return string

func (Cookies) GetVals 

func (obj Cookies) GetVals(name string) []string

get cookie values by name, return []string

func (Cookies) Gets 

func (obj Cookies) Gets(name string) Cookies

get cookies by name

func (Cookies) String 

func (obj Cookies) String() string

return cookies with string,join with '; '

type DialClient 

type DialClient struct {
	// contains filtered or unexported fields
}

func NewDail 

func NewDail(option DialOption) *DialClient

func (*DialClient) DialContext 

func (obj *DialClient) DialContext(ctx context.Context, ctxData *RequestOption, network string, addr string) (net.Conn, error)

func (*DialClient) DialProxyContext 

func (obj *DialClient) DialProxyContext(ctx context.Context, ctxData *RequestOption, network string, proxyTlsConfig *tls.Config, proxyUrls ...*url.URL) (net.Conn, error)

func (*DialClient) ProxyDialContext 

func (obj *DialClient) ProxyDialContext(ctx context.Context, ctxData *RequestOption, network string, addr string) (net.Conn, error)

func (*DialClient) Socks5Proxy 

func (obj *DialClient) Socks5Proxy(ctx context.Context, ctxData *RequestOption, network string, proxyUrl *url.URL, remoteUrl *url.URL) (conn net.Conn, err error)

type DialOption 

type DialOption struct {
	DialTimeout time.Duration
	KeepAlive   time.Duration
	LocalAddr   *net.TCPAddr  //network card ip
	AddrType    gtls.AddrType //first ip type
	Dns         *net.UDPAddr
	GetAddrType func(host string) gtls.AddrType
}

type Event 

type Event struct {
	Data    string //data
	Event   string //event
	Id      string //id
	Retry   int    //retry num
	Comment string //comment info
}

type File 

type File struct {
	FileName    string
	ContentType string
	Content     any
}

Upload files with form-data,

type Jar 

type Jar interface {
	ClearCookies()
	GetCookies(u *url.URL) Cookies
	SetCookies(*url.URL, Cookies)
}

type Log 

type Log struct {
	Id   string  `json:"id"`
	Type LogType `json:"type"`
	Time time.Time
	Msg  any `json:"msg"`
}

type LogType 

type LogType string
const (
	LogType_DNSLookup LogType = "DNSLookup"

	LogType_TCPConnect LogType = "TCPConnect"

	LogType_TLSHandshake LogType = "TLSHandshake"

	LogType_ProxyDNSLookup LogType = "ProxyDNSLookup"

	LogType_ProxyTCPConnect LogType = "ProxyTCPConnect"

	LogType_ProxyTLSHandshake LogType = "ProxyTLSHandshake"

	LogType_ProxyConnectRemote LogType = "ProxyConnectRemote"

	LogType_ResponseHeader LogType = "ResponseHeader"

	LogType_ResponseBody LogType = "ResponseBody"
)

type OrderMap 

type OrderMap struct {
	// contains filtered or unexported fields
}

func NewOrderMap 

func NewOrderMap() *OrderMap

func (*OrderMap) Del 

func (obj *OrderMap) Del(key string)

func (*OrderMap) Keys 

func (obj *OrderMap) Keys() []string

func (*OrderMap) MarshalJSON 

func (obj *OrderMap) MarshalJSON() ([]byte, error)

func (*OrderMap) Set 

func (obj *OrderMap) Set(key string, val any)

type RequestOption 

type RequestOption struct {
	Logger          func(Log)                                                                             //debuggable
	H3              bool                                                                                  //开启http3
	OrderHeaders    []string                                                                              //order headers
	Ja3Spec         ja3.Ja3Spec                                                                           //custom ja3Spec,use ja3.CreateSpecWithStr or ja3.CreateSpecWithId create
	H2Ja3Spec       ja3.H2Ja3Spec                                                                         //custom h2 fingerprint
	Proxy           string                                                                                //proxy,support http,https,socks5,example：http://127.0.0.1:7005
	Proxys          []string                                                                              //proxy list,support http,https,socks5,example：http://127.0.0.1:7005
	ForceHttp1      bool                                                                                  //force  use http1 send requests
	Ja3             bool                                                                                  //enable ja3 fingerprint
	DisCookie       bool                                                                                  //disable cookies,not use cookies
	DisDecode       bool                                                                                  //disable auto decode
	DisUnZip        bool                                                                                  //disable auto zip decode
	Bar             bool                                                                                  //enable bar display
	OptionCallBack  func(ctx context.Context, option *RequestOption) error                                //option callback,if error is returnd, break request
	ResultCallBack  func(ctx context.Context, option *RequestOption, response *Response) error            //result callback,if error is returnd,next errCallback
	ErrCallBack     func(ctx context.Context, option *RequestOption, response *Response, err error) error //error callback,if error is returnd,break request
	RequestCallBack func(ctx context.Context, request *http.Request, response *http.Response) error       //request and response callback,if error is returnd,reponse is error

	MaxRetries            int           //try num
	MaxRedirect           int           //redirect num ,<0 no redirect,==0 no limit
	Headers               any           //request headers：json,map，header
	Timeout               time.Duration //request timeout
	ResponseHeaderTimeout time.Duration //ResponseHeaderTimeout ,default:300
	TlsHandshakeTimeout   time.Duration
	UserAgent             string //headers User-Agent value

	GetProxy    func(ctx context.Context, url *url.URL) (string, error)   //proxy callback:support https,http,socks5 proxy
	GetProxys   func(ctx context.Context, url *url.URL) ([]string, error) //proxys callback:support https,http,socks5 proxy
	GetAddrType func(host string) gtls.AddrType

	//network card ip
	DialTimeout time.Duration //dial tcp timeout,default:15
	KeepAlive   time.Duration //keepalive,default:30
	LocalAddr   *net.TCPAddr
	Dns         *net.UDPAddr  //dns
	AddrType    gtls.AddrType //dns parse addr type                                             //tls timeout,default:15
	Jar         Jar           //custom cookies
	TlsConfig   *tls.Config
	UtlsConfig  *utls.Config

	// other option
	Method      string //method
	Url         *url.URL
	Host        string
	Referer     string //set headers referer value
	ContentType string //headers Content-Type value
	Cookies     any    // cookies,support :json,map,str，http.Header

	Params any //url params，join url query,json,map
	Json   any //send application/json,support io.Reader,：string,[]bytes,json,map
	Data   any //send application/x-www-form-urlencoded, support io.Reader, string,[]bytes,json,map
	Form   any //send multipart/form-data,file upload,support io.Reader, json,map
	Text   any //send text/xml,support: io.Reader, string,[]bytes,json,map
	Body   any //not setting context-type,support io.Reader, string,[]bytes,json,map

	Stream   bool             //disable auto read
	WsOption websocket.Option //websocket option
	DisProxy bool             //force disable proxy
	// contains filtered or unexported fields
}

Options for sending requests

func GetRequestOption 

func GetRequestOption(ctx context.Context) *RequestOption

func (*RequestOption) Client 

func (obj *RequestOption) Client() *Client

type Response 

type Response struct {
	// contains filtered or unexported fields
}

func Connect 

func Connect(ctx context.Context, href string, options ...RequestOption) (resp *Response, err error)

sends a Connect request and returns the response.

func Delete 

func Delete(ctx context.Context, href string, options ...RequestOption) (resp *Response, err error)

sends a Delete request and returns the response.

func Get 

func Get(ctx context.Context, href string, options ...RequestOption) (resp *Response, err error)

sends a GET request and returns the response. 

func Head(ctx context.Context, href string, options ...RequestOption) (resp *Response, err error)

sends a Head request and returns the response.

func Options 

func Options(ctx context.Context, href string, options ...RequestOption) (resp *Response, err error)

sends a Options request and returns the response.

func Patch 

func Patch(ctx context.Context, href string, options ...RequestOption) (resp *Response, err error)

sends a Patch request and returns the response.

func Post 

func Post(ctx context.Context, href string, options ...RequestOption) (resp *Response, err error)

sends a Post request and returns the response.

func Put 

func Put(ctx context.Context, href string, options ...RequestOption) (resp *Response, err error)

sends a Put request and returns the response.

func Request 

func Request(ctx context.Context, method string, href string, options ...RequestOption) (resp *Response, err error)

Define a function named Request that takes in four parameters:

func Trace 

func Trace(ctx context.Context, href string, options ...RequestOption) (resp *Response, err error)

sends a Trace request and returns the response.

func (*Response) Body 

func (obj *Response) Body() io.ReadCloser

func (*Response) CloseBody 

func (obj *Response) CloseBody()

close body

func (*Response) CloseConn 

func (obj *Response) CloseConn()

safe close conn

func (*Response) Content 

func (obj *Response) Content() []byte

return content with []byte

func (*Response) ContentEncoding 

func (obj *Response) ContentEncoding() string

return content encoding

func (*Response) ContentLength 

func (obj *Response) ContentLength() int64

return content length

func (*Response) ContentType 

func (obj *Response) ContentType() string

return content type

func (*Response) Cookies 

func (obj *Response) Cookies() Cookies

return response cookies

func (*Response) Decode 

func (obj *Response) Decode(encoding string)

change decoding with content

func (*Response) ForceCloseConn 

func (obj *Response) ForceCloseConn()

force close conn

func (*Response) Headers 

func (obj *Response) Headers() http.Header

return response headers

func (*Response) Html 

func (obj *Response) Html() *bs4.Client

return content with parse html

func (*Response) IsNewConn 

func (obj *Response) IsNewConn() bool

conn is new conn

func (*Response) IsSSE 

func (obj *Response) IsSSE() bool

func (*Response) IsStream 

func (obj *Response) IsStream() bool

return true if response is stream

func (*Response) IsWebSocket 

func (obj *Response) IsWebSocket() bool

return true if response is other stream

func (*Response) Json 

func (obj *Response) Json(vals ...any) (*gson.Client, error)

return content with json and you can parse struct

func (*Response) Location 

func (obj *Response) Location() (*url.URL, error)

return URL redirected address

func (*Response) Map 

func (obj *Response) Map() (data map[string]any, err error)

return content with map[string]any

func (*Response) Proto 

func (obj *Response) Proto() string

return response Proto

func (*Response) Proxys 

func (obj *Response) Proxys() []*url.URL

conn proxy

func (*Response) ReadBody 

func (obj *Response) ReadBody() (err error)

read body

func (*Response) SSE 

func (obj *Response) SSE() *SSE

return SSE client

func (*Response) SetContent 

func (obj *Response) SetContent(val []byte)

set response content with []byte

func (*Response) Status 

func (obj *Response) Status() string

return response status

func (*Response) StatusCode 

func (obj *Response) StatusCode() int

return response status code

func (*Response) Text 

func (obj *Response) Text() string

return content with string

func (*Response) Url 

func (obj *Response) Url() *url.URL

return response url

func (*Response) WebSocket 

func (obj *Response) WebSocket() *websocket.Conn

return websocket client

type SSE 

type SSE struct {
	// contains filtered or unexported fields
}

func (*SSE) Close 

func (obj *SSE) Close()

close SSE

func (*SSE) Range 

func (obj *SSE) Range() iter.Seq2[Event, error]

func (*SSE) Recv 

func (obj *SSE) Recv() (Event, error)

recv SSE envent data

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.