urllib

README

urllib

urllib is an libs help you to curl remote url.

How to use?

GET

you can use Get to crawl data.

import "github.com/GiterLab/goots/urllib"

str, err := urllib.Get("http://tobyzxj.me/").String()
if err != nil {
    	// error
}
fmt.Println(str)

POST

POST data to remote url

req := urllib.Post("http://tobyzxj.me/")
req.Param("username","tobyzxj")
req.Param("password","123456")
str, err := req.String()
if err != nil {
    	// error
}
fmt.Println(str)

Set timeout

The default timeout is 60 seconds, function prototype:

SetTimeout(connectTimeout, readWriteTimeout time.Duration)

Exmaple:

// GET
urllib.Get("http://tobyzxj.me/").SetTimeout(100 * time.Second, 30 * time.Second)

// POST
urllib.Post("http://tobyzxj.me/").SetTimeout(100 * time.Second, 30 * time.Second)

Debug

If you want to debug the request info, set the debug on

urllib.Get("http://tobyzxj.me/").Debug(true)

Set HTTP Basic Auth

str, err := Get("http://tobyzxj.me/").SetBasicAuth("user", "passwd").String()
if err != nil {
    	// error
}
fmt.Println(str)

Set HTTPS

If request url is https, You can set the client support TSL:

urllib.SetTLSClientConfig(&tls.Config{InsecureSkipVerify: true})

More info about the tls.Config please visit http://golang.org/pkg/crypto/tls/#Config

Set HTTP Version

some servers need to specify the protocol version of HTTP

urllib.Get("http://tobyzxj.me/").SetProtocolVersion("HTTP/1.1")

Set Cookie

some http request need setcookie. So set it like this:

cookie := &http.Cookie{}
cookie.Name = "username"
cookie.Value  = "tobyzxj"
urllib.Get("http://tobyzxj.me/").SetCookie(cookie)

Upload file

urllib support mutil file upload, use req.PostFile()

req := urllib.Post("http://tobyzxj.me/")
req.Param("username","tobyzxj")
req.PostFile("uploadfile1", "urllib.pdf")
str, err := req.String()
if err != nil {
    	// error
}
fmt.Println(str)

See godoc for further documentation and examples.

• godoc.org/github.com/GiterLab/goots/urllib

Documentation

Overview

Package urllib is a httplib for ots2

Index

• func SetDefaultSetting(setting HttpSettings)
• func TimeoutDialer(cTimeout time.Duration, rwTimeout time.Duration) func(net, addr string) (c net.Conn, err error)
• type HttpRequest
  • func Delete(url string) *HttpRequest
  • func Get(url string) *HttpRequest
  • func Head(url string) *HttpRequest
  • func Post(url string) *HttpRequest
  • func Put(url string) *HttpRequest
  • func (b *HttpRequest) Body(data interface{}) *HttpRequest
  • func (b *HttpRequest) Bytes() ([]byte, error)
  • func (b *HttpRequest) Debug(isdebug bool) *HttpRequest
  • func (b *HttpRequest) DumpBody(isdump bool) *HttpRequest
  • func (b *HttpRequest) DumpRequest() []byte
  • func (b *HttpRequest) Header(key, value string) *HttpRequest
  • func (b *HttpRequest) JsonBody(obj interface{}) (*HttpRequest, error)
  • func (b *HttpRequest) Param(key, value string) *HttpRequest
  • func (b *HttpRequest) PostFile(formname, filename string) *HttpRequest
  • func (b *HttpRequest) Response() (*http.Response, error)
  • func (b *HttpRequest) SendOut() (*http.Response, error)
  • func (b *HttpRequest) SetBasicAuth(username, password string) *HttpRequest
  • func (b *HttpRequest) SetCookie(cookie *http.Cookie) *HttpRequest
  • func (b *HttpRequest) SetEnableCookie(enable bool) *HttpRequest
  • func (b *HttpRequest) SetHost(host string) *HttpRequest
  • func (b *HttpRequest) SetProtocolVersion(vers string) *HttpRequest
  • func (b *HttpRequest) SetProxy(proxy func(*http.Request) (*url.URL, error)) *HttpRequest
  • func (b *HttpRequest) SetTLSClientConfig(config *tls.Config) *HttpRequest
  • func (b *HttpRequest) SetTimeout(connectTimeout, readWriteTimeout time.Duration) *HttpRequest
  • func (b *HttpRequest) SetTransport(transport http.RoundTripper) *HttpRequest
  • func (b *HttpRequest) SetUserAgent(useragent string) *HttpRequest
  • func (b *HttpRequest) Setting(setting HttpSettings) *HttpRequest
  • func (b *HttpRequest) String() (string, error)
  • func (b *HttpRequest) ToFile(filename string) error
  • func (b *HttpRequest) ToJson(v interface{}) error
  • func (b *HttpRequest) ToXml(v interface{}) error
• type HttpSettings
  • func GetDefaultSetting() *HttpSettings

Functions

func SetDefaultSetting

func SetDefaultSetting(setting HttpSettings)

Overwrite default settings

func TimeoutDialer

func TimeoutDialer(cTimeout time.Duration, rwTimeout time.Duration) func(net, addr string) (c net.Conn, err error)

TimeoutDialer returns functions of connection dialer with timeout settings for http.Transport Dial field.

Types

type HttpRequest

type HttpRequest struct {
	// contains filtered or unexported fields
}

HttpRequest provides more useful methods for requesting one url than http.Request.

func Delete

func Delete(url string) *HttpRequest

Delete returns *HttpRequest DELETE method.

func Get

func Get(url string) *HttpRequest

Get returns *HttpRequest with GET method.

func Head

func Head(url string) *HttpRequest

Head returns *HttpRequest with HEAD method.

func Post

func Post(url string) *HttpRequest

Post returns *HttpRequest with POST method.

func Put

func Put(url string) *HttpRequest

Put returns *HttpRequest with PUT method.

func (*HttpRequest) Body

func (b *HttpRequest) Body(data interface{}) *HttpRequest

Body adds request raw body. it supports string and []byte.

func (*HttpRequest) Bytes

func (b *HttpRequest) Bytes() ([]byte, error)

Bytes returns the body []byte in response. it calls Response inner.

func (*HttpRequest) Debug

func (b *HttpRequest) Debug(isdebug bool) *HttpRequest

Debug sets show debug or not when executing request.

func (*HttpRequest) DumpBody

func (b *HttpRequest) DumpBody(isdump bool) *HttpRequest

Dump Body.

func (*HttpRequest) DumpRequest

func (b *HttpRequest) DumpRequest() []byte

return the DumpRequest

func (*HttpRequest) Header

func (b *HttpRequest) Header(key, value string) *HttpRequest

Header add header item string in request.

func (*HttpRequest) JsonBody

func (b *HttpRequest) JsonBody(obj interface{}) (*HttpRequest, error)

JsonBody adds request raw body encoding by JSON.

func (*HttpRequest) Param

func (b *HttpRequest) Param(key, value string) *HttpRequest

Param adds query param in to request. params build query string as ?key1=value1&key2=value2...

func (*HttpRequest) PostFile

func (b *HttpRequest) PostFile(formname, filename string) *HttpRequest

func (*HttpRequest) Response

func (b *HttpRequest) Response() (*http.Response, error)

Response executes request client gets response mannually.

func (*HttpRequest) SendOut

func (b *HttpRequest) SendOut() (*http.Response, error)

func (*HttpRequest) SetBasicAuth

func (b *HttpRequest) SetBasicAuth(username, password string) *HttpRequest

SetBasicAuth sets the request's Authorization header to use HTTP Basic Authentication with the provided username and password.

func (*HttpRequest) SetCookie

func (b *HttpRequest) SetCookie(cookie *http.Cookie) *HttpRequest

SetCookie add cookie into request.

func (*HttpRequest) SetEnableCookie

func (b *HttpRequest) SetEnableCookie(enable bool) *HttpRequest

SetEnableCookie sets enable/disable cookiejar

func (*HttpRequest) SetHost

func (b *HttpRequest) SetHost(host string) *HttpRequest

Set HOST

func (*HttpRequest) SetProtocolVersion

func (b *HttpRequest) SetProtocolVersion(vers string) *HttpRequest

Set the protocol version for incoming requests. Client requests always use HTTP/1.1.

func (*HttpRequest) SetProxy

func (b *HttpRequest) SetProxy(proxy func(*http.Request) (*url.URL, error)) *HttpRequest

Set http proxy example:

func(req *http.Request) (*url.URL, error) {
	u, _ := url.ParseRequestURI("http://127.0.0.1:8118")
	return u, nil
}

func (*HttpRequest) SetTLSClientConfig

func (b *HttpRequest) SetTLSClientConfig(config *tls.Config) *HttpRequest

SetTLSClientConfig sets tls connection configurations if visiting https url.

func (*HttpRequest) SetTimeout

func (b *HttpRequest) SetTimeout(connectTimeout, readWriteTimeout time.Duration) *HttpRequest

SetTimeout sets connect time out and read-write time out for Request.

func (*HttpRequest) SetTransport

func (b *HttpRequest) SetTransport(transport http.RoundTripper) *HttpRequest

Set transport to

func (*HttpRequest) SetUserAgent

func (b *HttpRequest) SetUserAgent(useragent string) *HttpRequest

SetUserAgent sets User-Agent header field

func (*HttpRequest) Setting

func (b *HttpRequest) Setting(setting HttpSettings) *HttpRequest

Change request settings

func (*HttpRequest) String

func (b *HttpRequest) String() (string, error)

String returns the body string in response. it calls Response inner.

func (*HttpRequest) ToFile

func (b *HttpRequest) ToFile(filename string) error

ToFile saves the body data in response to one file. it calls Response inner.

func (*HttpRequest) ToJson

func (b *HttpRequest) ToJson(v interface{}) error

ToJson returns the map that marshals from the body bytes as json in response . it calls Response inner.

func (*HttpRequest) ToXml

func (b *HttpRequest) ToXml(v interface{}) error

ToXml returns the map that marshals from the body bytes as xml in response . it calls Response inner.

type HttpSettings

type HttpSettings struct {
	ShowDebug        bool
	UserAgent        string
	ConnectTimeout   time.Duration
	ReadWriteTimeout time.Duration
	TlsClientConfig  *tls.Config
	Proxy            func(*http.Request) (*url.URL, error)
	Transport        http.RoundTripper
	EnableCookie     bool
	Gzip             bool
	DumpBody         bool
}

HttpSettings

func GetDefaultSetting

func GetDefaultSetting() *HttpSettings

Get current default settings