grequests

package module
v0.0.0-...-77495ed Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 22, 2015 License: Apache-2.0 Imports: 18 Imported by: 0

README

GRequests

A Go "clone" of the great and famous Requests library

Build Status GoDoc Coverage Status

License

GRequests is licensed under the Apache License, Version 2.0. See LICENSE for the full license text

Features

  • Asynchronous and synchronous functionality built in
  • Works with every version of Go from 1.3
  • Responses can be serialized into JSON and XML
  • Easy file uploads
  • Easy file downloads
  • Support for the following HTTP verbs GET, HEAD, POST, PUT, DELETE, PATCH, OPTIONS

Install

go get -u github.com/levigross/grequests

Usage

import "github.com/levigross/grequests"

Basic Examples

Basic GET request:

resp, err := grequests.Get("http://httpbin.org/get", nil)
// You can modify the request by passing an optional RequestOptions struct

if err != nil {
	log.Fatalln("Unable to make request: ", err)
}

fmt.Println(resp.String())
// {
//   "args": {},
//   "headers": {
//     "Accept": "*/*",
//     "Host": "httpbin.org",

If an error occurs all of the other properties and methods of a Response will be nil

Quirks

Request Quirks

When passing parameters to be added to a URL, if the URL has existing parameters that contradict with what has been passed within Params – Params will be the "source of authority" and overwrite the contradicting URL parameter.

Lets see how it works...

ro := &RequestOptions{
	Params: map[string]string{"Hello": "Goodbye"},
}
Get("http://httpbin.org/get?Hello=World", ro)
// The URL is now http://httpbin.org/get?Hello=Goodbye

Response Quirks

Order matters! This is because grequests.Response is implemented as an io.ReadCloser which proxies the http.Response.Body io.ReadCloser interface. It also includes an internal buffer for use in Response.String() and Response.Bytes().

Here are a list of methods that consume the http.Response.Body io.ReadCloser interface.

  • Response.JSON
  • Response.XML
  • Response.DownloadToFile
  • Response.Close
  • Response.Read

The following methods make use of an internal byte buffer

  • Response.String
  • Response.Bytes

In the code below, once the file is downloaded – the Response struct no longer has access to the request bytes

response := Get("http://some-wonderful-file.txt", nil)

if err := response.DownloadToFile("randomFile"); err != nil {
	log.Println("Unable to download file: ", err)
}

// At this point the .String and .Bytes method will return empty responses

response.Bytes() == nil // true
response.String() == "" // true

But if we were to call response.Bytes() or response.String() first, every operation will succeed until the internal buffer is cleared:

response := Get("http://some-wonderful-file.txt", nil)

// This call to .Bytes caches the request bytes in an internal byte buffer – which can be used again and again until it is cleared
response.Bytes() == `file-bytes`
response.String() == "file-string"

// This will work because it will use the internal byte buffer
if err := resp.DownloadToFile("randomFile"); err != nil {
	log.Println("Unable to download file: ", err)
}

// Now if we clear the internal buffer....
response.ClearInternalBuffer()

// At this point the .String and .Bytes method will return empty responses

response.Bytes() == nil // true
response.String() == "" // true

Documentation

Overview

Package grequests implements a friendly API over Go's existing net/http library

Example (AcceptInvalidTLSCert) 
package main

import (
	"log"

	"github.com/levigross/grequests"
)

func main() {
	ro := &grequests.RequestOptions{InsecureSkipVerify: true}
	resp, err := grequests.Get("https://www.pcwebshop.co.uk/", ro)

	if err != nil {
		log.Println("Unable to make request", err)
	}

	if resp.Ok != true {
		log.Println("Request did not return OK")
	}
}

Output:
Example (BasicAuth) 
package main

import (
	"log"

	"github.com/levigross/grequests"
)

func main() {
	ro := &grequests.RequestOptions{Auth: []string{"Levi", "Bot"}}
	resp, err := grequests.Get("http://httpbin.org/get", ro)
	// Not the usual JSON so copy and paste from below

	if err != nil {
		log.Println("Unable to make request", err)
	}

	if resp.Ok != true {
		log.Println("Request did not return OK")
	}
}

Output:
Example (BasicGet) 
package main

import (
	"log"

	"github.com/levigross/grequests"
)

func main() {
	// This is a very basic GET request
	resp, err := grequests.Get("http://httpbin.org/get", nil)

	if err != nil {
		log.Println(err)
	}

	if resp.Ok != true {
		log.Println("Request did not return OK")
	}

	log.Println(resp.String())
}

Output:
Example (BasicGetCustomHTTPClient) 
package main

import (
	"log"
	"net/http"

	"github.com/levigross/grequests"
)

func main() {
	// This is a very basic GET request
	resp, err := grequests.Get("http://httpbin.org/get", &grequests.RequestOptions{HTTPClient: http.DefaultClient})

	if err != nil {
		log.Println(err)
	}

	if resp.Ok != true {
		log.Println("Request did not return OK")
	}

	log.Println(resp.String())
}

Output:
Example (Cookies) 
package main

import (
	"log"
	"net/http"

	"github.com/levigross/grequests"
)

func main() {
	resp, err := grequests.Get("http://httpbin.org/cookies",
		&grequests.RequestOptions{
			Cookies: []http.Cookie{
				{
					Name:     "TestCookie",
					Value:    "Random Value",
					HttpOnly: true,
					Secure:   false,
				}, {
					Name:     "AnotherCookie",
					Value:    "Some Value",
					HttpOnly: true,
					Secure:   false,
				},
			},
		})

	if err != nil {
		log.Println("Unable to make request", err)
	}

	if resp.Ok != true {
		log.Println("Request did not return OK")
	}

	log.Println(resp.String())
}

Output:
Example (CustomHTTPHeader) 
package main

import (
	"log"

	"github.com/levigross/grequests"
)

func main() {
	ro := &grequests.RequestOptions{UserAgent: "LeviBot 0.1",
		Headers: map[string]string{"X-Wonderful-Header": "1"}}
	resp, err := grequests.Get("http://httpbin.org/get", ro)
	// Not the usual JSON so copy and paste from below

	if err != nil {
		log.Println("Unable to make request", err)
	}

	if resp.Ok != true {
		log.Println("Request did not return OK")
	}
}

Output:
Example (CustomUserAgent) 
package main

import (
	"log"

	"github.com/levigross/grequests"
)

func main() {
	ro := &grequests.RequestOptions{UserAgent: "LeviBot 0.1"}
	resp, err := grequests.Get("http://httpbin.org/get", ro)

	if err != nil {
		log.Fatal("Oops something went wrong: ", err)
	}

	if resp.Ok != true {
		log.Println("Request did not return OK")
	}

	log.Println(resp.String())
}

Output:
Example (DownloadFile) 
package main

import (
	"log"

	"github.com/levigross/grequests"
)

func main() {
	resp, err := grequests.Get("http://httpbin.org/get", nil)

	if err != nil {
		log.Println("Unable to make request", err)
	}

	if resp.Ok != true {
		log.Println("Request did not return OK")
	}

	if err := resp.DownloadToFile("randomFile"); err != nil {
		log.Println("Unable to download to file: ", err)
	}

	if err != nil {
		log.Println("Unable to download file", err)
	}

}

Output:
Example (Parse_XML) 
package main

import (
	"encoding/xml"
	"io"
	"log"

	"github.com/levigross/grequests"
)

func main() {
	type GetXMLSample struct {
		XMLName xml.Name `xml:"slideshow"`
		Title   string   `xml:"title,attr"`
		Date    string   `xml:"date,attr"`
		Author  string   `xml:"author,attr"`
		Slide   []struct {
			Type  string `xml:"type,attr"`
			Title string `xml:"title"`
		} `xml:"slide"`
	}

	resp, err := grequests.Get("http://httpbin.org/xml", nil)

	if err != nil {
		log.Println("Unable to make request", err)
	}

	if resp.Ok != true {
		log.Println("Request did not return OK")
	}

	userXML := &GetXMLSample{}

	// func xmlASCIIDecoder(charset string, input io.Reader) (io.Reader, error) {
	// 	return input, nil
	// }

	// If the server returns XML encoded in another charset (not UTF-8) – you
	// must provide an encoder function that looks like the one I wrote above.

	// If you an consuming UTF-8 just pass `nil` into the second arg
	if err := resp.XML(userXML, xmlASCIIDecoder); err != nil {
		log.Println("Unable to consume the response as XML: ", err)
	}

	if userXML.Title != "Sample Slide Show" {
		log.Printf("Invalid XML serialization %#v", userXML)
	}
}

func xmlASCIIDecoder(charset string, input io.Reader) (io.Reader, error) {
	return input, nil
}

Output:
Example (PostFileUpload) 
package main

import (
	"log"

	"github.com/levigross/grequests"
)

func main() {

	fd, err := grequests.FileUploadFromDisk("test_files/mypassword")

	if err != nil {
		log.Println("Unable to open file: ", err)
	}

	// This will upload the file as a multipart mime request
	resp, err := grequests.Post("http://httpbin.org/post",
		&grequests.RequestOptions{
			Files: fd,
			Data:  map[string]string{"One": "Two"},
		})

	if err != nil {
		log.Println("Unable to make request", resp.Error)
	}

	if resp.Ok != true {
		log.Println("Request did not return OK")
	}
}

Output:
Example (PostForm) 
package main

import (
	"log"

	"github.com/levigross/grequests"
)

func main() {
	resp, err := grequests.Post("http://httpbin.org/post",
		&grequests.RequestOptions{Data: map[string]string{"One": "Two"}})

	// This is the basic form POST. The request body will be `one=two`

	if err != nil {
		log.Println("Cannot post: ", err)
	}

	if resp.Ok != true {
		log.Println("Request did not return OK")
	}
}

Output:
Example (PostJSONAJAX) 
package main

import (
	"log"

	"github.com/levigross/grequests"
)

func main() {
	resp, err := grequests.Post("http://httpbin.org/post",
		&grequests.RequestOptions{
			JSON:   map[string]string{"One": "Two"},
			IsAjax: true, // this adds the X-Requested-With: XMLHttpRequest header
		})

	if err != nil {
		log.Println("Unable to make request", resp.Error)
	}

	if resp.Ok != true {
		log.Println("Request did not return OK")
	}

}

Output:
Example (PostXML) 
package main

import (
	"log"

	"github.com/levigross/grequests"
)

func main() {

	type XMLPostMessage struct {
		Name   string
		Age    int
		Height int
	}

	resp, err := grequests.Post("http://httpbin.org/post",
		&grequests.RequestOptions{XML: XMLPostMessage{Name: "Human", Age: 1, Height: 1}})
	// The request body will contain the XML generated by the `XMLPostMessage` struct

	if err != nil {
		log.Println("Unable to make request", resp.Error)
	}

	if resp.Ok != true {
		log.Println("Request did not return OK")
	}
}

Output:
Example (Proxy) 
package main

import (
	"log"
	"net/url"

	"github.com/levigross/grequests"
)

func main() {
	proxyURL, err := url.Parse("http://127.0.0.1:8080") // Proxy URL
	if err != nil {
		log.Panicln(err)
	}

	resp, err := grequests.Get("http://www.levigross.com/",
		&grequests.RequestOptions{Proxies: map[string]*url.URL{proxyURL.Scheme: proxyURL}})

	if err != nil {
		log.Println(err)
	}

	if resp.Ok != true {
		log.Println("Request did not return OK")
	}

	log.Println(resp)
}

Output:
Example (Session) 
package main

import (
	"log"

	"github.com/levigross/grequests"
)

func main() {
	session := grequests.NewSession(nil)

	resp, err := session.Get("http://httpbin.org/cookies/set", &grequests.RequestOptions{Params: map[string]string{"one": "two"}})

	if err != nil {
		log.Fatal("Cannot set cookie: ", err)
	}

	if resp.Ok != true {
		log.Println("Request did not return OK")
	}

	log.Println(resp.String())

}

Output:
Example (UrlQueryParams) 
package main

import (
	"log"

	"github.com/levigross/grequests"
)

func main() {
	ro := &grequests.RequestOptions{
		Params: map[string]string{"Hello": "World", "Goodbye": "World"},
	}
	resp, err := grequests.Get("http://httpbin.org/get", ro)
	// url will now be http://httpbin.org/get?hello=world&goodbye=world

	if err != nil {
		log.Println("Unable to make request", err)
	}

	if resp.Ok != true {
		log.Println("Request did not return OK")
	}
}

Output:

Index

Examples

Constants

This section is empty.

Variables

View Source 
var (
	// ErrRedirectLimitExceeded is the error returned when the request responded
	// with too many redirects
	ErrRedirectLimitExceeded = errors.New("grequests: Request exceeded redirect count")

	// RedirectLimit is a tunable variable that specifies how many times we can
	// redirect in response to a redirect. This is the global variable, if you
	// wish to set this on a request by request basis, set it within the
	// `RequestOptions` structure
	RedirectLimit = 30

	// SensitiveHTTPHeaders is a map of sensitive HTTP headers that a user
	// doesn't want passed on a redirect. This is the global variable, if you
	// wish to set this on a request by request basis, set it within the
	// `RequestOptions` structure
	SensitiveHTTPHeaders = map[string]struct{}{
		"WWW-Authenticate":    {},
		"Authorization":       {},
		"Proxy-Authorization": {},
	}
)

Functions

func BuildHTTPClient 

func BuildHTTPClient(ro RequestOptions) *http.Client

BuildHTTPClient is a function that will return a custom HTTP client based on the request options provided the check is in UseDefaultClient

Types

type FileUpload 

type FileUpload struct {
	// Filename is the name of the file that you wish to upload. We use this to guess the mimetype as well as pass it onto the server
	FileName string

	// FileContents is happy as long as you pass it a io.ReadCloser (which most file use anyways)
	FileContents io.ReadCloser
}

FileUpload is a struct that is used to specify the file that a User wishes to upload.

func FileUploadFromDisk 

func FileUploadFromDisk(fileName string) ([]FileUpload, error)

FileUploadFromDisk allows you to create a FileUpload struct slice by just specifying a location on the disk

func FileUploadFromGlob 

func FileUploadFromGlob(fileSystemGlob string) ([]FileUpload, error)

FileUploadFromGlob allows you to create a FileUpload struct slice by just specifying a glob location on the disk this function will gloss over all errors in the files and only upload the files that don't return errors from the glob

type RequestOptions 

type RequestOptions struct {

	// Data is a map of key values that will eventually convert into the query string of a GET request or the
	// body of a POST request.
	Data map[string]string

	// Params is a map of query strings that may be used within a GET request
	Params map[string]string

	// Files is where you can include files to upload. The use of this data structure is limited to POST requests
	Files []FileUpload

	// JSON can be used when you wish to send JSON within the request body
	JSON interface{}

	// XML can be used if you wish to send XML within the request body
	XML interface{}

	// Headers if you want to add custom HTTP headers to the request, this is your friend
	Headers map[string]string

	// InsecureSkipVerify is a flag that specifies if we should validate the server's TLS certificate. It should be noted that
	// Go's TLS verify mechanism doesn't validate if a certificate has been revoked
	InsecureSkipVerify bool

	// DisableCompression will disable gzip compression on requests
	DisableCompression bool

	// UserAgent allows you to set an arbitrary custom user agent
	UserAgent string

	// Auth allows you to specify a user name and password that you wish to use when requesting
	// the URL. It will use basic HTTP authentication formatting the username and password in base64
	// the format is []string{username, password}
	Auth []string

	// IsAjax is a flag that can be set to make the request appear to be generated by browser Javascript
	IsAjax bool

	// Cookies is an array of `http.Cookie` that allows you to attach cookies to your request
	Cookies []http.Cookie

	// UseCookieJar will create a custom HTTP client that will process and store HTTP cookies when they are sent down
	UseCookieJar bool

	// Proxies is a map in the following format *protocol* => proxy address e.g http => http://127.0.0.1:8080
	Proxies map[string]*url.URL

	// TLSHandshakeTimeout specifies the maximum amount of time waiting to
	// wait for a TLS handshake. Zero means no timeout.
	TLSHandshakeTimeout time.Duration

	// DialTimeout is the maximum amount of time a dial will wait for
	// a connect to complete.
	DialTimeout time.Duration

	// KeepAlive specifies the keep-alive period for an active
	// network connection. If zero, keep-alive are not enabled.
	DialKeepAlive time.Duration

	// HTTPClient can be provided if you wish to supply a custom HTTP client
	// this is useful if you want to use an OAUTH client with your request
	HTTPClient *http.Client

	// RedirectLocationTrusted is a flag that will enable all headers to be
	// forwarded to the redirect location. Otherwise, the headers specified in
	// `SensitiveHTTPHeaders` will be removed from the request
	RedirectLocationTrusted bool

	SensitiveHTTPHeaders map[string]struct{}

	// RedirectLimit is the acceptable amount of redirects that we should expect before returning an error
	// be default this is set to 30. You can change this globally by modifying the `RedirectLimit` variable
	RedirectLimit int
}

RequestOptions is the location that of where the data

type Response 

type Response struct {

	// Ok is a boolean flag that validates that the server returned a 2xx code
	Ok bool

	// This is the Go error flag – if something went wrong within the request, this flag will be set.
	Error error

	// We want to abstract (at least at the moment) the Go http.Response object away. So we are going to make use of it
	// internal but not give the user access
	RawResponse *http.Response

	// StatusCode is the HTTP Status Code returned by the HTTP Response. Taken from resp.StatusCode
	StatusCode int

	// Header is a net/http/Header structure
	Header http.Header
	// contains filtered or unexported fields
}

Response is what is returned to a user when they fire off a request

func Delete 

func Delete(url string, ro *RequestOptions) (*Response, error)

Delete takes 2 parameters and returns a Response struct. These two options are:

  1. A URL
  2. A RequestOptions struct

If you do not intend to use the `RequestOptions` you can just pass nil

func Get 

func Get(url string, ro *RequestOptions) (*Response, error)

Get takes 2 parameters and returns a Response Struct. These two options are:

  1. A URL
  2. A RequestOptions struct

If you do not intend to use the `RequestOptions` you can just pass nil 

func Head(url string, ro *RequestOptions) (*Response, error)

Head takes 2 parameters and returns a Response channel. These two options are:

  1. A URL
  2. A RequestOptions struct

If you do not intend to use the `RequestOptions` you can just pass nil

func Options 

func Options(url string, ro *RequestOptions) (*Response, error)

Options takes 2 parameters and returns a Response struct. These two options are:

  1. A URL
  2. A RequestOptions struct

If you do not intend to use the `RequestOptions` you can just pass nil

func Patch 

func Patch(url string, ro *RequestOptions) (*Response, error)

Patch takes 2 parameters and returns a Response struct. These two options are:

  1. A URL
  2. A RequestOptions struct

If you do not intend to use the `RequestOptions` you can just pass nil

func Post 

func Post(url string, ro *RequestOptions) (*Response, error)

Post takes 2 parameters and returns a Response channel. These two options are:

  1. A URL
  2. A RequestOptions struct

If you do not intend to use the `RequestOptions` you can just pass nil

func Put 

func Put(url string, ro *RequestOptions) (*Response, error)

Put takes 2 parameters and returns a Response struct. These two options are:

  1. A URL
  2. A RequestOptions struct

If you do not intend to use the `RequestOptions` you can just pass nil

func (*Response) Bytes 

func (r *Response) Bytes() []byte

Bytes returns the response as a byte array

func (*Response) ClearInternalBuffer 

func (r *Response) ClearInternalBuffer()

ClearInternalBuffer is a function that will clear the internal buffer that we use to hold the .String() and .Bytes() data. Once you have used these functions – you may want to free up the memory.

func (*Response) Close 

func (r *Response) Close() error

Close is part of our ability to support io.ReadCloser if someone wants to make use of the raw body

func (*Response) DownloadToFile 

func (r *Response) DownloadToFile(fileName string) error

DownloadToFile allows you to download the contents of the response to a file

func (*Response) JSON 

func (r *Response) JSON(userStruct interface{}) error

JSON is a method that will populate a struct that is provided `userStruct` with the JSON returned within the response body

func (*Response) Read 

func (r *Response) Read(p []byte) (n int, err error)

Read is part of our ability to support io.ReadCloser if someone wants to make use of the raw body

func (*Response) String 

func (r *Response) String() string

String returns the response as a string

func (*Response) XML 

func (r *Response) XML(userStruct interface{}, charsetReader XMLCharDecoder) error

XML is a method that will populate a struct that is provided `userStruct` with the XML returned within the response body

type Session 

type Session struct {

	// HTTPClient is the client that we will use to request the resources
	HTTPClient *http.Client
}

Session allows a user to make use of persistent cookies in between HTTP requests

func NewSession 

func NewSession(ro *RequestOptions) *Session

NewSession returns a session struct which enables can be used to maintain establish a persistent state with the server This function will set UseCookieJar to true as that is the purpose of using the session

func (*Session) CloseIdleConnections 

func (s *Session) CloseIdleConnections()

CloseIdleConnections closes the idle connections that a session client may make use of

func (*Session) Delete 

func (s *Session) Delete(url string, ro *RequestOptions) (*Response, error)

Delete takes 2 parameters and returns a Response struct. These two options are:

  1. A URL
  2. A RequestOptions struct

If you do not intend to use the `RequestOptions` you can just pass nil A new session is created by calling NewSession with a request options struct

func (*Session) Get 

func (s *Session) Get(url string, ro *RequestOptions) (*Response, error)

Get takes 2 parameters and returns a Response Struct. These two options are:

  1. A URL
  2. A RequestOptions struct

If you do not intend to use the `RequestOptions` you can just pass nil A new session is created by calling NewSession with a request options struct

func (*Session) Head 

func (s *Session) Head(url string, ro *RequestOptions) (*Response, error)

Head takes 2 parameters and returns a Response channel. These two options are:

  1. A URL
  2. A RequestOptions struct

If you do not intend to use the `RequestOptions` you can just pass nil A new session is created by calling NewSession with a request options struct

func (*Session) Options 

func (s *Session) Options(url string, ro *RequestOptions) (*Response, error)

Options takes 2 parameters and returns a Response struct. These two options are:

  1. A URL
  2. A RequestOptions struct

If you do not intend to use the `RequestOptions` you can just pass nil A new session is created by calling NewSession with a request options struct

func (*Session) Patch 

func (s *Session) Patch(url string, ro *RequestOptions) (*Response, error)

Patch takes 2 parameters and returns a Response struct. These two options are:

  1. A URL
  2. A RequestOptions struct

If you do not intend to use the `RequestOptions` you can just pass nil A new session is created by calling NewSession with a request options struct

func (*Session) Post 

func (s *Session) Post(url string, ro *RequestOptions) (*Response, error)

Post takes 2 parameters and returns a Response channel. These two options are:

  1. A URL
  2. A RequestOptions struct

If you do not intend to use the `RequestOptions` you can just pass nil A new session is created by calling NewSession with a request options struct

func (*Session) Put 

func (s *Session) Put(url string, ro *RequestOptions) (*Response, error)

Put takes 2 parameters and returns a Response struct. These two options are:

  1. A URL
  2. A RequestOptions struct

If you do not intend to use the `RequestOptions` you can just pass nil A new session is created by calling NewSession with a request options struct

type XMLCharDecoder 

type XMLCharDecoder func(charset string, input io.Reader) (io.Reader, error)

XMLCharDecoder is a helper type that takes a stream of bytes (not encoded in UTF-8) and returns a reader that encodes the bytes into UTF-8. This is done because Go's XML library only supports XML encoded in UTF-8

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.