lib

Documentation

Overview

Package lib - this parts contains the *http.Client used to make requests.

Could possibly just use another HTTP library that has some of these constants defined ...

Package lib - This is a debug.

Package lib - this parts contains the *http.Client used to make requests.

Could possibly just use another HTTP library that has some of these constants defined ...

Since the std. library doesn't contain these things, I need to redefine them. A good library to use would be https://github.com/labstack/echo/blob/master/echo.go. In particular see these lines, https://github.com/labstack/echo/blob/master/echo.go#L153.

Index

• Constants
• Variables
• func GetHeaderUserAgentValue() string
• func NewRequest(method, url string, body io.Reader) (*http.Request, error)
• func SetDefaultHeaders(req *http.Request) *http.Request
• type DebugTransport
  • func (d *DebugTransport) Logger() *zerolog.Logger
  • func (d *DebugTransport) RoundTrip(req *http.Request) (*http.Response, error)
• type HTTPClient
  • func NewHTTPClient(insecureSkipVerify bool, timeout time.Duration, loggerParent *zerolog.Logger) (*HTTPClient, error)

Constants

const (
	HTTPClientTimeoutDefault = 15 * time.Second
	MaxIdleConns             = 10
	IdleConnTimeout          = 30 * time.Second
	DisableCompression       = true
)

From https://golang.org/pkg/net/http/#pkg-overview:

For control over proxies, TLS configuration, keep-alives, compression, and other settings, create a Transport:

tr := &http.Transport{
	MaxIdleConns:       10,
	IdleConnTimeout:    30 * time.Second,
	DisableCompression: true,
}

client := &http.Client{Transport: tr} resp, err := client.Get("https://example.com")

const (
	HeaderAcceptKey       = "Accept"
	HeaderCacheControlKey = "Cache-Control"
	HeaderUserAgentKey    = "User-Agent"
	HeaderContentTypeKey  = "Content-Type"
)

HTTP standard headers.

const (
	HeaderAcceptValueAll           = "*/*"
	HeaderCacheControlValueNoCache = "no-cache"
)

Header standard headers values that we use.

const (
	Version = "0.0.1"
	Website = "https://bana.io"
)

const (
	// ContentTypeHTMLUTF8 - Content-Type: Possible returns types.
	//
	// Examples:
	//
	// * "content-type: text/html; charset=UTF-8"
	// * "content-type: text/html; charset=utf-8"
	// * "content-type: text/html;charset=utf-8"
	//
	// All the examples above are valid. Note the spacing and casing does not seem to matter.
	// ContentTypeHTMLUTF8 = "content-type: text/html; charset=UTF-8"
	// Remove above line.
	ContentTypeHTMLUTF8 = "text/html; charset=UTF-8"
)

Variables

var ContentTypeHTMLValidValues = []string{

	"text/html; charset=UTF-8",
	"text/html; charset=utf-8",
	"text/html;charset=utf-8",
	"text/html; charset=UTF-8",
}

ContentTypeHTMLValidValues - Content-Type: Possible returns types. There are possibly more - I need to consult the RFCs. `ContentTypeHTMLValidValues` is a global variable (gochecknoglobals) nolint:gochecknoglobals

var HeaderUserAgentValue = GetHeaderUserAgentValue()

HeaderUserAgentValue - `User-Agent` to use. Panic if we fail to get it. Example: "github.com/banaio/golang/project_template/0.0.1 (https://bana.io)".

`HeaderUserAgentValue` is a global variable (gochecknoglobals)go-lint nolint:gochecknoglobals

Functions

func GetHeaderUserAgentValue

func GetHeaderUserAgentValue() string

GetHeaderUserAgentValueModuleName - generally a bad idea to exit in the function. However, if we can't determine module name, something is really wrong.. Returns something like "github.com/banaio/golang/project_template/0.0.1 (https://bana.io)".

func NewRequest

func NewRequest(method, url string, body io.Reader) (*http.Request, error)

NewRequest - request.

func SetDefaultHeaders

func SetDefaultHeaders(req *http.Request) *http.Request

Types

type DebugTransport

type DebugTransport struct {
	*http.Transport
	LoggerParent *zerolog.Logger
}

DebugTransport - logs the requests and responses.

func (*DebugTransport) Logger

func (d *DebugTransport) Logger() *zerolog.Logger

Logger - Returns a contextual logger.

func (*DebugTransport) RoundTrip

func (d *DebugTransport) RoundTrip(req *http.Request) (*http.Response, error)

type HTTPClient

type HTTPClient struct {
	*http.Client `json:"net_http_client,omitempty"`
	LoggerParent *zerolog.Logger `json:"logger_parent,omitempty"` // Logger: parent logger. Use `Crawler.Logger()` to get contextual logger.
}

HTTPClient - Just wraps *http.Client.

func NewHTTPClient

func NewHTTPClient(insecureSkipVerify bool, timeout time.Duration, loggerParent *zerolog.Logger) (*HTTPClient, error)

NewHTTPClient - Make new client for issuing requests.