http

package
v1.0.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 11, 2021 License: AGPL-3.0 Imports: 14 Imported by: 0

Documentation

Overview

Package http provides a 0 allocation wrapper around the fasthttp library.

The wrapper facilitates our redirect handling, our custom "target" and request building format, and handling the various options we can attach to building a request.

Most structs in this package have a corresponding Acquire* and Release* function for using sync.Pool 'd objects. This is to minimise allocations in your request hotloop. We recommend using Acquire* and Release* wherever possible to ensure that unecessary allocations are avoided.

The Target type provides our wrapper around the full context needed to perform a http request. There are a few quirks when using targets that one has to be weary of. These quirks are side-effects of the zero-locking and and minimal allocation behaviour. Admittedly, this is very developer unfriendly, and changes to the API that make it more usable while maintaining the performance are welcome.

  • We expect Targets to be instantiated then updated as required
  • Once Target.ParseHostHeader() has been called further modifications will not be respected
  • Target.HTTPClient() will cache the first set of options that are used. Future modifications are ignored
  • All operations on a target are thread safe to use
Example (TargetUsage)

Example TargetUsage shows how to acquire a target, perform a request then release the target using the provided helper functions. 

t := AcquireTarget()
t.Hostname = "example.com"
t.Port = 443
t.IsTLS = true
t.BasePath = "basepath"

// you have to call ParseHostHeader after you're done instantiating the client, otherwise
// the request will have no host header and will fail
t.ParseHostHeader()

req := Request{
	Route:  &Route{Path: []byte("/")},
	Target: t,
}
config := &Config{
	Timeout:     1 * time.Second,
	ReadHeaders: false,
	ReadBody:    false,
}

c := t.HTTPClient(5, 1*time.Second)
resp, err := DoClient(c, req, config)
if err != nil {
	log.Error().Err(err).Msg("failed to make request")
}

log.Info().Msgf("success: %+v", resp)
ReleaseTarget(t)

Output:

Index

Examples

Constants

This section is empty.

Variables

View Source 
var (
	ChunkedRoutesPool sync.Pool
)

Functions

func ReleaseChunkedRoutes 

func ReleaseChunkedRoutes(h *ChunkedRoutes)

releaseChunkedRoutes releases a host into the shared header pool

func ReleaseHeader 

func ReleaseHeader(h *Header)

ReleaseHeader releases a host into the shared header pool

func ReleaseRequest 

func ReleaseRequest(h *Request)

ReleaseRequest releases a host into the shared header pool

func ReleaseResponse 

func ReleaseResponse(h *Response)

ReleaseResponse releases a host into the shared header pool

func ReleaseTarget 

func ReleaseTarget(h *Target)

ReleaseTarget releases a host into the shared target pool

func StatusCodeIsRedirect 

func StatusCodeIsRedirect(statusCode int) bool

StatusCodeIsRedirect returns true if the status code indicates a redirect.

Types

type BackupClient 

type BackupClient = fasthttp.Client

BackupClient is a normal fasthttpClient that can adapt to different hosts This is used to handle redirects that change the host/port of the request

type ChunkedRoutes 

type ChunkedRoutes [][]*Route

func AcquireChunkedRoutes 

func AcquireChunkedRoutes() *ChunkedRoutes

acquireChunkedRoutes retrieves a host from the shared header pool

func ChunkRoutes 

func ChunkRoutes(items []*Route, src *ChunkedRoutes, chunks int) *ChunkedRoutes

type Config 

type Config struct {
	// Timeout is the duration to wait when performing a DoTimeout request
	Timeout time.Duration `toml:"timeout" json:"timeout" mapstructure:"timeout"`
	// MaxRedirects corresponds to how many redirects to follow. 0 means the first request will return and no
	// redirects are followed
	MaxRedirects int `toml:"max_redirects" json:"max_redirects" mapstructure:"max_redirects"`

	// ReadBody defines whether to copy the body into the Response object. We will always read the body of the request
	// off the wire to perform the length and word count calculations
	ReadBody bool
	// ReadHeaders defines whether to copy the headers into the Response object. We will always peek the location header
	// to determine whether to follow redirects
	ReadHeaders bool
	// BlacklistRedirects is a slice of strings that are substring matched against location headers.
	// if a string is blacklisted, e.g. okta.com, the redirect is not followed
	BlacklistRedirects []string `toml:"blacklist_redirects" json:"blacklist_redirects" mapstructure:"blacklist_redirects"`

	// ExtraHeaders are added to the request last and will overwrite the route headers
	ExtraHeaders []Header
	// contains filtered or unexported fields
}

Config provides all the options available to a request, this is used by DoClient

func (*Config) BackupClient 

func (c *Config) BackupClient() *BackupClient

BackupClient provides a generic http client that is not bound to a single host. it is generated on demand based off the config options. This is done only once per config. if you change the options after calling BackupClient(), you must call ResetBackupClient()

func (*Config) IsBlacklistedRedirect 

func (c *Config) IsBlacklistedRedirect(host []byte) bool

IsBlacklistedRedirect will compare the stored hosts against the provided host. We use a prefix match with linear probing to simplify the check. e.g. blacklist[okta.com, onelogin.com] will match against okta.com:80

func (*Config) ResetBackupClient 

func (c *Config) ResetBackupClient()

ResetBackupClient will update the settings on the BackupClient. This is to be called if the timeout is changed on the config after calling BackupClient

type Field 

type Field struct {
	Key  string    `toml:"key" json:"key" mapstructure:"key"`
	Type FieldType `toml:"type" json:"type" mapstructure:"type"`
}

func StringToFields 

func StringToFields(in string) (ret []Field)

StringToField will break down a URL path into fields This is a convenience function to convert /foo/bar into []Field{{"foo"}, {"bar"}}

func (*Field) AppendBytes 

func (f *Field) AppendBytes(dst []byte) []byte

Write the string to the specified writer

func (*Field) Bytes 

func (f *Field) Bytes() []byte

type FieldType 

type FieldType int
const (
	String FieldType = iota
	UUID
	Int
	Date
	Timestamp
	Format
)

type HTTPClient 

type HTTPClient = fasthttp.HostClient

HTTPClient is a type alias for the actual host client we use. We do this instead of using an interface to avoid reflecting

func NewHTTPClient 

func NewHTTPClient(host string, tls bool) *HTTPClient

NewHTTPClient will create a http client configured specifically for requesting against the targetted host. This is backed by the fasthttp.HostClient 

type Header struct {
	Key   string
	Value string
}

Header encapsulates a header key value entry TODO: replace strings with byte slices

func AcquireHeader 

func AcquireHeader() *Header

AcquireHeader retrieves a host from the shared header pool

func (*Header) AppendBytes 

func (h *Header) AppendBytes(b []byte) []byte

func (Header) MarshalZerologObject 

func (h Header) MarshalZerologObject(e *zerolog.Event)

func (*Header) String 

func (h *Header) String() string

func (*Header) Write 

func (h *Header) Write(buf io.Writer) (int, error)

type HeaderField 

type HeaderField struct {
	Key   Field
	Value Field
}

type Headers 

type Headers []Header

func (Headers) MarshalZerologArray 

func (rr Headers) MarshalZerologArray(a *zerolog.Array)

type Method 

type Method []byte
var (
	GET    Method = []byte("GET")
	POST   Method = []byte("POST")
	PATCH  Method = []byte("PATCH")
	DELETE Method = []byte("DELETE")
	PUT    Method = []byte("PUT")
	TRACE  Method = []byte("TRACE")

	ErrUnsupportedMethod = fmt.Errorf("unsupported method")
)

func MethodFromString 

func MethodFromString(m string) (Method, error)

type Range 

type Range struct {
	Min int
	Max int
}

func RangeFromString 

func RangeFromString(in string) (ret Range, err error)

RangeFromString will return a range from a string like 5-10

func (Range) String 

func (r Range) String() string

type Request 

type Request struct {
	Target *Target
	Route  *Route
}

func AcquireRequest 

func AcquireRequest() *Request

AcquireRequest retrieves a host from the shared header pool

func (*Request) String 

func (r *Request) String() string

func (*Request) WriteRequest 

func (r *Request) WriteRequest(dst *fasthttp.Request, basepath []byte)

WriteRequest will populate the request object with all the required information from the Target details, and from the route details The path is construction with target.Basepath + basepath + route.Path no slashes are added inbetween. we assume the user supplies everything

type Response 

type Response struct {
	StatusCode  int
	Words       int
	Lines       int
	BodyLength  int
	HTTPVersion string

	// these headers should not be used elsewhere, they are released into the pool on object release into the pool
	Headers []*Header
	Body    []byte

	// URI is the URI for the request. We store this here to avoid referencing the request when walking the tree
	// The first request will have the URI be nil. This is because you can reconstruct the first response
	// based off the origin request.
	// This will only be set in secondary/subsequent requests in a redirect chain
	URI []byte

	// OriginRequest corresponds to the original request used. this may not correlate to a redirected request
	OriginRequest Request
	Next          *Response
	Error         error // Whether an error occurred during the request
}

func AcquireResponse 

func AcquireResponse() *Response

AcquireResponse retrieves a host from the shared header pool

func DoClient 

func DoClient(c *HTTPClient, req Request, config *Config) (Response, error)

DoClient performs the provided request. We recommend avoiding letting the Response escape to the heap to prevent allocating where not necessary. This will handle the redirects for the request. Redirect responses are added to the linked list in the Response The returned response chain will have the Response.OriginRequest populated This will always read the body from the wire, but will only copy the body into the Response if config.ReadBody is true We will always perform the calculations for the BodyLength, Words and Lines, as these require 0 allocations given the response body is already read into memory

Responses part of the chain are allocated dynamically using AcquireResponse and should be appropriately released when the response is no longer needed

func (*Response) AddHeader 

func (r *Response) AddHeader(k, v []byte)

func (*Response) AppendRedirectChain 

func (r *Response) AppendRedirectChain(b []byte) []byte

func (*Response) Flatten 

func (r *Response) Flatten() (ret Responses)

func (Response) MarshalZerologObject 

func (r Response) MarshalZerologObject(e *zerolog.Event)

func (*Response) Reset 

func (r *Response) Reset()

func (*Response) String 

func (r *Response) String() string

type Responses 

type Responses []*Response

func (Responses) MarshalZerologArray 

func (rr Responses) MarshalZerologArray(a *zerolog.Array)

type Route 

type Route struct {
	Headers []Header
	Path    []byte
	Query   []byte
	Source  string
	Method  Method
	Body    []byte
}

func FilterSource 

func FilterSource(routes []*Route, want map[string]interface{}) []*Route

FilterSource will only return the routes that exist in the map provided

func UniqueSource 

func UniqueSource(routes []*Route) []*Route

UniqueSource will unique the routes based on the ID

func (Route) AppendBytes 

func (r Route) AppendBytes(b []byte) []byte

func (*Route) AppendPath 

func (r *Route) AppendPath(dst []byte) []byte

func (*Route) AppendQuery 

func (r *Route) AppendQuery(dst []byte) []byte

func (Route) AppendShortBytes 

func (r Route) AppendShortBytes(b []byte) []byte

func (Route) MarshalZerologObject 

func (r Route) MarshalZerologObject(e *zerolog.Event)

func (Route) String 

func (r Route) String() string

type RouteMap 

type RouteMap map[string][]*Route

func GroupRouteDepth 

func GroupRouteDepth(routes []*Route, depth int64) RouteMap

GroupRouteDepth will collate the routes into the corresponding depth of path

func (RouteMap) Flatten 

func (r RouteMap) Flatten() []*Route

func (RouteMap) FlattenCount 

func (r RouteMap) FlattenCount() int

type Target 

type Target struct {
	Hostname string // Hostname is the bare hostname without the port.

	// HostHeader is the host header to use in the request.
	// The host header for a request will be determined in the following order (If the value is nil, or empty, then
	// the next option will be chosen)
	// 1. HostHeader
	// 2. Hostname:Port
	HostHeader []byte

	IP       string // the IP is the address used to reach the server. If empty, Hostname will be used
	Port     int    // Port will be the port used to reach the server.
	IsTLS    bool   // IsTLS defines whether to use a TLS dialer or normal dialer
	BasePath string // BasePath allows for custom discovery on a target below the root directory

	// Header is an ordered list of http headers to add to the request.
	// Header behaviour is determined by fasthttp.Request  and fasthttp.RequestHeader
	// Authentication and Content-Type headers should be added to here
	// the Host header is separately governed by HostHeader
	Headers []Header
	// contains filtered or unexported fields
}

Target encapsulates the basic data required to scan any given application. This can be specified to attach to a specific vhost or subpath configured webserver.

When using target, you *MUST* call ParseHostHeader before calling AppendHostHeader to ensure the header is parsed. This developer unfriendly design decision was made to avoid having to create locks around generating the host header and to avoid allocating additional memory when requesting the host header. We recommend against changing the fields of a target after ParseHostHeader has been called as this will result in unexpected behaviour.

The request is sent to the value returned by Host(). this is derived from the IP:port or Hostname:port.

This target can be used to instantiate its own client using HTTPClient(maxconns, timeout).

The Target should only be instantiated once for each use and is passed around via a pointer to ensure that the HTTPClient is reused.

The Quarantine and Context features of a target can be used indicate to other readers across goroutines that the target is responding in an unexpected behaviour (i.e. responding to too many requests unexpected) and/or the target should be abandoned.

A sync.Pool is provided for minimising allocations when creating/reusing targets. You can use AcquireTarget() and ReleaseTarget() correspondingly to reuse targets. After a target has been released, it is not safe for reuse and can result in race conditions

func AcquireTarget 

func AcquireTarget() *Target

AcquireTarget retrieves a host from the shared target pool

func (*Target) AppendBytes 

func (t *Target) AppendBytes(b []byte) []byte

AppendBytes will append the full request details including the headers and scheme to the provided buffer e.g. http://google.com:80/foo {x-forwarded-for:127.0.0.1}

func (*Target) AppendHost 

func (t *Target) AppendHost(buf []byte) []byte

AppendHost will append the host to make the request including the port. e.g. foo.com:80 or if t.IP is set, 1.1.1.1:80 this can be used for the HostHeader if HostHeader is not set

func (*Target) AppendHostHeader 

func (t *Target) AppendHostHeader(buf []byte) []byte

AppendHostHeader will return the HostHeader to be used in the HTTP Request. This will first prioritize t.HostHeader, then t.Hostname:t.Port. You should always call ParseHostHeader before calling AppendHostHeader otherwise there might not be a host header. We avoid acquiring a lock since this is in the hotpath for requests we don't parse the host header here because that's expensive and requires acquiring a lock

func (*Target) AppendIPOrHostname 

func (t *Target) AppendIPOrHostname(buf []byte) []byte

AppendIPOrHostname will append the ip if set, otherwise the hostname use this to determine where to send the request, not the host header This does not include the port

func (*Target) AppendScheme 

func (t *Target) AppendScheme(buf []byte) []byte

AppendScheme will append the scheme to the host not including the ://

func (*Target) Bytes 

func (t *Target) Bytes() []byte

Bytes will return the same output as String. This is cached in t.b If the target is changed after Bytes() is called, the changes will not be reflected

func (*Target) Cancel 

func (t *Target) Cancel()

Cancel will cancel the context associated with the host. This can be used to expire the target's validity across multiple running goroutines. This will simply cancel the context associated with the host. It relies upon user implementation to ensure that further interactions with the target do not occur

func (*Target) Context 

func (t *Target) Context() context.Context

Context will return the context of this object. If nil, it will initialize a context for this target This context can be used across goroutines to assess the validity of the goroutine. This is independent of the quarantine state. 

select {
case <-target.Context().Done():
	return
case default:
	resp, err := http.DoClient(target.HTTPClient(5, time.Second), req, &config.HTTP)
}

func (*Target) HTTPClient 

func (t *Target) HTTPClient(maxConnections int, timeout time.Duration) *HTTPClient

HTTPClient will return a HTTPClient configured for the particular target with the configured maxConnections and timeout. This is cached after the first call, so subsequent changes to the Host and IsTLS after the first call of HTTPClient will not be respected

func (*Target) HitIncr 

func (t *Target) HitIncr() int64

HitIncr will threadsafe increment the target hit counter. This should be called whenever a request is performed against the host

func (*Target) HitReset 

func (t *Target) HitReset() int64

Reset returns the old quarantine counter

func (*Target) Hits 

func (t *Target) Hits() int64

Hits will return the value of the number of hits this target has been used for use HitIncr to incremenet the hits

func (*Target) Host 

func (t *Target) Host() string

Host will return the Host:Port or IP:Port of the target

func (*Target) ParseHostHeader 

func (t *Target) ParseHostHeader() []byte

ParseHostHeader will perform a thread safe update of t.HostHeader using the existing fields If t.HostHeader is already set, this operation will just return t.HostHeader otherwise, this will perform t.AppendHost(t.HostHeader[:0])

func (*Target) Quarantine 

func (t *Target) Quarantine()

Quarantine will threadsafe quarantine the host. To reset the quarantine state use QuarantineReset

func (*Target) QuarantineIncr 

func (t *Target) QuarantineIncr() int64

QuarantineIncr will threadsafe incremenet the quarantine counter This will return the new value of the quarantine counter

func (*Target) QuarantineReset 

func (t *Target) QuarantineReset() int64

QuarantineReset will threadsafe unquarantine the host

func (*Target) Quarantined 

func (t *Target) Quarantined() bool

Quarantined will return whether the target has been quarantined

func (*Target) SetContext 

func (t *Target) SetContext(c context.Context)

SetContext will overwrite this context and cancellation with the provided context

func (*Target) String 

func (t *Target) String() string

String will return a string representation of the target

func (*Target) Write 

func (t *Target) Write(b io.Writer) (int, error)

Write will write the target out to the buffer specified

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.