httpx

package
v1.3.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 9, 2023 License: MIT Imports: 35 Imported by: 40

Documentation

Overview

Package httpx containst the httpx common funcionality

Index

Constants

View Source 
const (
	// HTTP defines the plain http scheme
	HTTP = "http"
	// HTTPS defines the secure http scheme
	HTTPS = "https"
	// HTTPorHTTPS defines both http and https scheme in mutual exclusion
	HTTPorHTTPS = "http|https"
	// HTTPandHTTPS defines both http and https scheme
	HTTPandHTTPS = "http&https"
)

Variables

View Source 
var CSPHeaders = []string{
	"Content-Security-Policy",
	"Content-Security-Policy-Report-Only",
	"X-Content-Security-Policy-Report-Only",
	"X-Webkit-Csp-Report-Only",
}

CSPHeaders is an incomplete list of most common CSP headers

View Source 
var DefaultOptions = Options{
	RandomAgent:  true,
	Threads:      25,
	Timeout:      30 * time.Second,
	RetryMax:     5,
	MaxRedirects: 10,
	Unsafe:       false,
	CdnCheck:     true,
	ExcludeCdn:   false,

	VHostIgnoreStatusCode:    false,
	VHostIgnoreContentLength: true,
	VHostIgnoreNumberOfWords: false,
	VHostIgnoreNumberOfLines: false,
	VHostStripHTML:           false,
	VHostSimilarityRatio:     85,
	DefaultUserAgent:         "httpx - Open-source project (github.com/projectdiscovery/httpx)",
}

DefaultOptions contains the default options

Functions

func DecodeData added in v1.2.3 

func DecodeData(data []byte, headers http.Header) ([]byte, error)

ExtractTitle from a response

func DecodeKorean added in v1.2.0 

func DecodeKorean(s []byte) ([]byte, error)

func Decodebig5 added in v0.0.7 

func Decodebig5(s []byte) ([]byte, error)

Decodebig5 converts BIG5 to UTF-8

func Decodegbk added in v0.0.7 

func Decodegbk(s []byte) ([]byte, error)

Decodegbk converts GBK to UTF-8

func Encodebig5 added in v0.0.7 

func Encodebig5(s []byte) ([]byte, error)

Encodebig5 converts UTF-8 to BIG5

func ExtractTitle 

func ExtractTitle(r *Response) (title string)

ExtractTitle from a response

Types

type CSPData added in v1.0.3 

type CSPData struct {
	Domains []string `json:"domains,omitempty"`
}

CSPData contains the Content-Security-Policy domain list

type ChainItem added in v1.0.6 

type ChainItem struct {
	Request    string `json:"request,omitempty"`
	Response   string `json:"response,omitempty"`
	StatusCode int    `json:"status_code,omitempty"`
	Location   string `json:"location,omitempty"`
	RequestURL string `json:"request-url,omitempty"`
}

ChainItem request=>response

type CustomCallback 

type CustomCallback func(response *Response) (bool, error)

CustomCallback used in custom filters

type Filter 

type Filter interface {
	Filter(response *Response) (bool, error)
}

Filter defines a generic filter interface to apply to responses

type FilterCustom 

type FilterCustom struct {
	CallBacks []CustomCallback
}

FilterCustom defines a filter with callback functions applied

func (FilterCustom) Filter 

func (f FilterCustom) Filter(response *Response) (bool, error)

Filter a response with custom callbacks

type FilterRegex 

type FilterRegex struct {
	Regexs []string
}

FilterRegex defines a filter of type regex

func (FilterRegex) Filter 

func (f FilterRegex) Filter(response *Response) (bool, error)

Filter a response with regexes

type FilterString 

type FilterString struct {
	Keywords []string
}

FilterString defines a filter of type string

func (FilterString) Filter 

func (f FilterString) Filter(response *Response) (bool, error)

Filter a response with strings filtering

type HTTPX 

type HTTPX struct {
	Filters []Filter
	Options *Options

	CustomHeaders map[string]string

	Dialer *fastdialer.Dialer
	// contains filtered or unexported fields
}

HTTPX represent an instance of the library client

func New 

func New(options *Options) (*HTTPX, error)

New httpx instance

func (*HTTPX) AddFilter 

func (h *HTTPX) AddFilter(f Filter)

AddFilter cascade

func (*HTTPX) CSPGrab added in v1.0.3 

func (h *HTTPX) CSPGrab(r *Response) *CSPData

CSPGrab fills the CSPData

func (*HTTPX) CdnCheck added in v1.0.3 

func (h *HTTPX) CdnCheck(ip string) (bool, string, error)

CdnCheck verifies if the given ip is part of Cdn ranges

func (*HTTPX) Do 

func (h *HTTPX) Do(req *retryablehttp.Request, unsafeOptions UnsafeOptions) (*Response, error)

Do http request

func (*HTTPX) IsVirtualHost 

func (h *HTTPX) IsVirtualHost(req *retryablehttp.Request, unsafeOptions UnsafeOptions) (bool, error)

IsVirtualHost checks if the target endpoint is a virtual host

func (*HTTPX) NewRequest 

func (h *HTTPX) NewRequest(method, targetURL string) (req *retryablehttp.Request, err error)

NewRequest from url

func (*HTTPX) NewRequestWithContext added in v1.1.4 

func (h *HTTPX) NewRequestWithContext(ctx context.Context, method, targetURL string) (req *retryablehttp.Request, err error)

NewRequest from url

func (*HTTPX) SetCustomHeaders 

func (h *HTTPX) SetCustomHeaders(r *retryablehttp.Request, headers map[string]string)

SetCustomHeaders on the provided request

func (*HTTPX) SupportHTTP2 added in v1.0.0 

func (h *HTTPX) SupportHTTP2(protocol, method, targetURL string) bool

SupportHTTP2 checks if the target host supports HTTP2

func (*HTTPX) SupportPipeline added in v1.0.0 

func (h *HTTPX) SupportPipeline(protocol, method, host string, port int) bool

SupportPipeline checks if the target host supports HTTP1.1 pipelining by sending x probes and reading back responses expecting at least 2 with HTTP/1.1 or HTTP/1.0

func (*HTTPX) TLSGrab added in v1.0.3 

func (h *HTTPX) TLSGrab(r *http.Response) *clients.Response

TLSGrab fills the TLSData

func (*HTTPX) Verify 

func (h *HTTPX) Verify(req *retryablehttp.Request, unsafeOptions UnsafeOptions) (bool, error)

Verify the http calls and apply-cascade all the filters, as soon as one matches it returns true

type Options 

type Options struct {
	RandomAgent      bool
	DefaultUserAgent string
	HTTPProxy        string
	SocksProxy       string
	Threads          int
	CdnCheck         bool
	ExcludeCdn       bool
	// Timeout is the maximum time to wait for the request
	Timeout time.Duration
	// RetryMax is the maximum number of retries
	RetryMax      int
	CustomHeaders map[string]string
	// VHostSimilarityRatio 1 - 100
	VHostSimilarityRatio int
	FollowRedirects      bool
	FollowHostRedirects  bool
	MaxRedirects         int
	Unsafe               bool
	TLSGrab              bool
	ZTLS                 bool
	// VHOSTs options
	VHostIgnoreStatusCode     bool
	VHostIgnoreContentLength  bool
	VHostIgnoreNumberOfWords  bool
	VHostIgnoreNumberOfLines  bool
	VHostStripHTML            bool
	Allow                     []string
	Deny                      []string
	MaxResponseBodySizeToSave int64
	MaxResponseBodySizeToRead int64
	UnsafeURI                 string
	Resolvers                 []string

	SniName string
	// contains filtered or unexported fields
}

Options contains configuration options for the client

type Response 

type Response struct {
	StatusCode    int
	Headers       map[string][]string
	RawData       []byte // undecoded data
	Data          []byte // decoded data
	ContentLength int
	Raw           string
	RawHeaders    string
	Words         int
	Lines         int
	TLSData       *clients.Response
	CSPData       *CSPData
	HTTP2         bool
	Pipeline      bool
	Duration      time.Duration
	Chain         []httputil.ChainItem
}

Response contains the response to a server

func (*Response) GetChain added in v1.0.6 

func (r *Response) GetChain() string

GetChain dump the whole redirect chain as string

func (*Response) GetChainAsSlice added in v1.0.6 

func (r *Response) GetChainAsSlice() (chain []ChainItem)

GetChainAsSlice dump the whole redirect chain as structuerd slice

func (*Response) GetChainLastURL added in v1.0.7 

func (r *Response) GetChainLastURL() string

GetChainLastURL returns the final URL

func (*Response) GetChainStatusCodes added in v1.0.6 

func (r *Response) GetChainStatusCodes() []int

GetChainStatusCodes from redirects

func (*Response) GetHeader added in v0.0.3 

func (r *Response) GetHeader(name string) string

GetHeader value

func (*Response) GetHeaderPart added in v0.0.7 

func (r *Response) GetHeaderPart(name, sep string) string

GetHeaderPart with offset

func (*Response) HasChain added in v1.0.6 

func (r *Response) HasChain() bool

HasChain redirects

type Target added in v1.2.2 

type Target struct {
	Host       string
	CustomHost string
	CustomIP   string
}

Target of the scan with ip|host header customization

type UnsafeOptions added in v1.1.2 

type UnsafeOptions struct {
	URIPath string
}

RequestOverride contains the URI path to override the request

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.