utils

package
v1.0.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 18, 2024 License: MIT Imports: 22 Imported by: 18

Documentation

Index

Constants

View Source 
const (
	ReuseTransportAndClient = iota // Reuse client (maintaining cookies), reuse transport (keeping connections)
	ReuseTransport                 // Create new client (discarding cookies), reuse transport (keeping connections)
	ReuseNone                      // Create new client (discarding cookies) and transport (not keeping connections)
)
View Source 
const (
	// Waiting states
	StatusWaiting = "Waiting" // Default value of sql attribute on create, if not defined otherwise

	// Active states
	StatusRunning = "Running" // Scan is in progress

	// Success states
	StatusCompleted    = "Completed"               // Scan ran through without significant issues
	StatusDeadline     = "Completed With Deadline" // Deadline (scan timeout) reached
	StatusNotReachable = "Not Reachable"           // Connection/Socket error, target might not be online anymore
	StatusSkipped      = "Skipped"                 // Target might be on blacklist and not scanned

	// Error states
	StatusFailed     = "Failed"      // Scan crashed or vanished (e.g. agent restart, agent keyboard interrupt)
	StatusProxyError = "Proxy Error" // Proxy error might be suspicious, if a proxy is configured (web enum/crawler)
)

Variables

View Source 
var ClientFactory = func(transport *http.Transport, timeout time.Duration) *http.Client {

	return &http.Client{
		Transport: transport,
		Timeout:   timeout * 2,
	}
}

ClientFactory is a basic client factory that can be passed to the requester

View Source 
var InsecureTransportFactory = func(proxy *url.URL, timeout time.Duration) *http.Transport {

	return &http.Transport{
		TLSClientConfig:       InsecureTlsConfigFactory(),
		TLSHandshakeTimeout:   timeout,
		ResponseHeaderTimeout: timeout * 2,
		DisableKeepAlives:     false,
		Proxy:                 http.ProxyURL(proxy),
	}
}

InsecureTransportFactory is a basic transport factory that can be passed to the requester. This transport factory returns a transport with insecure TLS configuration. It is intended for scanning purposes and not recommended to establish trusted connections!

Functions

func AppendUnique 

func AppendUnique(slice []string, elements ...string) []string

AppendUnique adds elements to an existing slice which are not contained yet. If the original slice already has duplicates in it, they will remain.

func DeadlineReached 

func DeadlineReached(deadline time.Time) bool

DeadlineReached checks whether a given deadline has been reached. Returns false if deadline is zero-time.

func Equals 

func Equals(s1 []string, s2 []string) bool

Equals checks if two slices of strings contain the same elements and also the same amount of those, but with no regard to their order []string{"a","a","c"} == []string{"c","a","c"} >>> false []string{"z","z","x"} == []string{"x","z","z"} >>> true

func Execute 

func Execute(cmd string, args []string) error

Execute command without returning output. Just returning potential errors

func ExtractHostPort 

func ExtractHostPort(target *url.URL) (string, int)

ExtractHostPort extracts host and port from a given URL. If no port is specified the protocol defaults are returned.

func ExtractHtmlTitle 

func ExtractHtmlTitle(body []byte) string

ExtractHtmlTitle parses HTML content and extracts the HTML title

func Filter 

func Filter(input []string, filter func(string) bool) []string

Filter filters slice for elements where the given function returns true

func FormattedHeader 

func FormattedHeader(headers http.Header) string

FormattedHeader iterates HTTP response headers, sorts them and joins them to a newline separated string

func GetSubjectAlternativeNames 

func GetSubjectAlternativeNames(address string, port int, dialTimeout time.Duration) ([]string, error)

GetSubjectAlternativeNames Connects to SSL endpoint and extracts subject name and subject alternative names from the SSL certificate. This function does not check whether the peer certificate is a CA.

func HashSha1 

func HashSha1(data []byte, separator string) string

Returns the sha1 of a byte sequence

func HttpsIndicated 

func HttpsIndicated(resp *http.Response, respBody []byte) bool

HttpsIndicated indicates whether the wrong HTTP protocol (HTTP over HTTPS) might have been used. There are some web servers that allow HTTP connections to HTTPS ports, but indicate an error.

func InsecureTlsConfigFactory 

func InsecureTlsConfigFactory() *tls.Config

InsecureTlsConfigFactory returns an *INSECURE* SSL connection configuration allowing any supported SSL protocol, and skipping SSL verification routines. This configuration is intended to scan modules and may not be used for user interfaces!

func IsElevated 

func IsElevated() bool

IsElevated checks whether the current process is running with admin privileges on Linux

func IsValidAddress 

func IsValidAddress(s string) bool

IsValidAddress determines whether a given string is a valid IPv4, IPv6 or hostname, but NOT a network range

func IsValidExecutable 

func IsValidExecutable(path string, args ...string) error

IsValidExecutable checks whether a given path can be executed

func IsValidFile 

func IsValidFile(path string) error

IsValidFile checks whether a given path is existing and a file

func IsValidFolder 

func IsValidFolder(path string) error

IsValidFolder checks whether a given path is existing and a folder

func IsValidHostname 

func IsValidHostname(hostname string) bool

IsValidHostname determines whether a given hostname is a plausible one

func IsValidIp 

func IsValidIp(s string) bool

IsValidIp determines whether a given string is a valid IPv4/IPv6 address

func IsValidIpRange 

func IsValidIpRange(s string) bool

IsValidIpRange determines whether a given string is a valid network range

func IsValidIpV4 

func IsValidIpV4(s string) bool

IsValidIpV4 determines whether a given string is a valid IPv4 address

func IsValidIpV6 

func IsValidIpV6(s string) bool

IsValidIpV6 determines whether a given string is a valid IPv6 address

func Map 

func Map(slice []string, fn func(string) string) []string

Map applies a manipulation function to each element of a slice

func ProxyStringToUrl 

func ProxyStringToUrl(proxy string) (*url.URL, error)

func ReadBody 

func ReadBody(response *http.Response) (body []byte, encoding string, err error)

ReadBody detects the response's content encoding and returns accordingly decoded response body bytes. The response body might arrive arbitrary encoding. The response's encoding is detected from different sources (Content-Type response header, BOMs, HTML meta tag, RFC defaults,...)

func RemoveFromSlice 

func RemoveFromSlice(list []string, s string) []string

RemoveFromSlice removes a given element (and potential duplicates) from a slice and returns a new slice

func ResolvesToHostname 

func ResolvesToHostname(ip string, hostname string) bool

ResolvesToHostname checks whether a given IP reverse resolves to the expected hostname

func ResolvesToIp 

func ResolvesToIp(hostname string, expectedIp string) bool

ResolvesToIp resolves a given DNS name and checks whether the result matches the expected IP address.

func Reverse 

func Reverse(input []string)

Reverse orders slice in reverse order

func SameEndpoint 

func SameEndpoint(url *url.URL, endpointIp string, endpointPort int) bool

SameEndpoint detects whether a given URL is pointing to the given IP and port. SameEndpoint will return true as long as it is resolving to the given host/port. If the endpointIp is empty or endpointPort is -1 the respective value will not be checked.

func SameScope 

func SameScope(urlToCheck *url.URL, referenceUrl *url.URL) bool

SameScope detects whether a given URL has the same endpoint (host + port) as the given reference URL. SameScope will always return true as long as it is pointing to the same host/port as the reference URL.

func SanitizeFilename 

func SanitizeFilename(raw string, placeholder string) string

SanitizeFilename takes a desired file name and converts characters not allowed by the filesystem

func Shuffle 

func Shuffle(strings []string) []string

Shuffle randomizes slice of strings

func StacktraceIndented 

func StacktraceIndented(indent string) string

Takes the stacktrace from stack and formats it in a nicely indented way (starting with newline): 

Stacktrace:
	| goroutine 2775398 [running]:
	| go-scans/utils.(*Requester).Get(0xc001511810, 0xc001e39740, 0x2a, 0xc0001b2460, 0x20, 0x0, 0xe, 0xc0003063a0, 0xc, 0x0, ...)
	| 	C:/workplace/go/src/go-scans/scans/http.go:228 +0x2ac
	| go-scans/scans/webcrawler.(*Scanner).execute(0xc0006a89a0, 0x0)
	| 	C:/workplace/go/src/go-scans/scans/webcrawler/webcrawler.go:334 +0x5c3
	| go-scans/scans/webcrawler.(*Scanner).Run(0xc0006a89a0, 0xd18c2e28000, 0x0)
	| 	C:/workplace/go/src/go-scans/scans/webcrawler/webcrawler.go:230 +0x1ca
	| go-scans/agent/core.DoWebcrawler(0xc000272240, 0xb51e8, 0xc0001b2460, 0x20, 0x50, 0xc00317b270, 0x5, 0x5, 0xc001e5bb9c, 0x4, ...)
	| 	C:/workplace/go/src/go-scans/agent/core/core_webcrawler.go:101 +0x8ec
	| created by go-scans/agent/core.scanTaskLauncher
	| 	C:/workplace/go/src/go-scans/agent/core/core.go:323 +0xb28

func StrContained 

func StrContained(candidate string, slices ...[]string) bool

StrContained checks whether a given (exact) value is contained within one or multiple given slices

func SubstrContained 

func SubstrContained(candidate string, slices ...[]string) bool

SubstrContained checks whether a given substring can be found within the strings within the given slices. This function is like StrContained but not looking for *exact* matches.

func TitleFirstLetter 

func TitleFirstLetter(s string) string

TitleFirstLetter makes the fist letter (and only the first letter) of the string uppercase

func TrimToLower 

func TrimToLower(slice []string) []string

TrimToLower converts slice elements to lower case and trim whitespaces

func UniqueStrings 

func UniqueStrings(elements []string) []string

UniqueStrings gets rid of duplicate entries in the slice

func UrlToRelative 

func UrlToRelative(path string) string

UrlToRelative can be fed with a relative, absolute or garbage path and will try to convert it into a relative one An absolute path will be converted into a relative one. A relative path will be returned but without leading slash. Garbage input is interpreted as a relative path and be returned the same way without leading slash.

func ValidOrEmptyCredentials 

func ValidOrEmptyCredentials(domain string, user string, password string) bool

Types

type HttpFingerprint 

type HttpFingerprint struct {
	RespUrl      string
	ResponseCode int
	HtmlTitle    string
	HtmlLen      int
}

HttpFingerprint holds defining attributes of an HTTP response. These attributes can be used to compare different HTTP responses for being equal

func NewHttpFingerprint 

func NewHttpFingerprint(respUrl string, responseCode int, htmlTitle string, htmlContent string) *HttpFingerprint

NewHttpFingerprint creates a new HTTP fingerprint definition

func (*HttpFingerprint) KnownIn 

func (f *HttpFingerprint) KnownIn(knownFingerprints map[string]*HttpFingerprint, lengthVariability int) (string, bool)

KnownIn checks whether the fingerprint is already part of a list of fingerprints

func (*HttpFingerprint) Similar 

func (f *HttpFingerprint) Similar(f2 *HttpFingerprint, lengthVariability int) bool

Similar compares two HTTP fingerprints for being similar. Response URL, code and HTML title must match, while HTML content length need to be close by the defined threshold

func (*HttpFingerprint) String 

func (f *HttpFingerprint) String() string

String converts a fingerprint to its string representation

type Logger 

type Logger interface {
	Debugf(format string, v ...interface{})
	Infof(format string, v ...interface{})
	Warningf(format string, v ...interface{})
	Errorf(format string, v ...interface{})
}

Logger defines a minimum logger interface. This way the maximum flexibility in supported loggers can be offered. If your chosen logger does not implement one of the functions required by this interface, you can wrap it and append the missing exported function, redirecting to the original loggers suitable one.

type Requester 

type Requester struct {
	// contains filtered or unexported fields
}

Requester allows to comfortably initialize an HTTP requester that can be reused for multiple request and does automatically take care of resetting the HTTP client or transport if desired. Furthermore, it reduces the amount of boilerplate code and the amount of arguments that need to be passed for each HTTP request. Furthermore, this requester does automatically take care of NTLM authentication if credentials are provided. The requester's .Get() method is thread safe.

func NewRequester 

func NewRequester(
	reuseMode int,
	userAgent string,
	ntlmDomain string,
	ntlmUser string,
	ntlmPassword string,
	proxy *url.URL,
	timeout time.Duration,
	transportFactory func(proxy *url.URL, timeout time.Duration) *http.Transport,
	clientFactory func(transport *http.Transport, timeout time.Duration) *http.Client,
) *Requester

NewRequester returns a reusable and thread safe HTTP requester, which can automatically take care of reusing or resetting the underlying HTTP client/transport. Furthermore, the requester will automatically take care of NTLM authentication if required.

func (*Requester) Get 

func (r *Requester) Get(url_ string, vhost string) (resp *http.Response, redirects int, auth string, err error)

Get executes an HTTP GET request, following potential location redirects, watching out for authentication-required responses and automatically trying to authenticate if credentials are set. This method is thread safe and can be called from different goroutines, because the underlying *http.Client is. ATTENTION: The caller must take care of closing a successful response body ATTENTION: The transport connection can only be reused once the response body got read and closed.

func (*Requester) GetCookies 

func (r *Requester) GetCookies(url_ string) []*http.Cookie

GetCookies returns the cookies currently stored in a reusable HTTP client. This does only return cookies, if the requester's operation mode is reusing clients and if the used client has an initialized cookie jar.

type TaggedLogger 

type TaggedLogger struct {
	Logger
	// contains filtered or unexported fields
}

TaggedLogger is a small wrapper for the Logger interface, that allows to add an additional tag before every message. It should mainly be used to group information from different worker routines.

func NewTaggedLogger 

func NewTaggedLogger(logger Logger, tag string) *TaggedLogger

func (*TaggedLogger) Debugf 

func (l *TaggedLogger) Debugf(format string, v ...interface{})

func (*TaggedLogger) Errorf 

func (l *TaggedLogger) Errorf(format string, v ...interface{})

func (*TaggedLogger) Infof 

func (l *TaggedLogger) Infof(format string, v ...interface{})

func (*TaggedLogger) Warningf 

func (l *TaggedLogger) Warningf(format string, v ...interface{})

type TestLogger 

type TestLogger struct {
	*log.Logger
	// contains filtered or unexported fields
}

TestLogger wraps the default golang logger and extends it with the functions required to implement the Logger interface.

func NewTestLogger 

func NewTestLogger() *TestLogger

NewTestLogger returns a new standard golang logger compliant with the Logger interface

func (*TestLogger) Debugf 

func (l *TestLogger) Debugf(format string, v ...interface{})

func (*TestLogger) Errorf 

func (l *TestLogger) Errorf(format string, v ...interface{})

func (*TestLogger) Infof 

func (l *TestLogger) Infof(format string, v ...interface{})

func (*TestLogger) Warningf 

func (l *TestLogger) Warningf(format string, v ...interface{})

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.