urlutil

package
v0.0.45 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 22, 2023 License: MIT Imports: 11 Imported by: 0

README

urlutil

The package contains various helpers to interact with URLs

difference b/w net/url.URL and utils/url/URL

  • url.URL caters to variety of urls and for that reason its parsing is not that accurate under various conditions

  • utils/url/URL is a wrapper around url.URL that handles below edgecases and is able to parse complex (i.e non-RFC compilant urls but required in infosec) url edgecases.

  • url.URL allows u.Path without / prefix but it is not allowed in utils/url/URL and is autocorrected if / prefix is missing

  • Parsing URLs without scheme

// if below urls are parsed with url.Parse(). url parts(scheme,host,path etc) are not properly classified
scanme.sh
scanme.sh:443/port
scame.sh/with/path

  • Encoding of parameters(url.Values)

    • url.URL encodes all reserved characters(as per RFC(s)) in parameter key-value pair (i.e url.Values{})
    • If reserved/special characters are url encoded then integrity of specially crafted payloads (lfi,xss,sqli) is lost.
    • utils/url/URL uses utils/url/Params to store/handle parameters and integrity of all such payload is preserved
    • utils/url/URL also provides options to customize url encoding using global variable and function params

  • Parsing Unsafe/Invalid Paths

    • while parsing urls url.Parse() either discards or re-encodes some of the specially crafted payloads
    • If a non valid url encoding is given in url (ex: scanme.sh/%invalid) url.Parse() returns error and url is not parsed
    • Such cases are implicitly handled if unsafe is true
// Example urls for above condition
scanme.sh/?some'param=`'+OR+ORDER+BY+1--
scanme.sh/?some[param]=<script>alert(1)</script>
scanme.sh/%invalid/path
  • utils/url/URL has some extra methods
    • .TrimPort()
    • .MergePath(newrelpath string, unsafe bool)
    • .UpdateRelPath(newrelpath string, unsafe bool)
    • .Clone() and more
Note

utils/url/URL embeds url.URL and thus inherits and exposes all url.URL methods and variables. Its ok to use any method from url.URL (directly/indirectly) except url.URL.Query() and url.URL.String() (due to parameter encoding issues). In any case if it is not possible to follow above point (ex: directly updating/referencing http.Request.URL) .Update() method should be called before accessing them which updates url.URL instance for this edgecase. (Not required if above rule is followed)

Documentation

Index

Constants

View Source 
const (
	HTTP             = "http"
	HTTPS            = "https"
	SchemeSeparator  = "://"
	DefaultHTTPPort  = "80"
	DefaultHTTPSPort = "443"
)

Variables

View Source 
var AllowLegacySeperator bool = false

Legacy Seperator (i.e `;`) is used as seperator for parameters this was removed in go >=1.17

View Source 
var DisableAutoCorrect bool

disables autocorrect related to parsing Ex: if input is admin url.Parse considers admin as host which is not a valid domain name

View Source 
var MustEscapeCharSet []rune = []rune{'?', '#', '@', ';', '&', ',', '[', ']', '^'}

MustEscapeCharSet are special chars that are always escaped and are based on reserved chars from RFC Some of Reserved Chars From RFC were excluded and some were added for various reasons and goal here is to encode parameters key and value only

View Source 
var RFCEscapeCharSet []rune = []rune{'!', '*', '\'', '(', ')', ';', ':', '@', '&', '=', '+', '$', ',', '/', '?', '%', '#', '[', ']'}

Reserved Chars from RFC ! * ' ( ) ; : @ & = + $ , / ? % # [ ]

Functions

func AutoMergeRelPaths 

func AutoMergeRelPaths(path1 string, path2 string) (string, error)

AutoMergeRelPaths merges two relative paths including parameters and returns final string

func ParamEncode 

func ParamEncode(data string) string

ParamEncode encodes Key characters only. key characters include whitespaces + non printable chars + non-ascii also this does not double encode encoded characters

func PercentEncoding 

func PercentEncoding(data string) string

PercentEncoding encodes all characters to percent encoded format just like burpsuite decoder

func URLEncodeWithEscapes 

func URLEncodeWithEscapes(data string, charset ...rune) string

URLEncodeWithEscapes URL encodes data with given special characters escaped (similar to burpsuite intruder) Note `MustEscapeCharSet` is not included

Types

type OrderedParams 

type OrderedParams struct {
	// contains filtered or unexported fields
}

OrderedParams is a map that preserves the order of elements

func NewOrderedParams 

func NewOrderedParams() *OrderedParams

NewOrderedParams creates a new ordered params

func (*OrderedParams) Add 

func (o *OrderedParams) Add(key string, value ...string)

Add Parameters to store

func (*OrderedParams) Clone 

func (o *OrderedParams) Clone() *OrderedParams

Clone returns a copy of the ordered params

func (*OrderedParams) Decode 

func (o *OrderedParams) Decode(raw string)

Decode is opposite of Encode() where ("bar=baz&foo=quux") is parsed Parameters are loosely parsed to allow any scenario

func (*OrderedParams) Del 

func (o *OrderedParams) Del(key string)

Del deletes values associated with key

func (*OrderedParams) Encode 

func (o *OrderedParams) Encode() string

Encode returns encoded parameters by preserving order

func (*OrderedParams) Get 

func (o *OrderedParams) Get(key string) string

Get returns first value of given key

func (*OrderedParams) GetAll 

func (o *OrderedParams) GetAll(key string) []string

GetAll returns all values of given key or returns empty slice if key doesn't exist

func (*OrderedParams) Has 

func (o *OrderedParams) Has(key string) bool

Has returns if given key exists

func (*OrderedParams) IsEmpty 

func (o *OrderedParams) IsEmpty() bool

IsEmpty checks if the OrderedParams is empty

func (*OrderedParams) Iterate 

func (o *OrderedParams) Iterate(f func(key string, value []string) bool)

Iterate iterates over the OrderedParams

func (*OrderedParams) Merge 

func (o *OrderedParams) Merge(raw string)

Merges given paramset into existing one with base as priority

func (*OrderedParams) Set 

func (o *OrderedParams) Set(key string, value string)

Set sets the key to value and replaces if already exists

func (*OrderedParams) Update 

func (o *OrderedParams) Update(key string, value []string)

Update is similar to Set but it takes value as slice (similar to internal implementation of url.Values)

type Params 

type Params map[string][]string

func GetParams 

func GetParams(query url.Values) Params

GetParams return Params type using url.Values

func NewParams 

func NewParams() Params

func (Params) Add 

func (p Params) Add(key string, value ...string)

Add Parameters to store

func (Params) Decode 

func (p Params) Decode(raw string)

Decode is opposite of Encode() where ("bar=baz&foo=quux") is parsed Parameters are loosely parsed to allow any scenario

func (Params) Del 

func (p Params) Del(key string)

Del deletes values associated with key

func (Params) Encode 

func (p Params) Encode() string

Encode URL encodes and returns values ("bar=baz&foo=quux") sorted by key.

func (Params) Get 

func (p Params) Get(key string) string

Get returns first value of given key

func (Params) Has 

func (p Params) Has(key string) bool

Has returns if given key exists

func (Params) Merge 

func (p Params) Merge(x Params)

Merges given paramset into existing one with base as priority

func (Params) Set 

func (p Params) Set(key string, value string)

Set sets the key to value and replaces if already exists

type URL 

type URL struct {
	*url.URL

	Original   string         // original or given url(without params if any)
	Unsafe     bool           // If request is unsafe (skip validation)
	IsRelative bool           // If URL is relative
	Params     *OrderedParams // Query Parameters

}

URL a wrapper around net/url.URL

func Parse 

func Parse(inputURL string) (*URL, error)

ParseURL

func ParseRelativePath 

func ParseRelativePath(inputURL string, unsafe bool) (*URL, error)

ParseRelativePath parses and returns relative path

func ParseURL 

func ParseURL(inputURL string, unsafe bool) (*URL, error)

Parse and return URL

func (*URL) Clone 

func (u *URL) Clone() *URL

Clone

func (*URL) EscapedString 

func (u *URL) EscapedString() string

EscapedString returns a string that can be used as filename (i.e stripped of / and params etc)

func (*URL) GetRelativePath 

func (u *URL) GetRelativePath() string

GetRelativePath ex: /some/path?param=true#fragment

func (*URL) MergePath 

func (u *URL) MergePath(newrelpath string, unsafe bool) error

mergepath merges given relative path

func (*URL) Query 

func (u *URL) Query() *OrderedParams

Query returns Query Params

func (*URL) String 

func (u *URL) String() string

String

func (*URL) TrimPort 

func (u *URL) TrimPort()

TrimPort if any

func (*URL) Update 

func (u *URL) Update()

Updates internal wrapped url.URL with any changes done to Query Parameters

func (*URL) UpdatePort 

func (u *URL) UpdatePort(newport string)

Updates port

func (*URL) UpdateRelPath 

func (u *URL) UpdateRelPath(newrelpath string, unsafe bool) error

UpdateRelPath updates relative path with new path (existing params are not removed)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.