xurls

README

xurls

Extract urls from text using regular expressions. Requires Go 1.18 or later.

import "mvdan.cc/xurls/v2"

func main() {
	rxRelaxed := xurls.Relaxed()
	rxRelaxed.FindString("Do gophers live in golang.org?")  // "golang.org"
	rxRelaxed.FindString("This string does not have a URL") // ""

	rxStrict := xurls.Strict()
	rxStrict.FindAllString("must have scheme: http://foo.com/.", -1) // []string{"http://foo.com/"}
	rxStrict.FindAllString("no scheme, no match: foo.com", -1)       // []string{}
}

Since API is centered around regexp.Regexp, many other methods are available, such as finding the byte indexes for all matches.

The regular expressions are compiled when the API is first called. Any subsequent calls will use the same regular expression pointers.

cmd/xurls

To install the tool globally:

go install mvdan.cc/xurls/v2/cmd/xurls@latest

$ echo "Do gophers live in http://golang.org?" | xurls
http://golang.org

Documentation

Overview

Package xurls extracts urls from plain text using regular expressions.

Example

rx := xurls.Relaxed()
fmt.Println(rx.FindString("Do gophers live in http://golang.org?"))
fmt.Println(rx.FindAllString("foo.com is http://foo.com/.", -1))

Output:

http://golang.org
[foo.com http://foo.com/]

Index

• Variables
• func Relaxed() *regexp.Regexp
• func Strict() *regexp.Regexp
• func StrictMatchingScheme(exp string) (*regexp.Regexp, error)

Examples

• Package

Variables

var AnyScheme = `(?:[a-zA-Z][a-zA-Z.\-+]*://|` + anyOf(SchemesNoAuthority...) + `:)`

AnyScheme can be passed to StrictMatchingScheme to match any possibly valid scheme, and not just the known ones.

var PseudoTLDs = []string{}

PseudoTLDs is a sorted list of some widely used unofficial TLDs.

Sources:

• https://en.wikipedia.org/wiki/Pseudo-top-level_domain
• https://en.wikipedia.org/wiki/Category:Pseudo-top-level_domains
• https://tools.ietf.org/html/draft-grothoff-iesg-special-use-p2p-names-00
• https://www.iana.org/assignments/special-use-domain-names/special-use-domain-names.xhtml

var Schemes = []string{
	`http`,
	`https`}

Schemes is a sorted list of all IANA assigned schemes.

Source: https://www.iana.org/assignments/uri-schemes/uri-schemes-1.csv

var SchemesNoAuthority = []string{
	`bitcoin`,
	`cid`,
	`file`,
	`magnet`,
	`mailto`,
	`mid`,
	`sms`,
	`tel`,
	`xmpp`,
}

SchemesNoAuthority is a sorted list of some well-known url schemes that are followed by ":" instead of "://". The list includes both officially registered and unofficial schemes.

var SchemesUnofficial = []string{
	`gemini`,
	`jdbc`,
	`moz-extension`,
	`postgres`,
	`postgresql`,
	`slack`,
	`zoommtg`,
	`zoomus`,
}

SchemesUnofficial is a sorted list of some well-known url schemes which aren't officially registered just yet. They tend to correspond to software.

Mostly collected from https://en.wikipedia.org/wiki/List_of_URI_schemes#Unofficial_but_common_URI_schemes.

var TLDs = []string{
	`com`,
	`biz`,
	`org`,
	`net`,
	`us`,
	`uk`,
	`de`,
	`edu`}

TLDs is a sorted list of all public top-level domains.

Sources:

• https://data.iana.org/TLD/tlds-alpha-by-domain.txt
• https://publicsuffix.org/list/effective_tld_names.dat

Functions

func Relaxed

func Relaxed() *regexp.Regexp

Relaxed produces a regexp that matches any URL matched by Strict, plus any URL with no scheme or email address.

func Strict

func Strict() *regexp.Regexp

Strict produces a regexp that matches any URL with a scheme in either the Schemes or SchemesNoAuthority lists.

func StrictMatchingScheme

func StrictMatchingScheme(exp string) (*regexp.Regexp, error)

StrictMatchingScheme produces a regexp similar to Strict, but requiring that the scheme match the given regular expression. See AnyScheme too.