mpsl

README

Mozilla Public Suffix List database

MPSL is a domain parser based on Mozilla's public suffix list.

Domain to parse may contain the following parts:

• Top level domain (TLD) - the last part of the domain that doesn't contain a dots (example: com, org, ...).
• Effective TLD (eTLD) - the last part that contains arbitrary count of the dots (example: co.uk, cdn.prod.atlassian-dev.net, ...).
• Effective TLD plus one (eTLD+1) - eTLD + domain name (or first subdomain).
• ICANN flag - is TLD/eTLD provided by public ICANN list.

This package provides a possibility to get all parts at once or get them separately by call corresponding methods (see below).

Installation

The package requires access to pre-downloaded public suffix database or URL to the list file to fetch it on the fly (preferred way is to give URL to full PSL database).

There is three ways to initialize the database:

db, err := mpsl.New(hasher)
// way 1: load from local file
err = db.Load("/path/to/psl/file")
// way 2: fetch from remote URL
err = db.Fetch("http://url/to/psl/file") // or FetchFull() to fetch official PSL file.
// way 3: load from local file (if file's mod time is less than expire) or fetch from remote URL (and save result to the local file).
err = db.LoadOrFetchIf("/path/to/psl/file", "http://url/to/psl/file", time.Hour * 24) // or LoadOrFetchFullIf(...).

All PSL data will be stored in special storage optimized for fast access and minimal pointers count (three pointers in fact).

Usage

After initialization database will ready to parse the domains:

domain := "a.b.c.amazon.co.uk"
tld, icann := db.GetTLDStr(domain) // "uk", true
etld := db.GetEffectiveTLDStr(domain) // "co.uk"
etld1 := db.GetEffectiveTLDPlusOneStr(domain) // "amazon.co.uk"
// or get all parts at once
tld, etld, etld1, icann := db.ParseStr(domain) // the same data ...

Documentation

Index

• Constants
• Variables
• type DB
  • func New(hasher hash.BHasher) (*DB, error)
  • func (db *DB) Fetch(dbURL string) (err error)
  • func (db *DB) FetchFull() error
  • func (db *DB) GetETLD(hostname []byte) []byte
  • func (db *DB) GetETLD1(hostname []byte) []byte
  • func (db *DB) GetETLD1Str(hostname string) string
  • func (db *DB) GetETLDStr(hostname string) string
  • func (db *DB) GetEffectiveTLD(hostname []byte) (etld []byte)
  • func (db *DB) GetEffectiveTLDPlusOne(hostname []byte) (etld1 []byte)
  • func (db *DB) GetEffectiveTLDPlusOneStr(hostname string) (etld1 string)
  • func (db *DB) GetEffectiveTLDStr(hostname string) (etld string)
  • func (db *DB) GetTLD(hostname []byte) (tld []byte, icann bool)
  • func (db *DB) GetTLDStr(hostname string) (tld string, icann bool)
  • func (db *DB) Load(dbFile string) (err error)
  • func (db *DB) LoadOrFetchFullIf(dbFile string, expire time.Duration) error
  • func (db *DB) LoadOrFetchIf(dbFile, dbURL string, expire time.Duration) error
  • func (db *DB) Parse(hostname []byte) (tld, etld, etld1 []byte, icann bool)
  • func (db *DB) ParseStr(hostname string) (tld, etld, etld1 string, icann bool)
  • func (db *DB) Reset()

Constants

const (
	// FullURL of full known PSL data.
	FullURL = "https://raw.githubusercontent.com/publicsuffix/list/master/public_suffix_list.dat"
)

Variables

var (
	ErrBadDB    = errors.New("cache uninitialized, use New()")
	ErrNoHasher = errors.New("no hasher provided")
)

Types

type DB

type DB struct {
	// contains filtered or unexported fields
}

DB is an implementation of Mozilla Public Suffix List database.

func New

func New(hasher hash.BHasher) (*DB, error)

New makes new instance of the DB.

func (*DB) Fetch

func (db *DB) Fetch(dbURL string) (err error)

Fetch loads PSL data from given URL.

func (*DB) FetchFull

func (db *DB) FetchFull() error

FetchFull loads full known PSL data from publicsuffix.org github profile.

func (*DB) GetETLD

func (db *DB) GetETLD(hostname []byte) []byte

GetETLD is a shorthand alias of GetEffectiveTLD.

func (*DB) GetETLD1

func (db *DB) GetETLD1(hostname []byte) []byte

GetETLD1 is a shorthand alias of GetEffectiveTLDPlusOne.

func (*DB) GetETLD1Str

func (db *DB) GetETLD1Str(hostname string) string

GetETLD1Str is a shorthand alias of GetEffectiveTLDPlusOneStr.

func (*DB) GetETLDStr

func (db *DB) GetETLDStr(hostname string) string

GetETLDStr is a shorthand alias of GetEffectiveTLDStr.

func (*DB) GetEffectiveTLD

func (db *DB) GetEffectiveTLD(hostname []byte) (etld []byte)

GetEffectiveTLD returns only eTLD part of hostname.

func (*DB) GetEffectiveTLDPlusOne

func (db *DB) GetEffectiveTLDPlusOne(hostname []byte) (etld1 []byte)

GetEffectiveTLDPlusOne return only eTLD1 part of hostname.

func (*DB) GetEffectiveTLDPlusOneStr

func (db *DB) GetEffectiveTLDPlusOneStr(hostname string) (etld1 string)

GetEffectiveTLDPlusOneStr return only eTLD1 part of string hostname.

func (*DB) GetEffectiveTLDStr

func (db *DB) GetEffectiveTLDStr(hostname string) (etld string)

GetEffectiveTLDStr returns only eTLD part of string hostname.

func (*DB) GetTLD

func (db *DB) GetTLD(hostname []byte) (tld []byte, icann bool)

GetTLD returns TLD part of hostname and ICANN flag.

func (*DB) GetTLDStr

func (db *DB) GetTLDStr(hostname string) (tld string, icann bool)

GetTLDStr returns TLD part of string hostname and ICANN flag.

func (*DB) Load

func (db *DB) Load(dbFile string) (err error)

Load extracts PSL data from the local file.

func (*DB) LoadOrFetchFullIf

func (db *DB) LoadOrFetchFullIf(dbFile string, expire time.Duration) error

LoadOrFetchFullIf is similar to LoadOrFetchIf but loads data from full URL.

func (*DB) LoadOrFetchIf

func (db *DB) LoadOrFetchIf(dbFile, dbURL string, expire time.Duration) error

LoadOrFetchIf loads PSL data from local file (if file is younger than expire param) or load data from given URL.

After fetching from URL raw PSL data will be stored to local file (if possible).

func (*DB) Parse

func (db *DB) Parse(hostname []byte) (tld, etld, etld1 []byte, icann bool)

Parse parses hostname to separate parts: TLD, eTLD, eTLD1 and ICANN flag.

func (*DB) ParseStr

func (db *DB) ParseStr(hostname string) (tld, etld, etld1 string, icann bool)

ParseStr parses hostname string to separate parts: TLD, eTLD, eTLD1 and ICANN flag.

func (*DB) Reset

func (db *DB) Reset()

Reset flushes all database data.