db

Documentation

Index

• Variables
• func Connect(uri string) error
• func Disconnect() error
• func Insert(d string) (bool, error)
• func InsertNotFound(d string) (bool, error)
• func InsertTopList(d string) (bool, error)
• func Lookup(d string, days int) ([]string, error)
• func LookupFull(d string, days int) ([]string, error)
• func RandomDomainUpdater(wg *sync.WaitGroup)
• func RecordsUpdate(d string, ignoreError bool, ignoreUpdated bool) error
• func RecordsUpdateUpdatedTime(d string) error
• func RecordsUpdatedRecently(d string) (bool, error)
• func RecordsUpdater()
• func ScannerGetIndex(name string) (int64, error)
• func ScannerUpdateIndex(name string, index int64) error
• func Starts(d string) ([]string, error)
• func TLD(d string) ([]string, error)
• func TopListUpdater(wg *sync.WaitGroup)
• type DomainSchema
  • func (d *DomainSchema) FullDomain() string
  • func (d *DomainSchema) String() string
• type FastDomainSchema
  • func (d *FastDomainSchema) String() string
• type NotFoundSchema
• type RecordSchema
  • func Records(d string, days int) ([]RecordSchema, error)
• type ScannerSchema
  • func GetStat() (total int64, updated int64, valid int64, scanners []ScannerSchema, err error)
  • func ScannerGetIndexes() ([]ScannerSchema, error)
  • func (s *ScannerSchema) UpdateSize() error
• type TopListSchema

Variables

var (
	Client *mongo.Client

	Domains  *mongo.Collection // The main collection to store the entries
	NotFound *mongo.Collection // Store domains that not found by Lookup
	TopList  *mongo.Collection // Store and count successful lookups
	Scanner  *mongo.Collection // Store informations about CT scanners
)

var (
	RecordsUpdaterDomainChan chan string
)

Functions

func Connect

func Connect(uri string) error

Connect connects to the database using the standard Connection URI.

func Disconnect

func Disconnect() error

Disconnect gracefully disconnect from the database.

func Insert

func Insert(d string) (bool, error)

Insert inserts the given domain d to the *domains* database. Checks if d is valid, do a Clean() and then splits into sub|domain|tld parts.

Returns true if d is new and inserted into the database. If domain is invalid, returns fault.ErrInvalidDomain. If failed to get parts of d (eg.: d is a TLD), returns ault.ErrGetPartsFailed.

NOTE: Use RecordsUpdate() after Insert()!

func InsertNotFound

func InsertNotFound(d string) (bool, error)

InsertNotFound inserts the given domain d to the *notFound* database. Checks if d is valid, do a Clean() and removes the subdomain from d.

Returns true if d is new and inserted into the database. If domain is invalid or failed to remove the subdomain, returns fault.ErrInvalidDomain.

func InsertTopList

func InsertTopList(d string) (bool, error)

InsertTopList inserts the given domain d to the *topList* database or increase the counter if exists. Checks if d is valid, do a Clean() and removes the subdomain from d.

Returns true if d is new and inserted into the database. If domain is invalid or failed to remove the subdomain, returns fault.ErrInvalidDomain.

func Lookup

func Lookup(d string, days int) ([]string, error)

Lookup validate, Clean() and query the DB and returns a list subdomains only. days specify, that the returned subdomain must had a valid record in the previous n days. If days is 0, return every subdomain that has a record regardless of the time. If days is -1, every subdomain returned, including domains that does not have a record.

If d has a subdomain, removes it before the query.

If d is invalid return fault.ErrInvalidDomain. If failed to get parts of d (eg.: d is a TLD), returns fault.ErrGetPartsFailed. If days if < -1, returns fault.ErrInvalidDays.

func LookupFull

func LookupFull(d string, days int) ([]string, error)

LookupFull validate, Clean() and query the DB and returns a list full domains with subdomain. days specify, that the returned subdomain must had a valid record in the previous n days. If days is 0, return every subdomain that has a record regardless of the time. If days is -1, every subdomain returned, including domains that does not have a record.

If d has a subdomain, removes it before the query.

If d is invalid return fault.ErrInvalidDomain. If failed to get parts of d (eg.: d is a TLD), returns ault.ErrGetPartsFailed. If days if < -1, returns fault.ErrInvalidDays.

func RandomDomainUpdater

func RandomDomainUpdater(wg *sync.WaitGroup)

RandomDomainUpdater is a function created to run as goroutine in the background. Select random entries (FQDNs) and send it to internalRecordsUpdaterDomainChan to update the records.

func RecordsUpdate

func RecordsUpdate(d string, ignoreError bool, ignoreUpdated bool) error

RecordsUpdate updates the records field for domain d if d is not update recently (in the previous hour). This function updates the "updated" field to the current time and the records in the database. If the same record found, updates the "time" field in element. If new record found, append it to the "records" field.

Checks if d is a wildcard record before update.

If ignoreError is true, common DNS errors are ignored. If ignoreUpdated is true, ignore when was the last update based on the "updated" timestamp.

If domain d is invalid, returns fault.ErrInvalidDomain. If failed to get parts of d (eg.: d is a TLD), returns fault.ErrGetPartsFailed.

func RecordsUpdateUpdatedTime

func RecordsUpdateUpdatedTime(d string) error

RecordsUpdateUpdatedTime updated the "updated" timestamp to the current time.

If d is invalid return fault.ErrInvalidDomain. If failed to get parts of d (eg.: d is a TLD), returns fault.ErrGetPartsFailed.

func RecordsUpdatedRecently

func RecordsUpdatedRecently(d string) (bool, error)

RecordsUpdatedRecently check whether domain d is updated recently (in the previous hour).

If d is invalid return fault.ErrInvalidDomain. If failed to get parts of d (eg.: d is a TLD), returns fault.ErrGetPartsFailed.

func RecordsUpdater

func RecordsUpdater()

func ScannerGetIndex

func ScannerGetIndex(name string) (int64, error)

ScannerGetIndex returns the index for scanner. The name is converted to lowercase.

func ScannerUpdateIndex

func ScannerUpdateIndex(name string, index int64) error

ScannerUpdateIndex updates the index for scanner. The name is converted to lowercase.

func Starts

func Starts(d string) ([]string, error)

Starts query the DB and returns a list of Second Level Domains (eg.: example) that starts with d.

Domain d must be a valid Second Level Domain (eg.: "example"). This function validate with IsValidSLD() and Clean().

Returns fault.ErrInvalidDomain is d is not a valid Second Level Domain.

func TLD

func TLD(d string) ([]string, error)

TLD query the DB and returns a list of TLDs for the given domain d.

Domain d must be a valid Second Level Domain (eg.: "example").

NOTE: This function not validate adn Clean() d!

func TopListUpdater

func TopListUpdater(wg *sync.WaitGroup)

TopListUpdater is a function created to run as goroutine in the background. Updates the domains and it subdomains in topList collection by sending every entries into internalRecordsUpdaterDomainChan. This function uses concurrent goroutines and print only/ignores any error.

Types

type DomainSchema

type DomainSchema struct {
	Domain  string         `bson:"domain" json:"domain"`
	TLD     string         `bson:"tld" json:"tld"`
	Sub     string         `bson:"sub" json:"sub"`
	Updated int64          `bson:"updated" json:"updated"`
	Records []RecordSchema `bson:"records,omitempty" json:"records,omitempty"`
}

Schema used in the "domains" collection.

func (*DomainSchema) FullDomain

func (d *DomainSchema) FullDomain() string

Returns the domain and tld only (eg.: domain.tld)

func (*DomainSchema) String

func (d *DomainSchema) String() string

Returns the full hostname (eg.: sub.domain.tld).

type FastDomainSchema

type FastDomainSchema struct {
	Domain string `bson:"domain" json:"domain"`
	TLD    string `bson:"tld" json:"tld"`
	Sub    string `bson:"sub" json:"sub"`
}

Schema used in Lookup() to ignore the Records field.

func (*FastDomainSchema) String

func (d *FastDomainSchema) String() string

Returns the full hostname (eg.: sub.domain.tld).

type NotFoundSchema

type NotFoundSchema struct {
	Domain string `bson:"domain" json:"domain"`
}

Schema used in *notFound* collection.

type RecordSchema

type RecordSchema struct {
	Type  uint16 `bson:"type" json:"type"`
	Value string `bson:"value" json:"value"`
	Time  int64  `bson:"time" json:"time"`
}

Schema used to store a record in DomainSchema

func Records

func Records(d string, days int) ([]RecordSchema, error)

Records query the DB and returns a list RecordSchema. days specify, that the returned record must be updated in the previous n days. If days is 0 or -1, return every record regardless of the time.

Returns records for the exact domain d.

If d is invalid return fault.ErrInvalidDomain. If failed to get parts of d (eg.: d is a TLD), returns ault.ErrGetPartsFailed. If days if < -1, returns fault.ErrInvalidDays.

type ScannerSchema

type ScannerSchema struct {
	Name  string `bson:"name" json:"name"`
	Index int64  `bson:"index" json:"index"`
	Size  int64  `bson:"-" json:"size"`
}

Schema used in "scanner" collection

func GetStat

func GetStat() (total int64, updated int64, valid int64, scanners []ScannerSchema, err error)

GetStat returns the total number of domains, the total number of updated domains and the total number of domains with valid record.

func ScannerGetIndexes

func ScannerGetIndexes() ([]ScannerSchema, error)

ScannerGetIndexes returns every entry from the "scanner" database.

func (*ScannerSchema) UpdateSize

func (s *ScannerSchema) UpdateSize() error

type TopListSchema

type TopListSchema struct {
	Domain string `bson:"domain" json:"domain"`
	Count  int    `bson:"count" json:"count"`
}

Schema used in *topList* collection.