backend

README

SQL-Setup

The SQL-Database consists of 8 tables and 5 views. A short explanation to them will be given here.

Tables

Scans-Table

The Scans-Table contains the Meta-Data for all scans. It has the following columns:

Table Columns

Column	Description
ScanID	The ID is an auto-increment, that specifies a Scan
SSLLabs	A boolean, that specifies if SSLLabs was/is used for this scan
SSLLabsVersion	The Version number of the table used for the SSLLabs-Results
Observatory	A boolean, that specifies if Observatory was/is used for this scan
ObservatoryVersion	The Version number of the table used for the Observatory-Results
SecurityHeaders	A boolean, that specifies if SecurityHeaders was/is used for this scan
SecurityHeadersVersion	The Version number of the table used for the SecurityHeaders-Results
Crawler	A boolean, that specifies if Crawler was/is used for this scan
CrawlerVersion	The Version number of the table used for the Crawler-Results
Unreachable	Number of hosts, that were not reachable
Total	Total number of hosts
Done	Boolean, implicating if the scan is finished
StartTime	Timestamp (Server Time) when the Scan was started
Config	The used API-Configurations in json-Format

Domains-Table

The Domains-Table contains all Domains that were scanned and is used to sort them in different lists. It has the following columns:

Table Columns

Column	Description
DomainID	The ID is an auto-increment, that specifies a Domain
DomainName	The host/domain name
ListID	An identifier to specify to which List a Domain belongs
isActive	A boolean, that specifies if a domain is active. Inactive domains are not scanned
nextScan	A boolean, that specifies if a domain is going to be in the next scan
CreationDate	A timestamp (Server Time) of the point in time the domain was added to the table

Unreachable-Table

The Unreachable-Table contains all Domains that were unreachable during a scan. It has the following columns:

Table Columns

Column	Description
ScanID	ID specifying the scan
DomainID	ID specifying the Domain
DNSError	A boolean, that specifies if the domain was unreachable due to a DNS-Lookup error (No such host!)

API-Tables

Table	Description
Crawler-Table	Description
Observatory-Table	Description
SecurityHeaders-Table	Description
SSLLabs-Table	Description
Certificates-Table	Description

Views

TestAggregation-View

This view combines the results from all APIs per Scan and Domain.

CompleteResults-View

This view just adds the DomainName to each entry in the TestAggregation-View.

CompleteGrades-View

This view just shows the Grades of each Test for each entry in the CompleteResults-View.

LatestResults-View

This view just shows the entries of the CompleteResults-View for the newest Scan.

LatestGrades-View

This view just shows the entries of the CompleteGrades-View for the newest Scan.

Documentation

Index

• Variables
• func ActiveDomainsWithDomain(active bool, domain string) error
• func ActiveDomainsWithListID(active bool, ListID string) error
• func GetConflictingDomains(domains []string, listID string) map[string]hooks.DomainsRowMetaInfo
• func GetDomains() ([]hooks.DomainsRow, error)
• func GetLastScan(id int) (hooks.ScanRow, error)
• func GetScans(table string, scanID int, scanStatus uint8) ([]hooks.DomainsReachable, error)
• func InsertNewScan(scan hooks.ScanRow) (hooks.ScanRow, error)
• func InsertScanData(tables []string, scanData []hooks.ScanData) error
• func OpenDatabase(conf SQLConfiguration) error
• func PrepareScanData(table string, scanID int, scanType int) error
• func RemoveDomainsWithDomain(domain string) error
• func RemoveDomainsWithDomainAndList(domain string, list string) error
• func RemoveDomainsWithListID(ListID string) error
• func RemoveIgnored(scan hooks.ScanRow) error
• func ResetDomains() error
• func SaveCertificateChains(rows []*hooks.CertificateChainRow, table string) error
• func SaveCertificates(rows []*hooks.CertificateRow, table string) error
• func SaveResults(table string, whereCond *structs.Struct, results *structs.Struct) error
• func SaveUnreachable(scanID int, DomainID int, DNSError bool) error
• func ScanDomainsWithDomain(domain string) error
• func ScanDomainsWithListID(list string) error
• func ScanDomainsWithProjectID(projectId string) error
• func UpdateDomainsFull(rows []hooks.DomainsRowMetaInfo) error
• func UpdateScan(scan hooks.ScanRow) error
• type SQLConfiguration
  • func ReadSQLConfig(file string) (SQLConfiguration, error)

Variables

var Logger *logrus.Entry

Functions

func ActiveDomainsWithDomain

func ActiveDomainsWithDomain(active bool, domain string) error

ActiveDomainsWithDomain changes the isActive field to the specified value for the given domain

func ActiveDomainsWithListID

func ActiveDomainsWithListID(active bool, ListID string) error

ActiveDomainsWithListID sets the active field of all domains in that list to the specified value

func GetConflictingDomains

func GetConflictingDomains(domains []string, listID string) map[string]hooks.DomainsRowMetaInfo

GetConflictingDomains returns the domains with conflicting listID in form of a map

func GetDomains

func GetDomains() ([]hooks.DomainsRow, error)

GetDomains returns the active domains, that are in the nextScan

func GetLastScan

func GetLastScan(id int) (hooks.ScanRow, error)

GetLastScan returns the specified by Id. If id==0, then it returns the last unfinished scan

func GetScans

func GetScans(table string, scanID int, scanStatus uint8) ([]hooks.DomainsReachable, error)

GetScans returns all Domains and their Reachability and how they should be tested. They are selected by the specified "scanID" and the "ScanStatus"

func InsertNewScan

func InsertNewScan(scan hooks.ScanRow) (hooks.ScanRow, error)

InsertNewScan creates a new Entry in the ScanTable and returns the ScanID

func InsertScanData

func InsertScanData(tables []string, scanData []hooks.ScanData) error

InsertScanData adds scanData to the scan-Databases

func OpenDatabase

func OpenDatabase(conf SQLConfiguration) error

OpenDatabase opens the database used for accessing domains and saving results

func PrepareScanData

func PrepareScanData(table string, scanID int, scanType int) error

PrepareScanData modifies the pending scans according to the "scanType". Entries are duplicated with diffrent "TestWithSSL"-Values, if both Protocols are supported and wanted. If the wanted Protocol is not supported, they are marked with SCAN_STATUS IGNORED.

func RemoveDomainsWithDomain

func RemoveDomainsWithDomain(domain string) error

RemoveDomainsWithDomain removes all specified domains from a list

func RemoveDomainsWithDomainAndList

func RemoveDomainsWithDomainAndList(domain string, list string) error

RemoveDomainsWithDomainAndList deletes the specified domains from the specified list

func RemoveDomainsWithListID

func RemoveDomainsWithListID(ListID string) error

RemoveDomainsWithListID deletes the specified list

func RemoveIgnored

func RemoveIgnored(scan hooks.ScanRow) error

Remove results with ScanStgatus ignored (2)

func ResetDomains

func ResetDomains() error

ResetDomains removes all domains from the next Scan

func SaveCertificateChains

func SaveCertificateChains(rows []*hooks.CertificateChainRow, table string) error

SaveCertificateChains stores all given certificate chains in the Table, while avoiding duplicate entries

func SaveCertificates

func SaveCertificates(rows []*hooks.CertificateRow, table string) error

SaveCertificates stores all given Certificates in the Table, while avoiding duplicate entries

func SaveResults

func SaveResults(table string, whereCond *structs.Struct, results *structs.Struct) error

SaveResults updates table columns defined by "results" for the row defined by "whereCond". The "whereCond"-Parameters are concatenated by ANDs

func SaveUnreachable

func SaveUnreachable(scanID int, DomainID int, DNSError bool) error

SaveUnreachable stores unreachable Domains in the corresponding table

func ScanDomainsWithDomain

func ScanDomainsWithDomain(domain string) error

ScanDomainsWithDomain adds the specified domain to the next Scan

func ScanDomainsWithListID

func ScanDomainsWithListID(list string) error

ScanDomainsWithListID adds all domains with the specified ListID to the nextScan

func ScanDomainsWithProjectID

func ScanDomainsWithProjectID(projectId string) error

ScanDomainsWithProjectID adds all domains with the specified ProjectId to the nextScan

func UpdateDomainsFull

func UpdateDomainsFull(rows []hooks.DomainsRowMetaInfo) error

UpdateDomainsFull updates the Domains-Table with the given domains.

func UpdateScan

func UpdateScan(scan hooks.ScanRow) error

UpdateScan updates the specified Scan

Types

type SQLConfiguration

type SQLConfiguration struct {
	SQLServer     string
	SQLUserID     string
	SQLPassword   string
	SQLDatabase   string
	SQLEncryption string
}

SQLConfiguration collects the Data needed for Connecting to an SQL-Database

func ReadSQLConfig

func ReadSQLConfig(file string) (SQLConfiguration, error)

ReadSQLConfig extracts the information out of the "sql_config.json" file