Documentation ¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Generate ¶
Generates a report in JSON format from the crawler client and config. The report contains the initial crawler info, the network info for each host visited, and the country representation table for each TI page visited.
func TILinkExtractor ¶
Returns a list of all outgoing links with the same host as the current link and the path contains "dota2/The_International/". It also checks agains the VisitedNetInfo map to ensure that the link has not been visited before as we are not collecting the outgoing links of all pages.
Types ¶
type CountryTableRow ¶
type ReportFormat ¶
type ReportFormat struct { Seed string `json:"seed"` MaxRPS float64 `json:"max_rps"` CrawlTime string `json:"crawl_time"` NetInfo map[string][]gocrawler.NetworkInfo `json:"network_info"` TIStats map[string][]CountryTableRow `json:"ti_stats"` }
Click to show internal directories.
Click to hide internal directories.