crawler

Documentation

Index

• func Collector(ctx context.Context, url string, projectPath string, cookieJar *cookiejar.Jar, ...) error
• func Crawl(ctx context.Context, site string, projectPath string, cookieJar *cookiejar.Jar, ...) error
• func Extractor(link string, projectPath string)
• func HTMLExtractor(link string, projectPath string)

Functions

func Collector

func Collector(ctx context.Context, url string, projectPath string, cookieJar *cookiejar.Jar, proxyString string, userAgent string) error

Collector searches for css, js, and images within a given link TODO improve for better performance

func Crawl

func Crawl(ctx context.Context, site string, projectPath string, cookieJar *cookiejar.Jar, proxyString string, userAgent string) error

Crawl asks the necessary crawlers for collecting links for building the web page

func Extractor

func Extractor(link string, projectPath string)

Extractor visits a link determines if its a page or sublink downloads the contents to a correct directory in project folder TODO add functionality for determining if page or sublink

func HTMLExtractor

func HTMLExtractor(link string, projectPath string)

HTMLExtractor ...