scrape

Documentation

Index

• func CrawlFiles(seedURLs []*url.URL, exactFileRegex string, debug bool) ([]string, error)
• func CrawlFolders(seedURLs []*url.URL, exactFolderRegex string, recursive bool, debug bool) ([]string, error)
• func CrawlFoldersPath(seedURLs []*url.URL, exactFolderRegex string, recursive bool, debug bool) ([]string, error)

Functions

func CrawlFiles

func CrawlFiles(seedURLs []*url.URL, exactFileRegex string, debug bool) ([]string, error)

CrawlFiles returns a list of file names found from the seed URL, filtered by file name regex.

func CrawlFolders

func CrawlFolders(seedURLs []*url.URL, exactFolderRegex string, recursive bool, debug bool) ([]string, error)

CrawlFolders returns a list of folder names found from each seed URL, filtered by folder name regex.

func CrawlFoldersPath

func CrawlFoldersPath(seedURLs []*url.URL, exactFolderRegex string, recursive bool, debug bool) ([]string, error)

CrawlFoldersPath returns a list of folder names found from each seed URL, filtered by folder name regex.