Documentation ¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Crawler ¶
type Crawler struct {
// contains filtered or unexported fields
}
Crawler represents a crawler instance that contains global configuration information.
type Flags ¶
type Flags struct { config.FileFlags Outlinks bool `subcmd:"outlinks,false,display extracted outlinks"` Progress bool `subcmd:"progress,true,'display progress of downloads'"` }
Flags represents the flags that are used to control the crawl.
type Resources ¶
type Resources struct { Extractors map[content.Type]outlinks.Extractor PopulateCrawlFS func(ctx context.Context, cfg config.CrawlService, factories map[string]crawlcmd.FSFactory) error NewContentFS func(ctx context.Context, cfg crawlcmd.CrawlCacheConfig) (content.FS, error) }
Resources represents the resources that are used by the crawler.
Click to show internal directories.
Click to hide internal directories.