crawl

Documentation

Index

• type CollyCrawler
  • func (s *CollyCrawler) Crawl(ctx context.Context, url string) (*Result, error)
  • func (s *CollyCrawler) CrawlMulti(ctx context.Context, urls []string) (map[string]*Result, error)
• type Crawler
  • func NewCollyCrawler() Crawler
  • func NewJinaCrawler() Crawler
• type JinaCrawler
  • func (c *JinaCrawler) Crawl(ctx context.Context, link string) (*Result, error)
• type Result

Types

type CollyCrawler

type CollyCrawler struct {
}

CollyCrawler 使用jina搜索

func (*CollyCrawler) Crawl

func (s *CollyCrawler) Crawl(ctx context.Context, url string) (*Result, error)

Crawl ...

func (*CollyCrawler) CrawlMulti

func (s *CollyCrawler) CrawlMulti(ctx context.Context, urls []string) (map[string]*Result, error)

CrawlMulti 爬取网页内容

type Crawler

type Crawler interface {
	Crawl(ctx context.Context, url string) (*Result, error)
}

Crawler 爬虫接口

func NewCollyCrawler

func NewCollyCrawler() Crawler

NewCollyCrawler 创建jina爬虫

func NewJinaCrawler

func NewJinaCrawler() Crawler

NewJinaCrawler 创建jina爬虫

type JinaCrawler

type JinaCrawler struct {
}

JinaCrawler 使用jina搜索

func (*JinaCrawler) Crawl

func (c *JinaCrawler) Crawl(ctx context.Context, link string) (*Result, error)

Crawl 爬取

type Result

type Result struct {
	Link    string
	Title   string
	Content string
}

Result 爬取结果