scrape

Documentation

Index

• Constants
• func OnHTMLDaumHeadlineNews(dc chan<- gachifinder.GachiData, s *Scrape)
• func OnHTMLNaverHeadlineNews(dc chan<- gachifinder.GachiData, s *Scrape)
• type ParsingHandler
• type Scrape
  • func (s *Scrape) Do(fs []ParsingHandler) <-chan gachifinder.GachiData
• type Scraper

Constants

const DaumNews = "news.daum.net"

DaumNews is the root domain for visiting.

const NaverNews = "news.naver.com"

NaverNews is the root domain for visiting.

Functions

func OnHTMLDaumHeadlineNews

func OnHTMLDaumHeadlineNews(dc chan<- gachifinder.GachiData, s *Scrape)

OnHTMLDaumHeadlineNews registers to subvisit and parse the scraped "new.daum.com" HTML.

func OnHTMLNaverHeadlineNews

func OnHTMLNaverHeadlineNews(dc chan<- gachifinder.GachiData, s *Scrape)

OnHTMLNaverHeadlineNews registers to subvisit and parse the scraped "new.naver.com" HTML.

Types

type ParsingHandler

type ParsingHandler func(chan<- gachifinder.GachiData, *Scrape)

ParsingHandler ...

type Scrape

type Scrape struct {
	Config *gachifinder.Config
	// contains filtered or unexported fields
}

Scrape struct.

func (*Scrape) Do

func (s *Scrape) Do(fs []ParsingHandler) <-chan gachifinder.GachiData

Do creates colly.collector and queue, and then do and wait till done

type Scraper

type Scraper interface {
	// Do is a producer in a part of a pipeline
	Do([]ParsingHandler) <-chan gachifinder.GachiData
}

Scraper interface is a crawling actor.