extraction

Documentation

Index

• func CheckHourlyMax(article *m.Article, currentTime time.Time, currentVisits int)
• func ExtractArticleURLsFromSearchResults(term string, page int) ([]string, error)
• func ExtractDataFromDocument(doc *gq.Document, url string, includeTitle bool, fromJson bool) *m.ExtractedBody
• func ExtractDataFromHTMLAtURL(url string, includeTitle bool) *m.ExtractedBody
• func ExtractDataFromHTMLString(html string, url string, includeTitle bool) *m.ExtractedBody
• func IsSameDay(t1 time.Time, t2 time.Time) bool
• func ParseAuthor(author string) []string
• func ParseAuthors(authors []string) []string
• func RoundHourDown(t time.Time) time.Time

Functions

func CheckHourlyMax

func CheckHourlyMax(article *m.Article, currentTime time.Time, currentVisits int)

Given an article, a currentTime, and a currentVisits variable, check the last value in the article.Visits array (lastInterval). Does an in-memory replace of the values if necessary

If currentTime.Hour() == lastInterval.Hour()

if currentVisits > lastInterval.Max
		lastInterval.Max = currentVisits

If currentTime.Hour() > lastInterval.Hour()

append onto article.Visits array using this hour

If currentTime.Hour() < lastInterval.Hour()

ignore

func ExtractArticleURLsFromSearchResults

func ExtractArticleURLsFromSearchResults(term string, page int) ([]string, error)

func ExtractDataFromDocument

func ExtractDataFromDocument(doc *gq.Document, url string, includeTitle bool, fromJson bool) *m.ExtractedBody

func ExtractDataFromHTMLAtURL

func ExtractDataFromHTMLAtURL(url string, includeTitle bool) *m.ExtractedBody

func ExtractDataFromHTMLString

func ExtractDataFromHTMLString(html string, url string, includeTitle bool) *m.ExtractedBody

func IsSameDay

func IsSameDay(t1 time.Time, t2 time.Time) bool

Are t1 and t2 on the same day?

func ParseAuthor

func ParseAuthor(author string) []string

func ParseAuthors

func ParseAuthors(authors []string) []string

func RoundHourDown

func RoundHourDown(t time.Time) time.Time