fetcher

README

Fetcher

Fetch news from a entrance url, filter update time in the range, and storage them into a json. So, Microservice can read this fetched data directly.

Skeleton

• fetcher.go: Fetch action implemented here
• links.go: Fetch links from an entrance url.
• article.go: Domain Object here to implement functions that microservice needed and invoked, such as Get, List and Search.
• db.go: This is for fetched data persistence, load and save data directly.

Fetch kernal methods

There are kernal methods used to modified to match and fetch better, you can clone and rewrite the parts to implement your microservice.

1. func (a *Article) fetchTitle() (string, error) {}
2. func (a *Article) fetchUpdateTime() (*timestamppb.Timestamp, error) {}
3. func (a *Article) fetchContent() (string, error) {}

Documentation

Index

• Variables
• func Fetch() error
• type Article
  • func NewArticle() *Article
  • func (a *Article) Get(id string) (*Article, error)
  • func (a *Article) List() ([]*Article, error)
  • func (a *Article) Search(keyword ...string) ([]*Article, error)
• type ByUpdateTime
  • func (u ByUpdateTime) Len() int
  • func (u ByUpdateTime) Less(i, j int) bool
  • func (u ByUpdateTime) Swap(i, j int)

Variables

var ErrIgnoreCate error = errors.New("article is ignored by category")

var ErrIgnoreVIP error = errors.New("article is ignored by vip required")

var ErrTimeOverDays error = errors.New("article update time out of range")

Functions

func Fetch

func Fetch() error

Fetch fetch and storage all stuffs to `db/articles.json`

Types

type Article

type Article struct {
	Id            string
	Title         string
	Content       string
	WebsiteId     string
	WebsiteDomain string
	WebsiteTitle  string
	UpdateTime    *timestamppb.Timestamp
	U             *url.URL
	// contains filtered or unexported fields
}

func NewArticle

func NewArticle() *Article

func (*Article) Get

func (a *Article) Get(id string) (*Article, error)

Get read database and return the data by rawurl.

func (*Article) List

func (a *Article) List() ([]*Article, error)

List get all articles from database

func (*Article) Search

func (a *Article) Search(keyword ...string) ([]*Article, error)

type ByUpdateTime

type ByUpdateTime []*Article

func (ByUpdateTime) Len

func (u ByUpdateTime) Len() int

func (ByUpdateTime) Less

func (u ByUpdateTime) Less(i, j int) bool

func (ByUpdateTime) Swap

func (u ByUpdateTime) Swap(i, j int)