crawler

Documentation

Index

• type Channel
• type Configs
• type Crawler
  • func NewCrawler(configs *Configs, service *youtube.Service) *Crawler
  • func (c *Crawler) GetPlaylistID(channelID string) (string, error)
  • func (c *Crawler) GetUpdatedVideos() []*Video
  • func (c *Crawler) GetUploads(playlistID string) ([]*Video, error)
  • func (c *Crawler) Run()
• type Video

Types

type Channel

type Channel struct {
	ID string `yaml:"id"`
}

Channel represents youtube channel schema.

type Configs

type Configs struct {
	Channels []Channel `yaml:"channels"`
	Teams    []string  `yaml:"teams"`
	Terms    []string  `yaml:"terms"`
	Excludes []string  `yaml:"excludes"`
	Playlist string    `yaml:"playlist"`
}

Configs is the config file for the crawler.

type Crawler

type Crawler struct {
	// contains filtered or unexported fields
}

Crawler crawls youtube channels and filters uploads that match the requirements.

func NewCrawler

func NewCrawler(configs *Configs, service *youtube.Service) *Crawler

NewCrawler Creates a new crawler object.

func (*Crawler) GetPlaylistID

func (c *Crawler) GetPlaylistID(channelID string) (string, error)

GetPlaylistID gets uploads ID.

func (*Crawler) GetUpdatedVideos

func (c *Crawler) GetUpdatedVideos() []*Video

GetUpdatedVideos returns a list of updated videos.

func (*Crawler) GetUploads

func (c *Crawler) GetUploads(playlistID string) ([]*Video, error)

GetUploads gets the uploaded videos for the given playlist.

func (*Crawler) Run

func (c *Crawler) Run()

Run runs the crawler.

type Video

type Video struct {
	ID           string `json:"id"`
	Title        string `json:"title"`
	ChannelTitle string `json:"channel_title"`
}

Video represents a simple video object.