scraper

Documentation

Index

• Constants
• type Message
  • func (m *Message) HTML() interface{}
  • func (m *Message) String() string
• type Scraper
  • func New(c *config.Config) (*Scraper, error)
  • func (s *Scraper) Close()
  • func (s *Scraper) Get(id int, remove bool) (*Message, error)
  • func (s *Scraper) LastScrape() time.Time
  • func (s *Scraper) Messages() []*Message
  • func (s *Scraper) Scrape()
  • func (s *Scraper) SetSettings(settings Settings) error
  • func (s *Scraper) Settings() Settings
• type Settings

Constants

const (
	OneboxImage = "image"
)

Types

type Message

type Message struct {
	ID          int       `json:"id"`
	URL         string    `json:"url"`
	Body        string    `json:"body"`
	Onebox      string    `json:"onebox"`
	Stars       int       `json:"stars"`
	Author      string    `json:"author"`
	AuthorImage string    `json:"author_image"`
	Created     time.Time `json:"created"`
}

Message is an individual message found by the scraper.

func (*Message) HTML

func (m *Message) HTML() interface{}

HTML returns an object that can be rendered in a template, either as a single string or as pre-escaped HTML.

func (*Message) String

func (m *Message) String() string

String returns the value that should be passed to Twitter to update the status.

type Scraper

type Scraper struct {
	// contains filtered or unexported fields
}

Scraper regularly scrapes the chat transcript for the current and previous day for messages that match the criteria in use. Once a message matches, it is added to the list of candidates for tweeting. The IDs of messages that are used is kept to prevent duplicates.

func New

func New(c *config.Config) (*Scraper, error)

New creates a new scraper.

func (*Scraper) Close

func (s *Scraper) Close()

Close shuts down the scraper and waits for it to exit.

func (*Scraper) Get

func (s *Scraper) Get(id int, remove bool) (*Message, error)

Get retrieves a message by its ID. It can also optionally remove the message from the scraper and prevent it from showing up in future scrapes.

func (*Scraper) LastScrape

func (s *Scraper) LastScrape() time.Time

LastScrape returns the time of last scrape.

func (*Scraper) Messages

func (s *Scraper) Messages() []*Message

Messages retrieves the current list of matching messages.

func (*Scraper) Scrape

func (s *Scraper) Scrape()

Scrape manually initiates a scrape of the transcript.

func (*Scraper) SetSettings

func (s *Scraper) SetSettings(settings Settings) error

SetSettings stores the current settings for the scraper.

func (*Scraper) Settings

func (s *Scraper) Settings() Settings

Settings retrieves the current settings for the scraper.

type Settings

type Settings struct {
	PollURL       string   `json:"poll_url"`       // Base URL for chat
	PollRoomID    int      `json:"poll_room_id"`   // Room to scrape
	PollHistory   int      `json:"poll_history"`   // How many days prior to begin scraping
	PollFrequency int      `json:"poll_frequency"` // Time in minutes between polling
	MinStars      int      `json:"min_stars"`      // Select messages with these many stars
	MatchingWords []string `json:"matching_words"` // Select messages with these words
}

Settings controls the behavior of the scraper. This includes such things as the room, polling rate, and matching criteria.