scrape

Documentation

Index

• func Run(outwr io.Writer, config *userconfig.Meta) error
• func StartLoop(s *Config, c *userconfig.Meta) error
• type Config

Functions

func Run

func Run(outwr io.Writer, config *userconfig.Meta) error

Run conducts a single scrape and email cycle and returns the first error encountered. It reads the user config anew at the beginning of each cycle. At the end of a scrape cycle, it sends an email or, depending on the config, writes a plaintext version of the email message to outwr.

func StartLoop

func StartLoop(s *Config, c *userconfig.Meta) error

StartLoop begins the main sequence of scraping websites for links every interval (defined by tc) with the provided config. If an s.ErrCh is provided, sends any errors to it. Send a struct{} to sc to stop the scraper.

Types

type Config

type Config struct {
	// For time.Ticker ticks
	TickCh <-chan time.Time
	// Writer for a message to display when a scrape has finished.The means
	// of display is controlled by the caller. Intended for email text shown
	// when the --noemail flag is used.
	OutputWr io.Writer
	// Number of rounds of scraping and emailing to perform before stopping
	// the scraper. Used for testing.
	IterationLimit uint
}