scrape

README

obsidian

Obsidian tools

Documentation

Overview

Package scrape helps write code to scrape a page.

Index

• Constants
• func ParseDate(header string) (time.Time, error)
• type Conn
  • func NewConn() *Conn
  • func (c *Conn) ConfigFromNetRc(machine string) error
  • func (c *Conn) FetchAndCache(uri string, expireDuration time.Duration) (header, contents string, fromCache bool, err error)
  • func (c *Conn) Login() error
  • func (c *Conn) PostURL(uri string, fields map[string]string) (string, error)

Constants

const (
	// NormalTimeout is how long we should wait before refetching from site
	NormalTimeout = time.Duration(time.Minute * 2)
	// NoCache will not use the cases (timeout is zero)
	NoCache = time.Duration(time.Minute * 0)

	// UA is the user Agent we will be using
	UA = `Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36`
)

Functions

func ParseDate

func ParseDate(header string) (time.Time, error)

ParseDate helps parse a date from HTTP server

Types

type Conn

type Conn struct {
	// LoginURL should be set before calling Login.
	LoginURL       string
	CookieJarFname string
	Verbose        int

	// FailedLogin should be set for a function that returns true if login may have timed out or failed.
	FailedLogin func(content string) bool

	// DontCache should be used for page results that shouldn't be cached
	DontCache func(content string) bool

	CacheNameFmt string
	UserName     string
	Password     string
	// contains filtered or unexported fields
}

Conn is the basic connection object.

func NewConn

func NewConn() *Conn

NewConn creates a new connection

func (*Conn) ConfigFromNetRc

func (c *Conn) ConfigFromNetRc(machine string) error

ConfigFromNetRc gets some configuration information from the ~/.netrc file.

func (*Conn) FetchAndCache

func (c *Conn) FetchAndCache(uri string, expireDuration time.Duration) (header, contents string, fromCache bool, err error)

FetchAndCache fetches and url and caches it for later.

func (*Conn) Login

func (c *Conn) Login() error

Login goes to the login url with a username and password.

func (*Conn) PostURL

func (c *Conn) PostURL(uri string, fields map[string]string) (string, error)

PostURL posts and url with the given map.