page

Documentation

Overview

Package page helps fetching a HTML page and its referenced CSS files.

Index

• Variables
• type File
• type LimitedReader
  • func NewLimitedReader(rc io.Reader, l int64) *LimitedReader
  • func (l *LimitedReader) Read(p []byte) (n int, err error)
• type Page
  • func New(ctx context.Context, u string) (*Page, error)
  • func (p *Page) Count() int
  • func (p *Page) NewFile(ctx context.Context, url string) (*File, error)
  • func (p *Page) Size() int64

Examples

• New

Variables

var (
	DefaultMaxPageSize  int64 = 1024 * 1024 * 10
	DefaultMaxFileCount int   = 15
	DefaultMaxFileSize  int64 = 1024 * 1024 * 5
)

Default limits for fetching a Page.

var (
	MaxPageSize  = DefaultMaxPageSize
	MaxFileCount = DefaultMaxFileCount
	MaxFileSize  = DefaultMaxFileSize
)

Limits for fetching a Page.

Types

type File

type File struct {
	Body string
	URL  *url.URL
}

File consists of the content and URL of a single file.

type LimitedReader

type LimitedReader struct {
	io.Reader
	N int64
}

LimitedReader reads at most N bytes and returns an error when more than N bytes could have been read.

func NewLimitedReader

func NewLimitedReader(rc io.Reader, l int64) *LimitedReader

NewLimitedReadCloser wraps a Reader and limits the amount of bytes it can read before returning an error.

func (*LimitedReader) Read

func (l *LimitedReader) Read(p []byte) (n int, err error)

type Page

type Page struct {
	HTML *File
	CSS  []*File
}

Page contains HTML and linked CSS files for a specific URL.

func New

func New(ctx context.Context, u string) (*Page, error)

New Page from a URL. Any linked CSS stylesheets will be downloaded.

Example

s := serve()
defer s.Close()
p, _ := New(context.Background(), s.URL+"/external.html")
fmt.Printf("%s: %s\n", p.HTML.URL.Path, p.HTML.Body)
fmt.Printf("%s: %s\n", p.CSS[0].URL.Path, p.CSS[0].Body)

Output:

/external.html: <link rel="stylesheet" href="style.css">
/style.css: body { color: #c0c0c0 }

func (*Page) Count

func (p *Page) Count() int

Count returns the amount of files.

func (*Page) NewFile

func (p *Page) NewFile(ctx context.Context, url string) (*File, error)

NewFile creates a new File by GETting it from url.

func (*Page) Size

func (p *Page) Size() int64

Size returns the length of files.