db

Documentation

Index

• Constants
• func DeleteFile(fs afero.Fs)
• type DB
  • func Open() *DB
  • func OpenDB(dir string, fs afero.Fs) *DB
  • func (store *DB) Close() error
  • func (store *DB) Lookup(u *urlpkg.URL) Item
  • func (store *DB) Store(url *urlpkg.URL, item Item)
• type Item
  • func (i Item) Empty() bool
  • func (i Item) EmptyContentType() bool
  • func (i Item) String() string
  • func (i Item) Strings() []string

Constants

const FileName = "goscrape-cache.txt"

Functions

func DeleteFile

func DeleteFile(fs afero.Fs)

Types

type DB

type DB struct {
	// contains filtered or unexported fields
}

DB provides a persistent store for HTTP ETags and other metadata to reduce network traffic when repeating a download session. If the store is unavailable for some reason, its methods are no-ops.

func Open

func Open() *DB

func OpenDB

func OpenDB(dir string, fs afero.Fs) *DB

func (*DB) Close

func (store *DB) Close() error

func (*DB) Lookup

func (store *DB) Lookup(u *urlpkg.URL) Item

Lookup finds the metadata for a given URL.

func (*DB) Store

func (store *DB) Store(url *urlpkg.URL, item Item)

Store stores the metadata for a given URL.

type Item

type Item struct {
	Code     int
	Location string // redirection
	Content  header.ContentType
	ETags    string
	Expires  time.Time
}

func (Item) Empty

func (i Item) Empty() bool

func (Item) EmptyContentType

func (i Item) EmptyContentType() bool

func (Item) String

func (i Item) String() string

func (Item) Strings

func (i Item) Strings() []string