storage

Documentation

Index

• func ContainsCookie(cookies []*http.Cookie, name string) bool
• func StringifyCookies(cookies []*http.Cookie) string
• func UnstringifyCookies(s string) []*http.Cookie
• type InMemoryStorage
  • func (s *InMemoryStorage) Close() error
  • func (s *InMemoryStorage) Cookies(u *url.URL) string
  • func (s *InMemoryStorage) Init() error
  • func (s *InMemoryStorage) IsVisited(requestID uint64) (bool, error)
  • func (s *InMemoryStorage) SetCookies(u *url.URL, cookies string)
  • func (s *InMemoryStorage) Visited(requestID uint64) error
• type Storage

Functions

func ContainsCookie

func ContainsCookie(cookies []*http.Cookie, name string) bool

ContainsCookie checks if a cookie name is represented in cookies

func StringifyCookies

func StringifyCookies(cookies []*http.Cookie) string

StringifyCookies serializes list of http.Cookies to string

func UnstringifyCookies

func UnstringifyCookies(s string) []*http.Cookie

UnstringifyCookies deserializes a cookie string to http.Cookies

Types

type InMemoryStorage

type InMemoryStorage struct {
	// contains filtered or unexported fields
}

InMemoryStorage is the default storage backend of colly. InMemoryStorage keeps cookies and visited urls in memory without persisting data on the disk.

func (*InMemoryStorage) Close

func (s *InMemoryStorage) Close() error

Close implements Storage.Close()

func (*InMemoryStorage) Cookies

func (s *InMemoryStorage) Cookies(u *url.URL) string

Cookies implements Storage.Cookies()

func (*InMemoryStorage) Init

func (s *InMemoryStorage) Init() error

Init initializes InMemoryStorage

func (*InMemoryStorage) IsVisited

func (s *InMemoryStorage) IsVisited(requestID uint64) (bool, error)

IsVisited implements Storage.IsVisited()

func (*InMemoryStorage) SetCookies

func (s *InMemoryStorage) SetCookies(u *url.URL, cookies string)

SetCookies implements Storage.SetCookies()

func (*InMemoryStorage) Visited

func (s *InMemoryStorage) Visited(requestID uint64) error

Visited implements Storage.Visited()

type Storage

type Storage interface {
	// Init initializes the storage
	Init() error
	// Visited receives and stores a request ID that is visited by the Collector
	Visited(requestID uint64) error
	// IsVisited returns true if the request was visited before IsVisited
	// is called
	IsVisited(requestID uint64) (bool, error)
	// Cookies retrieves stored cookies for a given host
	Cookies(u *url.URL) string
	// SetCookies stores cookies for a given host
	SetCookies(u *url.URL, cookies string)
}

Storage is an interface which handles Collector's internal data, like visited urls and cookies. The default Storage of the Collector is the InMemoryStorage. Collector's storage can be changed by calling Collector.SetStorage() function.