storage

Documentation

Index

• func ContainsCookie(cookies []*http.Cookie, name string) bool
• func StringifyCookies(cookies []*http.Cookie) string
• func UnstringifyCookies(s string) []*http.Cookie
• type InMemoryStorage
  • func (s *InMemoryStorage) ClearVisited(siteID int) error
  • func (s *InMemoryStorage) Close() error
  • func (s *InMemoryStorage) Cookies(u *url.URL) string
  • func (s *InMemoryStorage) Init() error
  • func (s *InMemoryStorage) IsVisited(requestID string) (bool, error)
  • func (s *InMemoryStorage) SetCookies(u *url.URL, cookies string)
  • func (s *InMemoryStorage) Visited(requestID string) error
• type Storage

Functions

func ContainsCookie

func ContainsCookie(cookies []*http.Cookie, name string) bool

ContainsCookie checks if a cookie name is represented in cookies

func StringifyCookies

func StringifyCookies(cookies []*http.Cookie) string

StringifyCookies serializes list of http.Cookies to string

func UnstringifyCookies

func UnstringifyCookies(s string) []*http.Cookie

UnstringifyCookies deserializes a cookie string to http.Cookies

Types

type InMemoryStorage

type InMemoryStorage struct {
	// contains filtered or unexported fields
}

InMemoryStorage is the default storage backend of colly. InMemoryStorage keeps cookies and visited urls in memory without persisting data on the disk.

func (*InMemoryStorage) ClearVisited

func (s *InMemoryStorage) ClearVisited(siteID int) error

Clear visited for site id

func (*InMemoryStorage) Close

func (s *InMemoryStorage) Close() error

Close implements Storage.Close()

func (*InMemoryStorage) Cookies

func (s *InMemoryStorage) Cookies(u *url.URL) string

Cookies implements Storage.Cookies()

func (*InMemoryStorage) Init

func (s *InMemoryStorage) Init() error

Init initializes InMemoryStorage

func (*InMemoryStorage) IsVisited

func (s *InMemoryStorage) IsVisited(requestID string) (bool, error)

IsVisited implements Storage.IsVisited()

func (*InMemoryStorage) SetCookies

func (s *InMemoryStorage) SetCookies(u *url.URL, cookies string)

SetCookies implements Storage.SetCookies()

func (*InMemoryStorage) Visited

func (s *InMemoryStorage) Visited(requestID string) error

Visited implements Storage.Visited()

type Storage

type Storage interface {
	// Init initializes the storage
	Init() error
	// Visited receives and stores a request ID that is visited by the Collector
	Visited(requestID string) error
	// IsVisited returns true if the request was visited before IsVisited
	// is called
	IsVisited(requestID string) (bool, error)
	// Clear visited for site id
	ClearVisited(siteID int) error
	// Cookies retrieves stored cookies for a given host
	Cookies(u *url.URL) string
	// SetCookies stores cookies for a given host
	SetCookies(u *url.URL, cookies string)
}

Storage is an interface which handles Collector's internal data, like visited urls and cookies. The default Storage of the Collector is the InMemoryStorage. Collector's storage can be changed by calling Collector.SetStorage() function.