queue

Documentation

Overview

Package queue manages the queue for a distributed crawler

Index

• Variables
• type Queuer
• type Redis
  • func (r *Redis) AddLink(lnk string) error
  • func (r *Redis) CountLinks() (int64, error)
  • func (r *Redis) DelayHost(host string, ttl time.Duration) error
  • func (r *Redis) QueueLink(ttl time.Duration) (string, error)
  • func (r *Redis) ReserveHost(host string, ttl time.Duration) error

Variables

var ErrAlreadyReserved = errors.New("host already reserved")

ErrAlreadyReserved indicates another worker reserved the host

var ErrNotQueued = errors.New("link already queued")

ErrNotQueued indicates a link was not queued

Types

type Queuer

type Queuer interface {
	CountLinks() (int64, error)
	AddLink(lnk string) error
	QueueLink(ttl time.Duration) (string, error)
	ReserveHost(host string, ttl time.Duration) error
	DelayHost(host string, ttl time.Duration) error
}

Queuer is handles links and our crawling queue

type Redis

type Redis struct {
	RedisPool *redis.Pool
}

Redis implements the Queuer interface

func (*Redis) AddLink

func (r *Redis) AddLink(lnk string) error

AddLink adds a link to our redis set

func (*Redis) CountLinks

func (r *Redis) CountLinks() (int64, error)

CountLinks counts the number of links in our queue

func (*Redis) DelayHost

func (r *Redis) DelayHost(host string, ttl time.Duration) error

DelayHost is like ReserveHost but makes sure the key is already set. If key doesn't exist then something is wrong (eg reservation < crawler's timeout)

func (*Redis) QueueLink

func (r *Redis) QueueLink(ttl time.Duration) (string, error)

QueueLink pops a link from our set

func (*Redis) ReserveHost

func (r *Redis) ReserveHost(host string, ttl time.Duration) error

ReserveHost reserves a host for crawling