crawler

Documentation

Index

• Variables
• func Init() (err error)
• func SpawnWorkers(ctx context.Context, wg *sync.WaitGroup, q queue.Queue, ...) error
• type Worker

Variables

var (
	ErrNoWorkToDo = errors.New("no work to do")
)

Functions

func Init

func Init() (err error)

func SpawnWorkers

func SpawnWorkers(ctx context.Context, wg *sync.WaitGroup, q queue.Queue, runtimeSettings settings.Settings) error

SpawnWorkers spawns n workers and returns an error if any ctx is used to stop workers q is a queue to get urls from outputDir is a directory to save results n is a number of workers to spawn

Types

type Worker

type Worker interface {
	Run(ctx context.Context, wg *sync.WaitGroup)
}