pool

README

This is a generic worker pool for the Go language. It's useful when you want to limit the number of goroutines running in parallel.

Installation:

go install github.com/stefantalpalaru/pool

Compile and run the examples:

go run examples/pool_example.go
go run examples/web_crawler.go

The last example is actually an exercise from the Go tour modified to use a worker pool for fetching and processing the URLs. The need to limit the number of concurrent requests in real web scraping scenarios was what prompted the creation of this package.

You can see the godoc generated documentation online at godoc.org/github.com/stefantalpalaru/pool.

Documentation

Overview

Package pool provides a worker pool.

Index

• type Job
• type Pool
  • func New(workers int) (pool *Pool)
  • func (pool *Pool) Add(f func(...interface{}) interface{}, args ...interface{})
  • func (pool *Pool) Results() (res []*Job)
  • func (pool *Pool) Run()
  • func (pool *Pool) Status() stats
  • func (pool *Pool) Stop()
  • func (pool *Pool) Wait()
  • func (pool *Pool) WaitForJob() *Job

Types

type Job

type Job struct {
	F      func(...interface{}) interface{}
	Args   []interface{}
	Result interface{}
	Err    error
	// contains filtered or unexported fields
}

Job holds all the data related to a worker's instance.

type Pool

type Pool struct {
	// contains filtered or unexported fields
}

Pool is the main data structure.

func New

func New(workers int) (pool *Pool)

New creates a new Pool.

func (*Pool) Add

func (pool *Pool) Add(f func(...interface{}) interface{}, args ...interface{})

Add creates a Job from the given function and args and adds it to the Pool.

func (*Pool) Results

func (pool *Pool) Results() (res []*Job)

Results retrieves the completed jobs.

func (*Pool) Run

func (pool *Pool) Run()

Run starts the Pool by launching the workers. It's OK to start an empty Pool. The jobs will be fed to the workers as soon as they become available.

func (*Pool) Status

func (pool *Pool) Status() stats

Status returns a "stats" instance.

func (*Pool) Stop

func (pool *Pool) Stop()

Stop will signal the workers to exit and wait for them to actually do that. It also releases any other resources (e.g.: it stops the supervisor goroutine) so call this method when you're done with the Pool instance to allow the GC to do its job.

func (*Pool) Wait

func (pool *Pool) Wait()

Wait blocks until all the jobs in the Pool are done.

func (*Pool) WaitForJob

func (pool *Pool) WaitForJob() *Job

WaitForJob blocks until a completed job is available and returns it. If there are no jobs running, it returns nil.