README
¶
Throttler - intelligent WaitGroups
Throttler fills the gap between sync.WaitGroup and manually monitoring your goroutines with channels. The API is almost identical to Wait Groups, but it allows you to set a max number of workers that can be running simultaneously. It uses channels internally to block until a job completes by calling Done() or until all jobs have been completed. It also provides a built in error channel that captures your goroutine errors and provides access to them as []error
after you exit the loop.
See a fully functional example on the playground at http://bit.ly/throttler-v3
Compare the Throttler example to the sync.WaitGroup example from http://golang.org/pkg/sync/#example_WaitGroup
How to use Throttler
// This example fetches several URLs concurrently,
// using a Throttler to block until all the fetches are complete.
// Compare to http://golang.org/pkg/sync/#example_WaitGroup
func ExampleThrottler() {
var urls = []string{
"http://www.golang.org/",
"http://www.google.com/",
"http://www.somestupidname.com/",
}
// Create a new Throttler that will get 2 urls at a time
t := throttler.New(2, len(urls))
for _, url := range urls {
// Launch a goroutine to fetch the URL.
go func(url string) {
// Fetch the URL.
err := http.Get(url)
// Let Throttler know when the goroutine completes
// so it can dispatch another worker
t.Done(err)
}(url)
// Pauses until a worker is available or all jobs have been completed
// Returning the total number of goroutines that have errored
// lets you choose to break out of the loop without starting any more
errorCount := t.Throttle()
}
}
vs How to use a sync.WaitGroup
// This example fetches several URLs concurrently,
// using a WaitGroup to block until all the fetches are complete.
func ExampleWaitGroup() {
var wg sync.WaitGroup
var urls = []string{
"http://www.golang.org/",
"http://www.google.com/",
"http://www.somestupidname.com/",
}
for _, url := range urls {
// Increment the WaitGroup counter.
wg.Add(1)
// Launch a goroutine to fetch the URL.
go func(url string) {
// Decrement the counter when the goroutine completes.
defer wg.Done()
// Fetch the URL.
http.Get(url)
}(url)
}
// Wait for all HTTP fetches to complete.
wg.Wait()
}
Documentation
¶
Overview ¶
Package throttler fills the gap between sync.WaitGroup and manually monitoring your goroutines with channels. The API is almost identical to Wait Groups, but it allows you to set a max number of workers that can be running simultaneously. It uses channels internally to block until a job completes by calling Done(err) or until all jobs have been completed.
After exiting the loop where you are using Throttler, you can call the `Err` or `Errs` method to check for errors. `Err` will return a single error representative of all the errors Throttler caught. The `Errs` method will return all the errors as a slice of errors (`[]error`).
Compare the Throttler example to the sync.WaitGroup example http://golang.org/pkg/sync/#example_WaitGroup
See a fully functional example on the playground at http://bit.ly/throttler-v3
Index ¶
- type Throttler
- func (t *Throttler) BatchEndIndex() int
- func (t *Throttler) BatchStartIndex() int
- func (t *Throttler) Done(err error)
- func (t *Throttler) Err() error
- func (t *Throttler) Errs() []error
- func (t *Throttler) SetMaxWorkers(maxWorkers int)
- func (t *Throttler) Throttle() int
- func (t *Throttler) TotalJobs() int
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Throttler ¶
type Throttler struct {
// contains filtered or unexported fields
}
Throttler stores all the information about the number of workers, the active workers and error information
Example ¶
This example fetches several URLs concurrently, using a Throttler to block until all the fetches are complete. Compare to http://golang.org/pkg/sync/#example_WaitGroup
Output:
Example (Errors) ¶
This example fetches several URLs concurrently, using a Throttler to block until all the fetches are complete and checks the errors returned. Compare to http://golang.org/pkg/sync/#example_WaitGroup
Output:
func New ¶
New returns a Throttler that will govern the max number of workers and will work with the total number of jobs. It panics if maxWorkers < 1.
func NewBatchedThrottler ¶
NewBatchedThrottler returns a Throttler (just like New), but also enables batching.
func (*Throttler) BatchEndIndex ¶
BatchEndIndex returns the ending index for the next batch. It either returns the full batch size or the remaining amount of jobs. The job count isn't modified until th.Throttle() is called, so if you don't call Throttle before executing this again, it will return the same index as before.
func (*Throttler) BatchStartIndex ¶
BatchStartIndex returns the starting index for the next batch. The job count isn't modified until th.Throttle() is called, so if you don't call Throttle before executing this again, it will return the same index as before
func (*Throttler) Done ¶
Done lets Throttler know that a job has been completed so that another worker can be activated. If Done is called less times than totalJobs, Throttle will block forever
func (*Throttler) SetMaxWorkers ¶
SetMaxWorkers lets you change the total number of workers that can run concurrently. NOTE: If all workers are currently running, this setting is not guaranteed to take effect until one of them completes and Throttle() is called again
func (*Throttler) Throttle ¶
Throttle works similarly to sync.WaitGroup, except inside your goroutine dispatch loop rather than after. It will not block until the number of active workers matches the max number of workers designated in the call to NewThrottler or all of the jobs have been dispatched. It stops blocking when Done has been called as many times as totalJobs.