README
¶
Concurrency
Packages for handling Go concurrency.
Introduction
The packages contained within this repository provide safer primitives for doing concurrent and parallel operations.
In addition, these packages integrate with Open Telemetry(OTEL) and provide execution traces to give insight into how your software is operating. Along with exported metrics within these packages the insights can then be used to provide deeper profiling of your software.
If used with gostdlib/foundation/telemetry/slog, OTEL traces will also contain your logging messages when using the slog or log packages.
A quick look
gouroutines/: A set of packages for safer goroutine spawning and goroutine reuse- Use
goroutines/pooledif you want:- Reuse of goroutines instead of spawning new ones
- The ability to limit the number of goroutines
- The ability to bypass the goroutine limit for some tasks
- Use
goroutines/limitedif you want:- A safer way to spawn goroutines with limits
- The ability to limit the number of goroutines
- The ability to bypass the goroutine limit for some tasks
- Use
pipelines/: A set of packages for creating streaming pipelines- Use
pipelines/stagedpipeif you want:- A safer way to build streaming pipelines
- Multiple independent input streams into the same pipeline for processing
- Want concurrency and parallel pipelines
- Use of stack for stream data or allocating on the heap
- Routing to different processing based on data
- Use
prim/: A set of packages for safer concurrency primatives built on goroutine pooling- Use
prim/waitif you want:- A safer version of
sync.WaitGroupfor parallel jobs - A parallel job runner that collects errors after all jobs complete
- A parallel job runner that CAN stops processing on the first error
- A parallel job runner that CAN be
Contextcancelled - Reuse and limiting of goroutines by supplying a
goroutines.Pool - Support for OpenTelemetry spans
- A safer version of
- Use
prim/slicesif you want:- To access elements in a slice in parallel to perform some operation
- Support for processing errors
- Support for OpenTelemetry spans
- Use
prim/chansif you want:- To parallel process a chan of values
- Support for processing errors
- Support for OpenTelemetry spans
- Use
Directories
¶
| Path | Synopsis |
|---|---|
|
Package goroutines provides the interfaces and defintions that goroutine pools must implement/use.
|
Package goroutines provides the interfaces and defintions that goroutine pools must implement/use. |
|
internal/register
Package register has a registration method that a goroutine.Pool implementation can use to register the pool.
|
Package register has a registration method that a goroutine.Pool implementation can use to register the pool. |
|
limited
Package limited provides a groutine execution Pool that spins a goroutine per Submit() but is hard limited to the number of goroutines that can run at any time.
|
Package limited provides a groutine execution Pool that spins a goroutine per Submit() but is hard limited to the number of goroutines that can run at any time. |
|
pooled
Package pooled provides a Pool of goroutines where you can submit Jobs to be run when by an exisiting goroutine instead of spinning off a new goroutine.
|
Package pooled provides a Pool of goroutines where you can submit Jobs to be run when by an exisiting goroutine instead of spinning off a new goroutine. |
|
pipelines
|
|
|
stagedpipe
Package stagedpipe offers a generic, concurrent and parallel pipeline based on a statemachine to process work.
|
Package stagedpipe offers a generic, concurrent and parallel pipeline based on a statemachine to process work. |
|
stagedpipe/examples/etl/bostonFoodViolations/pipelined/etl
Package etl contains the ETL statemachine for translating the boston food violations file into a postgres database.
|
Package etl contains the ETL statemachine for translating the boston food violations file into a postgres database. |
|
stagedpipe/internal/queue
Package queue provides a simple queue primitive that blocks on Pop() until an entry is available and can block on Push() when a size limit is applied.
|
Package queue provides a simple queue primitive that blocks on Pop() until an entry is available and can block on Push() when a size limit is applied. |
|
stagedpipe/testing/client
Package client provides a fake client to a fictional "identity" service to use in testing.
|
Package client provides a fake client to a fictional "identity" service to use in testing. |
|
Package prim provides concurrency primatives that can be used to run concurrent operations in a safer manner than the stdlib with additional instrumentation and reuse options to prevent costly setup and teardown of goroutines (they are cheap, but they are not free).
|
Package prim provides concurrency primatives that can be used to run concurrent operations in a safer manner than the stdlib with additional instrumentation and reuse options to prevent costly setup and teardown of goroutines (they are cheap, but they are not free). |
|
chans
Package chans provides functions for operating on recv channels in parallel.
|
Package chans provides functions for operating on recv channels in parallel. |
|
slices
Package slices provides functions for operating on slices in parallel.
|
Package slices provides functions for operating on slices in parallel. |
|
wait
Package wait provides a safer alternative to sync.WaitGroup.
|
Package wait provides a safer alternative to sync.WaitGroup. |
Click to show internal directories.
Click to hide internal directories.