README
¶
Benthos is a framework for creating declarative stream processors where a pipeline of one or more sources, an arbitrary series of processing stages, and one or more sinks can be configured in a single config:
input:
gcp_pubsub:
project: foo
subscription: bar
pipeline:
processors:
- mapping: |
root.message = this
root.meta.link_count = this.links.length()
root.user.age = this.user.age.number()
output:
redis_streams:
url: tcp://TODO:6379
stream: baz
max_in_flight: 20
Delivery Guarantees
Delivery guarantees can be a dodgy subject. Benthos processes and acknowledges messages using an in-process transaction model with no need for any disk persisted state, so when connecting to at-least-once sources and sinks it's able to guarantee at-least-once delivery even in the event of crashes, disk corruption, or other unexpected server faults.
This behaviour is the default and free of caveats, which also makes deploying and scaling Benthos much simpler.
Lint
Benthos uses golangci-lint for linting, which you can install with:
curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(go env GOPATH)/bin
And then run it with make lint.
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
internal
|
|
|
api
Package api implements a type used for creating the Benthos HTTP API.
|
Package api implements a type used for creating the Benthos HTTP API. |
|
batch
Package batch contains internal utilities for interacting with message batches.
|
Package batch contains internal utilities for interacting with message batches. |
|
batch/policy
Package policy provides tooling for creating and executing Benthos message batch policies.
|
Package policy provides tooling for creating and executing Benthos message batch policies. |
|
bloblang/field
Package field implements a bloblang interpolation function templating syntax used in some dynamic fields within Benthos.
|
Package field implements a bloblang interpolation function templating syntax used in some dynamic fields within Benthos. |
|
bloblang/mapping
Package mapping provides a parser for the full bloblang mapping spec.
|
Package mapping provides a parser for the full bloblang mapping spec. |
|
bloblang/query
Package query provides a parser for the right-hand side query part of the bloblang spec.
|
Package query provides a parser for the right-hand side query part of the bloblang spec. |
|
bundle
Package bundle contains singletons referenced throughout the Benthos codebase that allow imported components to add their constructors and documentation to a service.
|
Package bundle contains singletons referenced throughout the Benthos codebase that allow imported components to add their constructors and documentation to a service. |
|
component/input/config
Package config contains reusable config definitions and parsers for inputs defined via the public/service package.
|
Package config contains reusable config definitions and parsers for inputs defined via the public/service package. |
|
docs
Package docs provides useful functions for creating documentation from Benthos components
|
Package docs provides useful functions for creating documentation from Benthos components |
|
impl/io
Package io contains component implementations that have a small dependency footprint (mostly standard library) and interact with external systems via the filesystem and/or network sockets.
|
Package io contains component implementations that have a small dependency footprint (mostly standard library) and interact with external systems via the filesystem and/or network sockets. |
|
impl/pure
Package pure contains all component implementations that are pure, in that they do not interact with external systems.
|
Package pure contains all component implementations that are pure, in that they do not interact with external systems. |
|
manager
Package manager implements the types.Manager interface used for creating and sharing resources across a Benthos service.
|
Package manager implements the types.Manager interface used for creating and sharing resources across a Benthos service. |
|
old/util/throttle
Package throttle implements throttle strategies.
|
Package throttle implements throttle strategies. |
|
pipeline
Package pipeline contains structures that implement both the Producer and Consumer interfaces.
|
Package pipeline contains structures that implement both the Producer and Consumer interfaces. |
|
stream/manager
Package manager creates and manages multiple streams, providing an API for performing CRUD operations.
|
Package manager creates and manages multiple streams, providing an API for performing CRUD operations. |
|
tls
Package tls provides Benthos configuration fields and wrappers for a crypto/tls config.
|
Package tls provides Benthos configuration fields and wrappers for a crypto/tls config. |
|
tracing
Package tracing implements utility functions for interacting with a global tracing system.
|
Package tracing implements utility functions for interacting with a global tracing system. |
|
tracing/v2
Package tracing implements utility functions for interacting with a global tracing system.
|
Package tracing implements utility functions for interacting with a global tracing system. |
|
public
|
|
|
bloblang
Package bloblang provides high level APIs for registering custom Bloblang plugins, as well as for parsing and executing Bloblang mappings.
|
Package bloblang provides high level APIs for registering custom Bloblang plugins, as well as for parsing and executing Bloblang mappings. |
|
components/io
Package io contains component implementations that have a small dependency footprint (mostly standard library) and interact with external systems via the filesystem and/or network sockets.
|
Package io contains component implementations that have a small dependency footprint (mostly standard library) and interact with external systems via the filesystem and/or network sockets. |
|
components/pure
Package pure imports all component implementations that are pure, in that they do not interact with external systems.
|
Package pure imports all component implementations that are pure, in that they do not interact with external systems. |
|
components/pure/extended
Package extended contains component implementations that have a larger dependency footprint but do not interact with external systems (so an extension of pure components)
|
Package extended contains component implementations that have a larger dependency footprint but do not interact with external systems (so an extension of pure components) |
|
service
Package service provides a high level API for registering custom plugin components and executing either a standard Benthos CLI, or programmatically building isolated pipelines with a StreamBuilder API.
|
Package service provides a high level API for registering custom plugin components and executing either a standard Benthos CLI, or programmatically building isolated pipelines with a StreamBuilder API. |
|
service/servicetest
Package servicetest provides functions and utilities that might be useful for testing custom Benthos builds.
|
Package servicetest provides functions and utilities that might be useful for testing custom Benthos builds. |
|
wasm/tinygo
Package tinygo provides entry points that allow WASM modules compiled with TinyGo to be executed by Benthos using the `wasm` processor.
|
Package tinygo provides entry points that allow WASM modules compiled with TinyGo to be executed by Benthos using the `wasm` processor. |
Click to show internal directories.
Click to hide internal directories.