Documentation
¶
Overview ¶
Package lib contains a collection of packages used to create a Benthos stream pipeline. These packages can be used by third parties in order to easily orchestrate stream processing applications using Benthos as a foundation.
Please refer to the stream package for examples of how to create Benthos streams with your own custom processor implementations.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package api implements a type used for creating the Benthos HTTP API.
|
Package api implements a type used for creating the Benthos HTTP API. |
|
Package bloblang implements a language for mapping within Benthos and the parser combinators for parsing them.
|
Package bloblang implements a language for mapping within Benthos and the parser combinators for parsing them. |
|
x/field
Package field implements a bloblang interpolation function templating syntax used in some dynamic fields within Benthos.
|
Package field implements a bloblang interpolation function templating syntax used in some dynamic fields within Benthos. |
|
x/mapping
Package mapping provides a parser for the full bloblang mapping spec.
|
Package mapping provides a parser for the full bloblang mapping spec. |
|
x/query
Package query provides a parser for the right-hand side query part of the bloblang spec.
|
Package query provides a parser for the right-hand side query part of the bloblang spec. |
|
Package broker implements types used for routing inputs to outputs in non-trivial arrangements, such as fan-out or fan-in models.
|
Package broker implements types used for routing inputs to outputs in non-trivial arrangements, such as fan-out or fan-in models. |
|
Package buffer is both a types.Consumer and types.Producer implementation that is able to sit between other stream components, effectively decoupling their transaction channels by storing messages in a buffer implementation.
|
Package buffer is both a types.Consumer and types.Producer implementation that is able to sit between other stream components, effectively decoupling their transaction channels by storing messages in a buffer implementation. |
|
parallel
Package parallel contains implementations of various buffer types where the buffer can be consumed by any number of parallel consumer threads.
|
Package parallel contains implementations of various buffer types where the buffer can be consumed by any number of parallel consumer threads. |
|
single
Package single contains implementations of various buffer types where the buffer can only be consumed by a single thread (but any number of writers).
|
Package single contains implementations of various buffer types where the buffer can only be consumed by a single thread (but any number of writers). |
|
Package cache implements the types.Cache interface for storing and retrieving key/value pairs from a range of storage strategies.
|
Package cache implements the types.Cache interface for storing and retrieving key/value pairs from a range of storage strategies. |
|
Package condition contains logical operators that, based on their configuration, return boolean values from messages under certain circumstances.
|
Package condition contains logical operators that, based on their configuration, return boolean values from messages under certain circumstances. |
|
Package config defines the full configuration structure used by the Benthos service, including methods of normalising and linting user configuration.
|
Package config defines the full configuration structure used by the Benthos service, including methods of normalising and linting user configuration. |
|
Package input defines consumers for aggregating data from a variety of sources.
|
Package input defines consumers for aggregating data from a variety of sources. |
|
reader
Package reader defines implementations of an interface for generic message reading from various third party sources.
|
Package reader defines implementations of an interface for generic message reading from various third party sources. |
|
Package log contains utilities for logging in a modular interface.
|
Package log contains utilities for logging in a modular interface. |
|
Package manager implements the types.Manager interface used for creating and sharing resources across a Benthos service.
|
Package manager implements the types.Manager interface used for creating and sharing resources across a Benthos service. |
|
Package message contains implementations of types.Message.
|
Package message contains implementations of types.Message. |
|
batch
Package batch provides tooling for creating and executing Benthos message batch policies.
|
Package batch provides tooling for creating and executing Benthos message batch policies. |
|
io
Package io provides tooling for serialising and deserialising messages.
|
Package io provides tooling for serialising and deserialising messages. |
|
mapper
Package mapper implements ways of splicing and mapping batches of messages so that their derivative parts can be processed individually and merged back into the original batch.
|
Package mapper implements ways of splicing and mapping batches of messages so that their derivative parts can be processed individually and merged back into the original batch. |
|
metadata
Package metadata contains implementations of types.Metadata.
|
Package metadata contains implementations of types.Metadata. |
|
roundtrip
Package roundtrip provides tooling for composing request/response behaviour through messages.
|
Package roundtrip provides tooling for composing request/response behaviour through messages. |
|
tracing
Package tracing implements utility functions for recording opentracing events for messages.
|
Package tracing implements utility functions for recording opentracing events for messages. |
|
Package metrics contains a type for aggregating and propagating metrics to various services based on configuration.
|
Package metrics contains a type for aggregating and propagating metrics to various services based on configuration. |
|
Package output defines all sinks for sending Benthos messages to a variety of third party destinations.
|
Package output defines all sinks for sending Benthos messages to a variety of third party destinations. |
|
writer
Package writer defines implementations of an interface for generic message writing that outputs to various third party sinks.
|
Package writer defines implementations of an interface for generic message writing that outputs to various third party sinks. |
|
Package pipeline contains structures that implement both the Producer and Consumer interfaces.
|
Package pipeline contains structures that implement both the Producer and Consumer interfaces. |
|
Package processor contains implementations of types.Processor, which perform an arbitrary operation on a message and either returns >0 messages to be propagated towards a sink, or a response to be sent back to the message source.
|
Package processor contains implementations of types.Processor, which perform an arbitrary operation on a message and either returns >0 messages to be propagated towards a sink, or a response to be sent back to the message source. |
|
Package ratelimit implements the types.RateLimit interface for limiting access to resources shared service wide.
|
Package ratelimit implements the types.RateLimit interface for limiting access to resources shared service wide. |
|
Package response contains implementations of types.Response.
|
Package response contains implementations of types.Response. |
|
Package serverless contains shared components for serverless distributions of Benthos.
|
Package serverless contains shared components for serverless distributions of Benthos. |
|
lambda
Package lambda contains the execution logic for running Benthos as an AWS lambda function.
|
Package lambda contains the execution logic for running Benthos as an AWS lambda function. |
|
Package service contains the main execution logic of the Benthos service, which is responsible for parsing commandline flags, running a stream and capturing termination signals.
|
Package service contains the main execution logic of the Benthos service, which is responsible for parsing commandline flags, running a stream and capturing termination signals. |
|
test
Package test implements the Benthos service unit testing command.
|
Package test implements the Benthos service unit testing command. |
|
Package stream creates and manages a full Benthos stream pipeline, consisting of an input layer of consumers, an optional buffer layer, a processing pipelines layer, and an output layer of producers: Inputs -> Buffer -> Processing Pipelines -> Outputs The number of parallel input consumers, processing pipelines, and output producers depends on the configuration of the stream.
|
Package stream creates and manages a full Benthos stream pipeline, consisting of an input layer of consumers, an optional buffer layer, a processing pipelines layer, and an output layer of producers: Inputs -> Buffer -> Processing Pipelines -> Outputs The number of parallel input consumers, processing pipelines, and output producers depends on the configuration of the stream. |
|
manager
Package manager creates and manages multiple streams, providing an API for performing CRUD operations.
|
Package manager creates and manages multiple streams, providing an API for performing CRUD operations. |
|
test
|
|
|
integration
Package integration implements integration tests using docker.
|
Package integration implements integration tests using docker. |
|
Package tracer contains components able to send opentracing events.
|
Package tracer contains components able to send opentracing events. |
|
Package types defines any general structs and interfaces used throughout the benthos code base.
|
Package types defines any general structs and interfaces used throughout the benthos code base. |
|
checkpoint
Package checkpoint implements a mechanism for tracking checkpointed integer offsets for sequential read at-least-once queue systems such as Kafka or Kinesis.
|
Package checkpoint implements a mechanism for tracking checkpointed integer offsets for sequential read at-least-once queue systems such as Kafka or Kinesis. |
|
config
Package config contains utilities for reading and parsing service configuration files.
|
Package config contains utilities for reading and parsing service configuration files. |
|
disk
Package disk contains cross platform disk statistics utilities.
|
Package disk contains cross platform disk statistics utilities. |
|
http/auth
Package auth provides configuration fields and implementations of HTTP request authentication strategies.
|
Package auth provides configuration fields and implementations of HTTP request authentication strategies. |
|
retries
Package retries implements backoff strategies around a standard configuration scheme.
|
Package retries implements backoff strategies around a standard configuration scheme. |
|
text
Package text includes utilities for working with text that might contain variables.
|
Package text includes utilities for working with text that might contain variables. |
|
throttle
Package throttle implements throttle strategies.
|
Package throttle implements throttle strategies. |
|
tls
Package tls provides Benthos configuration fields and wrappers for a crypto/tls config.
|
Package tls provides Benthos configuration fields and wrappers for a crypto/tls config. |
|
Package x contains experimental packages that are subject to change outside of major version releases.
|
Package x contains experimental packages that are subject to change outside of major version releases. |
|
docs
Package docs provides useful functions for creating documentation from Benthos components
|
Package docs provides useful functions for creating documentation from Benthos components |
Click to show internal directories.
Click to hide internal directories.