types

package
v3.47.0-rc1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 1, 2021 License: MIT Imports: 5 Imported by: 44

Documentation

Overview

Package types defines any general structs and interfaces used throughout the benthos code base.

Benthos uses abstract types to represent arbitrary producers and consumers of data to its core components. This allows us to construct types for piping data in various arrangements without regard for the specific destinations and sources of our data.

The basic principle behind a producer/consumer relationship is that a producer pipes data to the consumer in lock-step, where for each message sent it will expect a response that confirms the message was received and propagated onwards.

Messages and responses are sent via channels, and in order to instigate this pairing each type is expected to create and maintain ownership of its respective sending channel.

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrTimeout    = errors.New("action timed out")
	ErrChanClosed = errors.New("channel was closed unexpectedly")
	ErrTypeClosed = errors.New("type was closed")

	ErrNotConnected = errors.New("not connected to target source or sink")

	ErrInvalidProcessorType = errors.New("processor type was not recognised")
	ErrInvalidCacheType     = errors.New("cache type was not recognised")
	ErrInvalidRateLimitType = errors.New("rate_limit type was not recognised")
	ErrInvalidConditionType = errors.New("condition type was not recognised")
	ErrInvalidBufferType    = errors.New("buffer type was not recognised")
	ErrInvalidInputType     = errors.New("input type was not recognised")
	ErrInvalidOutputType    = errors.New("output type was not recognised")

	ErrInvalidZMQType        = errors.New("invalid ZMQ socket type")
	ErrInvalidScaleProtoType = errors.New("invalid Scalability Protocols socket type")

	// ErrAlreadyStarted is returned when an input or output type gets started a
	// second time.
	ErrAlreadyStarted = errors.New("type has already been started")

	ErrMessagePartNotExist = errors.New("target message part does not exist")
	ErrBadMessageBytes     = errors.New("serialised message bytes were in unexpected format")
	ErrBlockCorrupted      = errors.New("serialised messages block was in unexpected format")

	ErrNoAck = errors.New("failed to receive acknowledgement")
)

Errors used throughout the codebase

View Source 
var (
	ErrInputNotFound     = errors.New("input not found")
	ErrCacheNotFound     = errors.New("cache not found")
	ErrConditionNotFound = errors.New("condition not found")
	ErrProcessorNotFound = errors.New("processor not found")
	ErrRateLimitNotFound = errors.New("rate limit not found")
	ErrOutputNotFound    = errors.New("output not found")
	ErrPluginNotFound    = errors.New("plugin not found")
	ErrKeyAlreadyExists  = errors.New("key already exists")
	ErrKeyNotFound       = errors.New("key does not exist")
	ErrPipeNotFound      = errors.New("pipe was not found")
)

Manager errors

View Source 
var (
	ErrMessageTooLarge = errors.New("message body larger than buffer space")
)

Buffer errors

View Source 
var FailFlagKey = "benthos_processing_failed"

FailFlagKey is a metadata key used for flagging processor errors in Benthos. If a message part has any non-empty value for this metadata key then it will be interpretted as having failed a processor step somewhere in the pipeline.

Functions

This section is empty.

Types

type BatchError added in v3.22.0 

type BatchError struct {
	// contains filtered or unexported fields
}

BatchError represents a general error across an entire batch of messages that can optionally be broken down into errors at the individual message level, identified by their index in the batch.

func NewBatchError added in v3.22.0 

func NewBatchError(err error) *BatchError

NewBatchError creates a fresh batch error from a general error message. Under normal use this root error will be provided. Once a batch error is initialized it is possible to add index specific errors with AddErrAt.

func (*BatchError) AddErrAt added in v3.22.0 

func (b *BatchError) AddErrAt(i int, err error) *BatchError

AddErrAt adds an error for a message at a specific index. If an error for the given index already exists it is overridden. A reference to the BatchError is returned for convenient chaining.

func (*BatchError) Error added in v3.22.0 

func (b *BatchError) Error() string

Error implements the common error interface.

func (*BatchError) IndexedErrors added in v3.22.0 

func (b *BatchError) IndexedErrors() map[int]error

IndexedErrors returns a map of message indexes to their individual errors.

func (*BatchError) Unwrap added in v3.22.0 

func (b *BatchError) Unwrap() error

Unwrap returns the underlying common error.

type Cache 

type Cache interface {
	// Get attempts to locate and return a cached value by its key, returns an
	// error if the key does not exist or if the command fails.
	Get(key string) ([]byte, error)

	// Set attempts to set the value of a key, returns an error if the command
	// fails.
	Set(key string, value []byte) error

	// SetMulti attempts to set the value of multiple keys, returns an error if
	// any of the keys fail.
	SetMulti(items map[string][]byte) error

	// Add attempts to set the value of a key only if the key does not already
	// exist, returns an error if the key already exists or if the command
	// fails.
	Add(key string, value []byte) error

	// Delete attempts to remove a key. Returns an error if a failure occurs.
	Delete(key string) error

	Closable
}

Cache is a key/value store that can be shared across components and executing threads of a Benthos service.

type CacheTTLItem added in v3.33.0 

type CacheTTLItem struct {
	Value []byte
	TTL   *time.Duration
}

CacheTTLItem contains a value to cache along with an optional TTL.

type CacheWithTTL added in v3.33.0 

type CacheWithTTL interface {
	// SetWithTTL attempts to set the value of a key, returns an error if the
	// command fails.
	SetWithTTL(key string, value []byte, ttl *time.Duration) error

	// SetMultiWithTTL attempts to set the value of multiple keys, returns an
	// error if any of the keys fail.
	SetMultiWithTTL(items map[string]CacheTTLItem) error

	// AddWithTTL attempts to set the value of a key only if the key does not
	// already exist, returns an error if the key already exists or if the
	// command fails.
	AddWithTTL(key string, value []byte, ttl *time.Duration) error

	Cache
}

CacheWithTTL is a key/value store that can be shared across components and executing threads of a Benthos service.

type Closable 

type Closable interface {
	// CloseAsync triggers the shut down of this component but should not block
	// the calling goroutine.
	CloseAsync()

	// WaitForClose is a blocking call to wait until the component has finished
	// shutting down and cleaning up resources.
	WaitForClose(timeout time.Duration) error
}

Closable defines a type that can be safely closed down and cleaned up. This interface is required for many components within Benthos, but if your implementation is stateless and does not require shutting down then this interface can be implemented with empty shims.

type Condition 

type Condition interface {
	// Check tests a message against a configured condition.
	Check(msg Message) bool
}

Condition reads a message, calculates a condition and returns a boolean.

type Consumer 

type Consumer interface {
	// Consume starts the type receiving transactions from a Transactor.
	Consume(<-chan Transaction) error
}

Consumer is the higher level consumer type.

type DudMgr 

type DudMgr struct {
	ID int
}

DudMgr is a noop implementation of a types.Manager.

func (DudMgr) GetCache 

func (f DudMgr) GetCache(name string) (Cache, error)

GetCache always returns ErrCacheNotFound.

func (DudMgr) GetCondition 

func (f DudMgr) GetCondition(name string) (Condition, error)

GetCondition always returns ErrConditionNotFound.

func (DudMgr) GetPipe 

func (f DudMgr) GetPipe(name string) (<-chan Transaction, error)

GetPipe attempts to find a service wide message producer by its name.

func (DudMgr) GetPlugin 

func (f DudMgr) GetPlugin(name string) (interface{}, error)

GetPlugin always returns ErrPluginNotFound.

func (DudMgr) GetRateLimit 

func (f DudMgr) GetRateLimit(name string) (RateLimit, error)

GetRateLimit always returns ErrRateLimitNotFound.

func (DudMgr) RegisterEndpoint 

func (f DudMgr) RegisterEndpoint(path, desc string, h http.HandlerFunc)

RegisterEndpoint is a noop.

func (DudMgr) SetPipe 

func (f DudMgr) SetPipe(name string, t <-chan Transaction)

SetPipe registers a message producer under a name.

func (DudMgr) UnsetPipe 

func (f DudMgr) UnsetPipe(name string, t <-chan Transaction)

UnsetPipe removes a named pipe.

type ErrUnexpectedHTTPRes 

type ErrUnexpectedHTTPRes struct {
	Code int
	S    string
}

ErrUnexpectedHTTPRes is an error returned when an HTTP request returned an unexpected response.

func (ErrUnexpectedHTTPRes) Error 

func (e ErrUnexpectedHTTPRes) Error() string

Error returns the Error string.

type HTTPMessage 

type HTTPMessage struct {
	Parts []string `json:"parts"`
}

HTTPMessage is a struct containing a benthos message, to be sent over the wire in an HTTP request. Message parts are sent as JSON strings.

type HTTPResponse 

type HTTPResponse struct {
	Error string `json:"error"`
}

HTTPResponse is a struct expected as an HTTP response after sending a message. The intention is to confirm that the message has been received successfully.

type Input 

type Input interface {
	Producer
	Closable

	// Connected returns a boolean indicating whether this input is currently
	// connected to its target.
	Connected() bool
}

Input is a closable Producer.

type Manager 

type Manager interface {
	// RegisterEndpoint registers a server wide HTTP endpoint.
	RegisterEndpoint(path, desc string, h http.HandlerFunc)

	// GetCache attempts to find a service wide cache by its name.
	GetCache(name string) (Cache, error)

	// GetCondition attempts to find a service wide condition by its name.
	GetCondition(name string) (Condition, error)

	// GetRateLimit attempts to find a service wide rate limit by its name.
	GetRateLimit(name string) (RateLimit, error)

	// GetPlugin attempts to find a service wide resource plugin by its name.
	GetPlugin(name string) (interface{}, error)

	// GetPipe attempts to find a service wide transaction chan by its name.
	GetPipe(name string) (<-chan Transaction, error)

	// SetPipe registers a transaction chan under a name.
	SetPipe(name string, t <-chan Transaction)

	// UnsetPipe removes a named transaction chan.
	UnsetPipe(name string, t <-chan Transaction)
}

Manager is an interface expected by Benthos components that allows them to register their service wide behaviours such as HTTP endpoints and event listeners, and obtain service wide shared resources such as caches.

func NoopMgr 

func NoopMgr() Manager

NoopMgr returns a Manager implementation that does nothing.

type Message 

type Message interface {
	// Get attempts to access a message part from an index. If the index is
	// negative then the part is found by counting backwards from the last part
	// starting at -1. If the index is out of bounds then an empty part is
	// returned.
	Get(p int) Part

	// SetAll replaces all parts of a message with a new set.
	SetAll(parts []Part)

	// Append appends new message parts to the message and returns the index of
	// last part to be added.
	Append(part ...Part) int

	// Len returns the number of parts this message contains.
	Len() int

	// Iter will iterate each message part in order, calling the closure
	// argument with the index and contents of the message part.
	Iter(f func(i int, part Part) error) error

	// Copy creates a shallow copy of the message, where the list of message
	// parts can be edited independently from the original version. However,
	// editing the byte array contents of a message part will alter the contents
	// of the original, and if another process edits the bytes of the original
	// it will also affect the contents of this message.
	Copy() Message

	// DeepCopy creates a deep copy of the message, where the message part
	// contents are entirely copied and are therefore safe to edit without
	// altering the original.
	DeepCopy() Message

	// CreatedAt returns the time at which the message was created.
	CreatedAt() time.Time
}

Message is an interface representing a payload of data that was received from an input. Messages contain multiple parts, where each part is a byte array. If an input supports only single part messages they will still be read as multipart messages with one part. Multiple part messages are synonymous with batches, and it is up to each component within Benthos to work with a batch appropriately.

type Metadata 

type Metadata interface {
	// Get returns a metadata value if a key exists, otherwise an empty string.
	Get(key string) string

	// Set sets the value of a metadata key.
	Set(key, value string) Metadata

	// Delete removes the value of a metadata key.
	Delete(key string) Metadata

	// Iter iterates each metadata key/value pair.
	Iter(f func(k, v string) error) error

	// Copy returns a copy of the metadata object that can be edited without
	// changing the contents of the original.
	Copy() Metadata
}

Metadata is an interface representing the metadata of a message part within a batch.

type Output 

type Output interface {
	Consumer
	Closable

	// Connected returns a boolean indicating whether this output is currently
	// connected to its target.
	Connected() bool
}

Output is a closable Consumer.

type OutputWriter added in v3.12.0 

type OutputWriter interface {
	Closable

	// WriteTransaction attempts to write a transaction to an output.
	WriteTransaction(context.Context, Transaction) error

	// Connected returns a boolean indicating whether this output is currently
	// connected to its target.
	Connected() bool
}

OutputWriter is a non-channel based output interface.

type Part 

type Part interface {
	// Get returns a slice of bytes which is the underlying data of the part.
	// It is not safe to edit the contents of this slice directly, to make
	// changes to the contents of a part the data should be copied and set using
	// SetData.
	Get() []byte

	// Metadata returns the metadata of a part.
	Metadata() Metadata

	// JSON attempts to parse the part as a JSON document and either returns the
	// result or an error. The resulting document is also cached such that
	// subsequent calls do not reparse the same data. If changes are made to the
	// document it must be set using SetJSON, otherwise the underlying byte
	// representation will not reflect the changes.
	JSON() (interface{}, error)

	// Set changes the underlying byte slice. Returns the edited message part
	// for daisy-chaining further calls.
	Set(d []byte) Part

	// SetMetadata changes the underlying metadata to a new object. Returns the
	// edited message part for daisy-chaining further calls.
	SetMetadata(m Metadata) Part

	// SetJSON attempts to marshal a JSON document into a byte slice and stores
	// the result as the new contents of the part. The document is cached such
	// that subsequent calls to JSON() receive it rather than reparsing the
	// resulting byte slice.
	SetJSON(doc interface{}) error

	// IsEmpty returns true if the message part has zero contents (not including
	// metadata).
	IsEmpty() bool

	// Copy creates a shallow copy of the message, where values and metadata can
	// be edited independently from the original version. However, editing the
	// byte slice contents will alter the contents of the original, and if
	// another process edits the bytes of the original it will also affect the
	// contents of this message.
	Copy() Part

	// DeepCopy creates a deep copy of the message part, where the contents are
	// copied and are therefore safe to edit without altering the original.
	DeepCopy() Part
}

Part is an interface representing a message part. It contains a byte array of raw data, metadata, and lazily parsed formats of the payload such as JSON.

type Pipeline 

type Pipeline interface {
	Producer
	Consumer
	Closable
}

Pipeline is an interface that implements both the Consumer and Producer interfaces, and can therefore be used to pipe messages from Producer to a Consumer.

type PipelineConstructorFunc 

type PipelineConstructorFunc func(i *int) (Pipeline, error)

PipelineConstructorFunc is a constructor to be called for each parallel stream pipeline thread in order to construct a custom pipeline implementation.

An integer pointer is provided to pipeline constructors that tracks the number of components spanning multiple pipelines. Each pipeline is expected to increment i by the number of components they contain, and may use the value for metric and logging namespacing.

type Processor 

type Processor interface {
	// ProcessMessage attempts to process a message. This method returns both a
	// slice of messages or a response indicating whether messages were dropped
	// due to an intermittent error (response.NewError) or were intentionally
	// filtered (response.NewAck).
	//
	// If an error occurs due to the contents of a message being invalid and you
	// wish to expose this as a recoverable fault you can use processor.FlagErr
	// to flag a message as having failed without dropping it.
	//
	// More information about this form of error handling can be found at:
	// https://www.benthos.dev/docs/configuration/error_handling
	ProcessMessage(Message) ([]Message, Response)

	Closable
}

Processor reads a message, performs some form of data processing to the message, and returns either a slice of >= 1 resulting messages or a response to return to the message origin.

type ProcessorConstructorFunc 

type ProcessorConstructorFunc func() (Processor, error)

ProcessorConstructorFunc is a constructor to be called for each parallel stream pipeline thread in order to construct a custom processor implementation.

type Producer 

type Producer interface {
	// TransactionChan returns a channel used for consuming transactions from
	// this type. Every transaction received must be resolved before another
	// transaction will be sent.
	TransactionChan() <-chan Transaction
}

Producer is a type that sends messages as transactions and waits for a response back, the response indicates whether the message was successfully propagated to a new destination (and can be discarded from the source.)

type RateLimit 

type RateLimit interface {
	// Access the rate limited resource. Returns a duration or an error if the
	// rate limit check fails. The returned duration is either zero (meaning the
	// resource may be accessed) or a reasonable length of time to wait before
	// requesting again.
	Access() (time.Duration, error)

	Closable
}

RateLimit is a strategy for limiting access to a shared resource, this strategy can be safely used by components in parallel.

type Response 

type Response interface {
	// Error returns a non-nil error if the message failed to reach its
	// destination.
	Error() error

	// SkipAck indicates that even though there may not have been an error in
	// processing a message, it should not be acknowledged. If SkipAck is false
	// and Error is nil then all unacknowledged messages should be acknowledged
	// also.
	//
	// TODO: V4 Remove this.
	SkipAck() bool
}

Response is a response from an output, agent or broker that confirms the input of successful message receipt.

type Transaction 

type Transaction struct {
	// Payload is the message payload of this transaction.
	Payload Message

	// ResponseChan should receive a response at the end of a transaction (once
	// the message is no longer owned by the receiver.) The response itself
	// indicates whether the message has been propagated successfully.
	ResponseChan chan<- Response
}

Transaction is a type respesenting a transaction containing a payload (the message) and a response channel, which is used to indicate whether the message was successfully propagated to the next destination.

func NewTransaction 

func NewTransaction(payload Message, resChan chan<- Response) Transaction

NewTransaction creates a new transaction object from a message payload and a response channel.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.