input

package
v1.1.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 6, 2023 License: MIT Imports: 20 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func AsyncReaderWithConnBackOff added in v1.1.2 

func AsyncReaderWithConnBackOff(boff backoff.BackOff) func(a *AsyncReader)

AsyncReaderWithConnBackOff set the backoff used for limiting connection attempts. If the maximum number of retry attempts is reached then the input will gracefully stop.

Types

type Async 

type Async interface {
	// Connect attempts to establish a connection to the source, if
	// unsuccessful returns an error. If the attempt is successful (or not
	// necessary) returns nil.
	Connect(ctx context.Context) error

	// ReadBatch attempts to read a new message from the source. If
	// successful a message is returned along with a function used to
	// acknowledge receipt of the returned message. It's safe to process the
	// returned message and read the next message asynchronously.
	ReadBatch(ctx context.Context) (message.Batch, AsyncAckFn, error)

	// Close triggers the shut down of this component and blocks until
	// completion or context cancellation.
	Close(ctx context.Context) error
}

Async is a type that reads Benthos messages from an external source and allows acknowledgements for a message batch to be propagated asynchronously.

type AsyncAckFn 

type AsyncAckFn func(context.Context, error) error

AsyncAckFn is a function used to acknowledge receipt of a message batch. The provided response indicates whether the message batch was successfully delivered. Returns an error if the acknowledge was not propagated.

type AsyncCutOff 

type AsyncCutOff struct {
	// contains filtered or unexported fields
}

AsyncCutOff is a wrapper for input.Async implementations that exits from WaitForClose immediately. This is only useful when the underlying readable resource cannot be closed reliably and can block forever.

func NewAsyncCutOff 

func NewAsyncCutOff(r Async) *AsyncCutOff

NewAsyncCutOff returns a new AsyncCutOff wrapper around a input.Async.

func (*AsyncCutOff) Close 

func (c *AsyncCutOff) Close(ctx context.Context) error

Close triggers the asynchronous closing of the reader.

func (*AsyncCutOff) Connect 

func (c *AsyncCutOff) Connect(ctx context.Context) error

Connect attempts to establish a connection to the source, if unsuccessful returns an error. If the attempt is successful (or not necessary) returns nil.

func (*AsyncCutOff) ReadBatch 

func (c *AsyncCutOff) ReadBatch(ctx context.Context) (message.Batch, AsyncAckFn, error)

ReadBatch attempts to read a new message from the source.

type AsyncPreserver 

type AsyncPreserver struct {
	// contains filtered or unexported fields
}

AsyncPreserver is a wrapper for input.Async implementations that keeps a buffer of sent messages until they are acknowledged. If an error occurs during message propagation the contents of the buffer will be resent instead of reading new messages until it is depleted. AsyncPreserver implements input.Async.

Wrapping an input with this type is useful when your source of messages doesn't have a concept of a NoAck (like Kafka), and instead of "rejecting" messages we always intend to simply retry them until success.

func NewAsyncPreserver 

func NewAsyncPreserver(r Async) *AsyncPreserver

NewAsyncPreserver returns a new AsyncPreserver wrapper around a input.Async.

func (*AsyncPreserver) Close 

func (p *AsyncPreserver) Close(ctx context.Context) error

Close triggers the shut down of this component and blocks until completion or context cancellation.

func (*AsyncPreserver) Connect 

func (p *AsyncPreserver) Connect(ctx context.Context) error

Connect attempts to establish a connection to the source, if unsuccessful returns an error. If the attempt is successful (or not necessary) returns nil.

func (*AsyncPreserver) ReadBatch 

func (p *AsyncPreserver) ReadBatch(ctx context.Context) (message.Batch, AsyncAckFn, error)

ReadBatch attempts to read a new message from the source.

type AsyncReader 

type AsyncReader struct {
	// contains filtered or unexported fields
}

AsyncReader is an input implementation that reads messages from an input.Async component.

func (*AsyncReader) Connected 

func (r *AsyncReader) Connected() bool

Connected returns a boolean indicating whether this input is currently connected to its target.

func (*AsyncReader) TransactionChan 

func (r *AsyncReader) TransactionChan() <-chan message.Transaction

TransactionChan returns a transactions channel for consuming messages from this input type.

func (*AsyncReader) TriggerCloseNow 

func (r *AsyncReader) TriggerCloseNow()

TriggerCloseNow triggers the shut down of this component but should not block the calling goroutine.

func (*AsyncReader) TriggerStopConsuming 

func (r *AsyncReader) TriggerStopConsuming()

TriggerStopConsuming instructs the input to start shutting down resources once all pending messages are delivered and acknowledged. This call does not block.

func (*AsyncReader) WaitForClose 

func (r *AsyncReader) WaitForClose(ctx context.Context) error

WaitForClose is a blocking call to wait until the component has finished shutting down and cleaning up resources.

type BrokerConfig 

type BrokerConfig struct {
	Copies   int                `json:"copies" yaml:"copies"`
	Inputs   []Config           `json:"inputs" yaml:"inputs"`
	Batching batchconfig.Config `json:"batching" yaml:"batching"`
}

BrokerConfig contains configuration fields for the Broker input type.

func NewBrokerConfig 

func NewBrokerConfig() BrokerConfig

NewBrokerConfig creates a new BrokerConfig with default values.

type Config 

type Config struct {
	Label        string             `json:"label" yaml:"label"`
	Type         string             `json:"type" yaml:"type"`
	Broker       BrokerConfig       `json:"broker" yaml:"broker"`
	Dynamic      DynamicConfig      `json:"dynamic" yaml:"dynamic"`
	Generate     GenerateConfig     `json:"generate" yaml:"generate"`
	Inproc       InprocConfig       `json:"inproc" yaml:"inproc"`
	Nanomsg      NanomsgConfig      `json:"nanomsg" yaml:"nanomsg"`
	NSQ          NSQConfig          `json:"nsq" yaml:"nsq"`
	Plugin       any                `json:"plugin,omitempty" yaml:"plugin,omitempty"`
	ReadUntil    ReadUntilConfig    `json:"read_until" yaml:"read_until"`
	Resource     string             `json:"resource" yaml:"resource"`
	Sequence     SequenceConfig     `json:"sequence" yaml:"sequence"`
	SFTP         SFTPConfig         `json:"sftp" yaml:"sftp"`
	Socket       SocketConfig       `json:"socket" yaml:"socket"`
	SocketServer SocketServerConfig `json:"socket_server" yaml:"socket_server"`
	STDIN        STDINConfig        `json:"stdin" yaml:"stdin"`
	Subprocess   SubprocessConfig   `json:"subprocess" yaml:"subprocess"`
	Processors   []processor.Config `json:"processors" yaml:"processors"`
}

Config is the all encompassing configuration struct for all input types. Deprecated: Do not add new components here. Instead, use the public plugin APIs. Examples can be found in: ./internal/impl.

func NewConfig 

func NewConfig() Config

NewConfig returns a configuration struct fully populated with default values. Deprecated: Do not add new components here. Instead, use the public plugin APIs. Examples can be found in: ./internal/impl.

func (*Config) UnmarshalYAML 

func (conf *Config) UnmarshalYAML(value *yaml.Node) error

UnmarshalYAML ensures that when parsing configs that are in a map or slice the default values are still applied.

type DynamicConfig 

type DynamicConfig struct {
	Inputs map[string]Config `json:"inputs" yaml:"inputs"`
	Prefix string            `json:"prefix" yaml:"prefix"`
}

DynamicConfig contains configuration for the Dynamic input type.

func NewDynamicConfig 

func NewDynamicConfig() DynamicConfig

NewDynamicConfig creates a new DynamicConfig with default values.

type GenerateConfig 

type GenerateConfig struct {
	Mapping string `json:"mapping" yaml:"mapping"`
	// internal can be both duration string or cron expression
	Interval  string `json:"interval" yaml:"interval"`
	Count     int    `json:"count" yaml:"count"`
	BatchSize int    `json:"batch_size" yaml:"batch_size"`
}

GenerateConfig contains configuration for the Bloblang input type.

func NewGenerateConfig 

func NewGenerateConfig() GenerateConfig

NewGenerateConfig creates a new BloblangConfig with default values.

type InprocConfig 

type InprocConfig string

InprocConfig is a configuration type for the inproc input.

func NewInprocConfig 

func NewInprocConfig() InprocConfig

NewInprocConfig creates a new inproc input config.

type NSQConfig 

type NSQConfig struct {
	Addresses       []string    `json:"nsqd_tcp_addresses" yaml:"nsqd_tcp_addresses"`
	LookupAddresses []string    `json:"lookupd_http_addresses" yaml:"lookupd_http_addresses"`
	Topic           string      `json:"topic" yaml:"topic"`
	Channel         string      `json:"channel" yaml:"channel"`
	UserAgent       string      `json:"user_agent" yaml:"user_agent"`
	TLS             btls.Config `json:"tls" yaml:"tls"`
	MaxInFlight     int         `json:"max_in_flight" yaml:"max_in_flight"`
	MaxAttempts     uint16      `json:"max_attempts" yaml:"max_attempts"`
}

NSQConfig contains configuration fields for the NSQ input type.

func NewNSQConfig 

func NewNSQConfig() NSQConfig

NewNSQConfig creates a new NSQConfig with default values.

type NanomsgConfig 

type NanomsgConfig struct {
	URLs        []string `json:"urls" yaml:"urls"`
	Bind        bool     `json:"bind" yaml:"bind"`
	SocketType  string   `json:"socket_type" yaml:"socket_type"`
	SubFilters  []string `json:"sub_filters" yaml:"sub_filters"`
	PollTimeout string   `json:"poll_timeout" yaml:"poll_timeout"`
}

NanomsgConfig contains configuration fields for the nanomsg input type.

func NewNanomsgConfig 

func NewNanomsgConfig() NanomsgConfig

NewNanomsgConfig creates a new NanomsgConfig with default values.

type ReadUntilConfig 

type ReadUntilConfig struct {
	Input       *Config `json:"input" yaml:"input"`
	Restart     bool    `json:"restart_input" yaml:"restart_input"`
	Check       string  `json:"check" yaml:"check"`
	IdleTimeout string  `json:"idle_timeout" yaml:"idle_timeout"`
}

ReadUntilConfig contains configuration values for the ReadUntil input type.

func NewReadUntilConfig 

func NewReadUntilConfig() ReadUntilConfig

NewReadUntilConfig creates a new ReadUntilConfig with default values.

func (ReadUntilConfig) MarshalJSON 

func (r ReadUntilConfig) MarshalJSON() ([]byte, error)

MarshalJSON prints an empty object instead of nil.

func (ReadUntilConfig) MarshalYAML 

func (r ReadUntilConfig) MarshalYAML() (any, error)

MarshalYAML prints an empty object instead of nil.

type SFTPConfig 

type SFTPConfig struct {
	Address        string                `json:"address" yaml:"address"`
	Credentials    sftpSetup.Credentials `json:"credentials" yaml:"credentials"`
	Paths          []string              `json:"paths" yaml:"paths"`
	Codec          string                `json:"codec" yaml:"codec"`
	DeleteOnFinish bool                  `json:"delete_on_finish" yaml:"delete_on_finish"`
	MaxBuffer      int                   `json:"max_buffer" yaml:"max_buffer"`
	Watcher        watcherConfig         `json:"watcher" yaml:"watcher"`
}

SFTPConfig contains configuration fields for the SFTP input type.

func NewSFTPConfig 

func NewSFTPConfig() SFTPConfig

NewSFTPConfig creates a new SFTPConfig with default values.

type STDINConfig 

type STDINConfig struct {
	Codec     string `json:"codec" yaml:"codec"`
	MaxBuffer int    `json:"max_buffer" yaml:"max_buffer"`
}

STDINConfig contains config fields for the STDIN input type.

func NewSTDINConfig 

func NewSTDINConfig() STDINConfig

NewSTDINConfig creates a STDINConfig populated with default values.

type SequenceConfig 

type SequenceConfig struct {
	ShardedJoin SequenceShardedJoinConfig `json:"sharded_join" yaml:"sharded_join"`
	Inputs      []Config                  `json:"inputs" yaml:"inputs"`
}

SequenceConfig contains configuration values for the Sequence input type.

func NewSequenceConfig 

func NewSequenceConfig() SequenceConfig

NewSequenceConfig creates a new SequenceConfig with default values.

type SequenceShardedJoinConfig 

type SequenceShardedJoinConfig struct {
	Type          string `json:"type" yaml:"type"`
	IDPath        string `json:"id_path" yaml:"id_path"`
	Iterations    int    `json:"iterations" yaml:"iterations"`
	MergeStrategy string `json:"merge_strategy" yaml:"merge_strategy"`
}

SequenceShardedJoinConfig describes an optional mechanism for performing sharded joins of structured data resulting from the input sequence. This is a way to merge the structured fields of fragmented datasets within memory even when the overall size of the data surpasses the memory available on the machine.

When configured the sequence of inputs will be consumed multiple times according to the number of iterations, and each iteration will process an entirely different set of messages by sharding them by the ID field.

Each message must be structured (JSON or otherwise processed into a structured form) and the fields will be aggregated with those of other messages sharing the ID. At the end of each iteration the joined messages are flushed downstream before the next iteration begins.

func NewSequenceShardedJoinConfig 

func NewSequenceShardedJoinConfig() SequenceShardedJoinConfig

NewSequenceShardedJoinConfig creates a new sequence sharding configuration with default values.

type SocketConfig 

type SocketConfig struct {
	Network   string `json:"network" yaml:"network"`
	Address   string `json:"address" yaml:"address"`
	Codec     string `json:"codec" yaml:"codec"`
	MaxBuffer int    `json:"max_buffer" yaml:"max_buffer"`
}

SocketConfig contains configuration values for the Socket input type.

func NewSocketConfig 

func NewSocketConfig() SocketConfig

NewSocketConfig creates a new SocketConfig with default values.

type SocketServerConfig 

type SocketServerConfig struct {
	Network   string                `json:"network" yaml:"network"`
	Address   string                `json:"address" yaml:"address"`
	Codec     string                `json:"codec" yaml:"codec"`
	MaxBuffer int                   `json:"max_buffer" yaml:"max_buffer"`
	TLS       SocketServerTLSConfig `json:"tls" yaml:"tls"`
}

SocketServerConfig contains configuration for the SocketServer input type.

func NewSocketServerConfig 

func NewSocketServerConfig() SocketServerConfig

NewSocketServerConfig creates a new SocketServerConfig with default values.

type SocketServerTLSConfig added in v1.1.0 

type SocketServerTLSConfig struct {
	CertFile   string `json:"cert_file" yaml:"cert_file"`
	KeyFile    string `json:"key_file" yaml:"key_file"`
	SelfSigned bool   `json:"self_signed" yaml:"self_signed"`
}

SocketServerTLSConfig contains config for TLS.

type Streamed 

type Streamed interface {
	// TransactionChan returns a channel used for consuming transactions from
	// this type. Every transaction received must be resolved before another
	// transaction will be sent.
	TransactionChan() <-chan message.Transaction

	// Connected returns a boolean indicating whether this input is currently
	// connected to its target.
	Connected() bool

	// TriggerStopConsuming instructs the input to start shutting down resources
	// once all pending messages are delivered and acknowledged. This call does
	// not block.
	TriggerStopConsuming()

	// TriggerCloseNow triggers the shut down of this component but should not
	// block the calling goroutine.
	TriggerCloseNow()

	// WaitForClose is a blocking call to wait until the component has finished
	// shutting down and cleaning up resources.
	WaitForClose(ctx context.Context) error
}

Streamed is a common interface implemented by inputs and provides channel based streaming APIs.

func NewAsyncReader 

func NewAsyncReader(
	typeStr string,
	r Async,
	mgr component.Observability,
	opts ...func(a *AsyncReader),
) (Streamed, error)

NewAsyncReader creates a new AsyncReader input type.

func WrapWithPipelines 

func WrapWithPipelines(in Streamed, pipeConstructors ...iprocessor.PipelineConstructorFunc) (Streamed, error)

WrapWithPipelines wraps an input with a variadic number of pipelines.

type SubprocessConfig 

type SubprocessConfig struct {
	Name          string   `json:"name" yaml:"name"`
	Args          []string `json:"args" yaml:"args"`
	Codec         string   `json:"codec" yaml:"codec"`
	RestartOnExit bool     `json:"restart_on_exit" yaml:"restart_on_exit"`
	MaxBuffer     int      `json:"max_buffer" yaml:"max_buffer"`
}

SubprocessConfig contains configuration for the Subprocess input type.

func NewSubprocessConfig 

func NewSubprocessConfig() SubprocessConfig

NewSubprocessConfig creates a new SubprocessConfig with default values.

type WithPipeline 

type WithPipeline struct {
	// contains filtered or unexported fields
}

WithPipeline is a type that wraps both an input type and a pipeline type by routing the input through the pipeline, and implements the input.Type interface in order to act like an ordinary input.

func WrapWithPipeline 

func WrapWithPipeline(in Streamed, pipeConstructor iprocessor.PipelineConstructorFunc) (*WithPipeline, error)

WrapWithPipeline routes an input directly into a processing pipeline and returns a type that manages both and acts like an ordinary input.

func (*WithPipeline) Connected 

func (i *WithPipeline) Connected() bool

Connected returns a boolean indicating whether this input is currently connected to its target.

func (*WithPipeline) TransactionChan 

func (i *WithPipeline) TransactionChan() <-chan message.Transaction

TransactionChan returns the channel used for consuming transactions from this input.

func (*WithPipeline) TriggerCloseNow 

func (i *WithPipeline) TriggerCloseNow()

TriggerCloseNow triggers the shut down of this component but should not block the calling goroutine.

func (*WithPipeline) TriggerStopConsuming 

func (i *WithPipeline) TriggerStopConsuming()

TriggerStopConsuming instructs the input to start shutting down resources once all pending messages are delivered and acknowledged. This call does not block.

func (*WithPipeline) WaitForClose 

func (i *WithPipeline) WaitForClose(ctx context.Context) error

WaitForClose is a blocking call to wait until the component has finished shutting down and cleaning up resources.

Source Files

View all Source files

Directories

Path Synopsis
Package config contains reusable config definitions and parsers for inputs defined via the public/service package.
 Package config contains reusable config definitions and parsers for inputs defined via the public/service package.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.