amazon

README

Streams Driver for Amazon Messaging Services

The stream driver for Amazon messaging services offers both Writer and Reader implementations through services such as Amazon Simple Notification Service (SNS) and Amazon Simple Queue Service (SQS).

Both Writer implementations share a base writer instance which encapsulates a concurrent batching buffering mechanism to enable message batch writing capabilities with high throughput.

Amazon Simple Notification Service

For this driver, only Writer implementation is available for use. A Reader implementation is not on the roadmap as SNS does not have polling mechanisms; instead, the service makes synchronous request directly to subscribers.

This built-in mechanism erases the need for a Reader implementation.

Amazon Simple Queue Service

This driver offers both Reader and Writer implementations.

Topic-Queue Chaining Pattern

The topic queue chaining pattern is a messaging pattern that can be used to decouple microservices. In this pattern, a topic is used to publish messages to a group of subscribers. Each subscriber is subscribed to the topic, but the messages are delivered to the subscribers individually. This allows the subscribers to process the messages in parallel, which can improve performance.

The topic queue chaining pattern is often used in conjunction with a message queue. In this case, the topic is used to publish messages to the message queue. The message queue then delivers the messages to the subscribers. This allows the subscribers to process the messages in a reliable and scalable way.

The topic queue chaining pattern is a powerful tool that can be used to improve the performance and scalability of microservices. It is a good choice for systems that need to be able to process a large number of messages in a short amount of time.

In other words, the pattern refers to adding a queue, in this case an SQS queue, between the SNS topic and each of the subscriber services/workers. As messages are buffered persistently in an SQS queue, it prevents lost messages if a subscriber process run into problems for many hours or days.

This driver offers the posibility to implement the topic-queue chaining pattern. Systems should use the Amazon SNS Writer implementation for message publishing and the Amazon SQS Reader implementation for consuming messages on their respective queues at subscriber/worker level.

Queues as buffering load balancers

An SQS queue in front of each subscriber service also acts as a buffering load balancer. This pattern is also called Queue-Based Load Leveling (reference doc).

Since every message is delivered to one of potentially many consumer processes, you can scale out the subscriber services, and the message load is distributed over the available consumer processes.

As messages are buffered in the queue, they are preserved during a scaling event, such as when you must wait until an additional consumer process becomes operational.

Moreover, this pattern may also used directly (no SNS topics required) if you are only insterested in load balance your application tasks (e.g. video processing, data analytics/enrichment).

The Writer implementation could be used as underlying communication component for the asynchronous command bus if the system implements the Command and Query Responsibility Segretation (CQRS) pattern.

Lastly, these queue characteristics help flatten peak loads for your consumer processes, buffering messages until consumers are available. This allows you to process messages at a pace decoupled from the message source.

Documentation

Index

• Constants
• type Config
• type WriteFunc
• type Writer
  • func (w Writer) Write(ctx context.Context, msgBatch []streams.Message) error

Constants

const (
	HeaderMessageID   = "streams-message-id"   // The unique identifier of a message.
	HeaderStreamName  = "streams-stream-name"  // Name of the stream of a message.
	HeaderStreamKey   = "streams-stream-key"   // Key of the stream from a message.
	HeaderContentType = "streams-content-type" // Type of data of a content from a message.
	HeaderMessageTime = "streams-message-time" // Timestamp in Unix milliseconds when the message was published.
)

Types

type Config

type Config struct {
	AccountID string // AWS Account identifier streams belongs to.
	Region    string // AWS Region a where streams are located.
}

Config is the basic configuration schema for Amazon messaging services.

type WriteFunc

type WriteFunc func(ctx context.Context, stream string, msgBatch []streams.Message) error

WriteFunc Amazon service-agnostic message writing function. A Writer instance will call this function which is implemented by actual drivers (Amazon SNS/SQS).

type Writer

type Writer struct {
	WriteFunc WriteFunc
}

A Writer is a generic message writer for Amazon services. This component centralizes a batching group buffering algorithm which later schedules and executes message writing tasks concurrently to increase write throughput.

This type is NOT ready for usage as standalone component, concrete writers (e.g. sns.Writer, sqs.Writer) should be used instead.

func (Writer) Write

func (w Writer) Write(ctx context.Context, msgBatch []streams.Message) error