reader

README

Contents

1. Overview
   1.1. Purpose
   1.2. Definitions
   
2. Configuration
   
3. Deployment
   3.1. Prerequisites
   3.2. Bare
   3.3. Docker
   3.4. K8s
      3.4.1. Helm
   
4. Usage
   
5. Design
   5.1. Requirements
   5.2. Approach
   5.3. Limitations
   
6. Contributing
   6.1. Versioning
   6.2. Issue Reporting
   6.3. Building
   6.4. Testing
      6.4.1. Functional
      6.4.2. Performance
   6.5. Releasing
   

1. Overview

1.1. Purpose

For a given incoming message, reader service fetches all matching subscriptions and routes the message to all found subscriptions destinations. The service is stateless and doesn't use any storage except the work queue.

1.2. Definitions

reader works with messages and subscriptions. Messages are in the CloudEvents format. The following subscription attributes are used by reader:

1. Id
2. Destinations

Subscription destination describes the destination route where the matching message should be sent to.

2. Configuration

The service is configurable using the environment variables:

Variable	Example value	Description
API_PORT	8080	gRPC API port
API_CONSUMER_BACKOFF	1s	Time to sleep if a consumer doesn't accept new messages
API_CONSUMER_URI	consumer:8080	Consumer dependency service URI
API_MATCHES_URI	matches:8080	Matches dependency service URI
API_MATCHES_BATCH_SIZE	100	Matches query results size limit
LOG_LEVEL	-4	Logging level
QUEUE_BATCH_SIZE	100	Work queue processing batch size
QUEUE_FALLBACK_ENABLED	true	Dead letter queue usage flag
QUEUE_FALLBACK_SUFFIX	fallback	Dead letter queue name suffix, the resulting name will be <QUEUE_NAME>-<QUEUE_FALLBACK_SUFFIX>
QUEUE_LIMIT	1000	Work queue length limit
QUEUE_NAME	reader	Work queue name
QUEUE_BACKOFF_EMPTY	1s	Time to sleep if work queue is empty
QUEUE_BACKOFF_ERROR	1s	Time to sleep if failed to poll the work queue
QUEUE_URI	queue:8080	Work queue service URI

3. Deployment

3.1. Prerequisites

The following dependency services should be deployed and available:

• Matches
• A queue service implementation, e.g. Queue-Nats
• A consumer service implementation, e.g. Consumer-Log

3.2. Bare

Preconditions:

1. Build patterns executive using make build

Then run the command:

API_PORT=8082 \
API_MATCHES_URI=localhost:8080 \
./reader

3.3. Docker

make run

3.4. K8s

3.4.1. Helm

Create a helm package from the sources:

helm package helm/reader/

Install the helm chart:

helm install reader ./reader-<CHART_VERSION>.tgz

where

• <CHART_VERSION> is the helm chart version

4. Usage

The service provides a gRPC API for routing a message.

Example command:

grpcurl \
  -plaintext \
  -proto api/grpc/service.proto \
  -d @ \
  localhost:8080 \
  reader.Service/SubmitBatch

Payload:

{
   "msgs": [
      {
         "id": "3426d090-1b8a-4a09-ac9c-41f2de24d5ac",
         "type": "example.type",
         "source": "example/uri",
         "spec_version": "1.0",
         "attributes": {
            "subject": {
               "ce_string": "Obi-Wan Kenobi"
            },
            "time": {
               "ce_timestamp": "1985-04-12T23:20:50.52Z"
            }
         },
         "text_data": "I felt a great disturbance in the force"
      },
      {
         "id": "3426d090-1b8a-4a09-ac9c-41f2de24d5ad",
         "type": "example.type",
         "source": "example/uri",
         "spec_version": "1.0",
         "attributes": {
            "subject": {
               "ce_string": "Yoda"
            },
            "time": {
               "ce_timestamp": "1985-05-11T12:02:05.25Z"
            }
         },
         "text_data": "Try not. Do or do not. There is no try."
      },
      {
         "id": "3426d090-1b8a-4a09-ac9c-41f2de24d5ae",
         "type": "example.type",
         "source": "example/uri",
         "spec_version": "1.0",
         "attributes": {
            "subject": {
               "ce_string": "Qui-Gon Jinn"
            },
            "time": {
               "ce_timestamp": "1985-06-08T14:31:41.16Z"
            }
         },
         "text_data": "The ability to speak does not make you intelligent."
      }
   ]
}

5. Design

5.1. Requirements

#	Summary	Description
REQ-1	TODO	TODO

5.2. Approach

TODO

5.3. Limitations

#	Summary	Description
LIM-1	TODO	TODO

6. Contributing

6.1. Versioning

The service uses the semantic versioning. The single source of the version info is the git tag:

git describe --tags --abbrev=0

6.2. Issue Reporting

TODO

6.3. Building

make build

Generates the sources from proto files, compiles and creates the reader executable.

6.4. Testing

6.4.1. Functional

make test

6.4.2. Performance

TODO

6.5. Releasing

To release a new version (e.g. 1.2.3) it's enough to put a git tag:

git tag -v1.2.3
git push --tags

The corresponding CI job is started to build a docker image and push it with the specified tag (+latest).