subscriptions

README

Contents

1. Overview
   1.1. Purpose
   1.2. Definitions
      1.2.1. Route
         1.2.1.1. Destination
         1.2.1.2. Condition
            1.2.1.2.1. Group Condition
            1.2.1.2.2. Key Condition
            1.2.1.2.3. Kiwi Condition
            1.2.1.2.4. Kiwi Tree Condition
            1.2.1.2.5. Kiwi Bird Condition
      1.2.2. Subscription
   
2. Configuration
   
3. Deployment
   3.1. Prerequisites
   3.2. Bare
   3.3. Docker
   3.4. K8s
      3.4.1. Helm
   
4. Usage
   4.1. Create
   4.2. Read
   4.3. Delete
   4.4. Search
      4.4.1. By Condition
      4.4.2. By Metadata
   
5. Design
   5.1. Requirements
   5.2. Approach
      5.2.1. Data Schema
         5.2.1.1. Subscription
         5.2.1.2. Route
         5.2.1.3. Group Condition
         5.2.1.4. Kiwi Condition
      5.2.2 Results Pagination
   5.3. Limitations
   
6. Contributing
   6.1. Versioning
   6.2. Issue Reporting
   6.3. Building
   6.4. Testing
      6.4.1. Functional
      6.4.2. Performance
   6.5. Releasing
   

1. Overview

Subscriptions storage service.

1.1. Purpose

The main function of the service is to find all subscriptions by a condition. For example, a message metadata is matching a certain pattern. Then it's necessary to find all subscriptions those have this pattern in the corresponding condition.

1.2. Definitions

1.2.1. Route

A subscription route describes the conditions to match an incoming message and a destination where the matching message should be routed to.

1.2.1.1. Destination

Destination is a free-form string describing a target (usually topic or subject) for a message matching the corresponding subscription. An incoming message should be routed to the specified destination when and only when subscription condition matches.

1.2.1.2. Condition

A condition represents a message matching criteria. The common properties for any type of condition are:

• unique id
• negation flag (Not). When the negation flag set to true the condition is treated as a negation, otherwise it's a proposition.

1.2.1.2.1. Group Condition

A group condition represents a group of child conditions coupled with a certain logic: And, Or, Xor.

1.2.1.2.2. Key Condition

A key condition is an abstract condition specifying a key that should match the message metadata key.

1.2.1.2.3. Kiwi Condition

A kiwi (Key-Input WIldcard) condition is a key condition containing the metadata value pattern. Also, kiwi condition has a partial attribute to represent whether a value part is allowed to match the pattern.

1.2.1.2.4. Kiwi Tree Condition

A kiwi-tree specific condition implementation of a kiwi condition. It comes with additional limitations on the pattern syntax but allows resolving a condition by a key/value pair in a O(log(N)) time.

1.2.1.2.5. Kiwi Bird Condition

A specific kiwi condition implementation. It should come without any limitation on the pattern syntax but will resolve a condition in O(N) time. Not implemented yet.

1.2.2. Subscription

Subscriptions is an entity linking the condition with the routes. A subscription also has unique id generated on creation and human-readable metadata.

2. Configuration

The service is configurable using the environment variables:

Variable	Example value	Description
API_PORT	8080	gRPC API port
DB_URI	mongodb+srv://localhost/?retryWrites=true&w=majority	DB connection URI
DB_NAME	subscriptions	DB name to store the data
DB_TABLE_NAME	subscriptions	DB table name to store the data
API_KIWI_TREE_COMPLETE_URI	kiwi-tree-complete:8080	Complete kiwi-tree dependency service URI
API_KIWI_TREE_PARTIAL_URI	kiwi-tree-partial:8080	Partial kiwi-tree dependency service URI

3. Deployment

3.1. Prerequisites

A general note is that there should be a MongoDB cluster deployed to be used for storing the pattern data. It's possible to obtain a free cluster for testing purposes using Atlas.

3.2. Bare

Preconditions:

1. Build patterns executive using make build
2. Run the kiwi-tree dependency services (x2: complete/partial)

Then run the command:

API_PORT=8080 \
DB_URI=mongodb+srv://localhost/\?retryWrites=true\&w=majority \
DB_NAME=subscriptions \
DB_TABLE_NAME=subscriptions \
API_KIWI_TREE_COMPLETE_URI=http://localhost:8081 \
API_KIWI_TREE_PARTIAL_URI=http://localhost:8082 \
./subscriptions

3.3. Docker

TODO: run the kiwi-tree (x2) and subscriptions in the same network

alternatively, it's possible to build and run the new docker image in place using the command: (note that the command below requires all env vars to be set in the file env.txt)

make run

3.4. K8s

TODO

3.4.1. Helm

Create a helm package from the sources:

helm package helm/subscriptions/

Install the helm chart:

helm install subscriptions ./subscriptions-<CHART_VERSION>.tgz \
  --values helm/subscriptions/values-db-uri.yaml

where

• values-db-uri.yaml contains the value override for the DB URI
• <CHART_VERSION> is the helm chart version

4. Usage

The service provides basic gRPC interface to perform the operation on subscriptions.

4.1. Create

Example:

grpcurl \
  -plaintext \
  -proto api/grpc/service.proto \
  -d '{"metadata" :{"description": "my subscription"}, "route": { "destinations": ["dst0"], "condition": {"kiwiTreeCondition": {"not" :false, "key": "key0", "pattern": "pattern*", "partial": false}}}}' \
  localhost:8080 \
  subscriptions.Service/Create

Yet another example:

grpcurl \
  -plaintext \
  -proto api/grpc/service.proto \
  -d '{"metadata": {"name": "sub1", "description": "my subscription 1"}, "route": {"destinations": ["dst1"], "condition": {"groupCondition": {"not": false, "logic": 0, "group": [{"kiwiTreeCondition": {"not": false, "key": "key0", "pattern": "pattern?", "partial": false}}, {"kiwiTreeCondition": {"not": true, "key": "key1", "pattern": "pattern1", "partial": true}}]}}}}' \
  localhost:8080 \
  subscriptions.Service/Create

4.2. Read

Example:

grpcurl \
  -plaintext \
  -proto api/grpc/service.proto \
  -d '{"id": "3426d090-1b8a-4a09-ac9c-41f2de24d5ac"}' \
  localhost:8080 \
  subscriptions.Service/Read

4.3. Delete

Example:

grpcurl \
  -plaintext \
  -proto api/grpc/service.proto \
  -d '{"id": "f7102c87-3ce4-4bb0-8527-b4644f685b13"}' \
  localhost:8080 \
  subscriptions.Service/Delete

4.4. Search

4.4.1. By Condition

The search by condition purpose is to be used by a resolver to find the matching subscriptions.

Example:

grpcurl \
  -plaintext \
  -proto api/grpc/service.proto \
  -d '{"limit": 100, "kiwiConditionQuery": {"key": "key0", "pattern": "pattern*", "partial": false}}' \
  localhost:8080 \
  subscriptions.Service/SearchByCondition

4.4.2. By Metadata

The search by metadata purpose is to be used by a user to find own subscriptions.

Example:

grpcurl \
  -plaintext \
  -proto api/grpc/service.proto \
  -d '{"limit": 100, "metadata": {"description": "my subscription"}}' \
  localhost:8080 \
  subscriptions.Service/SearchByMetadata

5. Design

5.1. Requirements

#	Summary	Description
REQ-1	Basic matching	Resolve subscriptions matching the input value
REQ-2	Logic	Support subscription logics for the multiple key-value matches (and, or, not)
REQ-3	Partial matching	Support partial (value might be split to lexemes) value matching
REQ-4	Pagination	Support query results pagination

5.2. Approach

5.2.1. Data Schema

Subscriptions are stored in the single table under the denormalized schema.

Example data:

- id: "2f63ea52-a66c-4b93-92f1-12aa2831cd2c"
  metadata:
    name: subscription0
    description: Anything related to orders that are not in Helsinki
    user: "e7fae6df-f0e7-4a6b-b8ae-3802a7927f7e"  
  route:
    destinations:
      - /dev/null
    condition:
      base:
        id: "123e4567-e89b-12d3-a456-426614174000"
        not: false
      logic: "And"
      group:
        - id: "14cadd71-c662-4f1a-8b0f-3b17dfb107f5"
          base:
            not: false
          partial: true
          key: "subject"
          pattern: "orders"
        - id: "c00e1228-fd78-4761-8f59-fbbfa690b9a9"
          base:
            not: true
          partial: false
          key: "location"
          pattern: "Helsinki"

5.2.1.1. Subscription

A subscription is immutable by design, hence there's no update operation for this. If a user wants to change a subscription they need to delete it 1st and then create again.

Attribute	Type	Description
id	String	Subscription UUID (generated on creation)
metadata	Map<String, String>	Human readable subscription metadata
route	Route	Subscription routing data
routes	Array of String	Destination routes to use for the matching messages delivery
condition	Condition (currently may be Group or Kiwi)	Message matching criteria

5.2.1.2. Route

Attribute	Type	Description
destinations	Array of String	Destination routes to use for the matching messages delivery
condition	Condition (currently may be Group or Kiwi)	Message matching criteria

5.2.1.3. Group Condition

Attribute	Type	Description
id	String	Condition UUID (generated on creation)
not	Boolean	Defines whether the conditions should act as a negation or not
logic	Enum of And/Or/Xor	Defines the grouping logic for the child conditions
group	Array of child conditions	Set of conditions in the group

5.2.1.4. Kiwi Condition

Attribute	Type	Description
id	String	Condition UUID (generated on creation)
not	Boolean	Defines whether the conditions should act as a negation or not
key	String	Metadata key
pattern	String	Metadata value matching pattern
partial	Boolean	If true, then allowed match any lexeme in a tokenized metadata value. Otherwise, entire value should match.

5.2.2. Results Pagination

The limit and cursor search parameters are used to support the results' pagination.

5.3. Limitations

#	Summary	Description
LIM-1	Root condition negation is not allowed	A subscription should not have root negation condition. Otherwise the subscription never matches anything in practice.
LIM-2	Optional condition negation is not allowed	TODO: A negation condition in the group with "Or"/"Xor" group logic doesn't have any effect

6. Contributing

6.1. Versioning

The service uses the semantic versioning. The single source of the version info is the git tag:

git describe --tags --abbrev=0

6.2. Issue Reporting

TODO

6.3. Building

make build

Generates the sources from proto files, compiles and creates the subscriptions executable.

6.4. Testing

6.4.1. Functional

make test

6.4.2. Performance

TODO

6.5. Releasing

To release a new version (e.g. 1.2.3) it's enough to put a git tag:

git tag -v1.2.3
git push --tags

The corresponding CI job is started to build a docker image and push it with the specified tag (+latest).