README
¶
backend
This repository contains the entire backend implementation for the cloud-lada project. It operates as a monorepo of go services.
Services
- Ingestor - Handles inbound sensor readings as HTTP requests and publishes them to an event bus
- Persistor - Handles sensor reading data from the event bus and writes them to a PostgreSQL compatible database
- Dumper - Creates dumps of daily sensor readings into a blob storage provider
Design
Streams of JSON-encoded objects are received via the ingestor, here is a sample format:
{"sensor": "speed", "value": 55.5, "timestamp": "2012-04-23T18:25:43.511Z"}
{"sensor": "speed", "value": 60.2, "timestamp": "2012-04-23T18:26:43.511Z"}
{"sensor": "speed", "value": 61.9, "timestamp": "2012-04-23T18:27:43.511Z"}
The reason for accepting JSON streams is that in the event that there is no internet connection for the on-board instruments (which is likely for some portions of the trip), they can be stored on disk. Once the connection is reestablished, these local files can be streamed to the ingestor. For the same reason, each reading contains a timestamp of when it was taken.
The ingestor will pass these readings individually to an event bus, where the persistor will handle them and store them in TimescaleDB appropriately for later querying. On a daily basis, the dumper will produce all readings produced in a day as a JSON-stream for later use by interested parties.
Deployment
This repository contains a Kustomize manifest in the manifests directory which can be used
to run one or more ingestor instances in a Kubernetes cluster. You can use make kustomize to produce a static manifest
that contains everything you need. You will need to override the ConfigMap and Secret resources for your specific
setup.
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
internal
|
|
|
dump
Package dump provides types that write a days worth of sensor data into a blob storage service.
|
Package dump provides types that write a days worth of sensor data into a blob storage service. |
|
ingest
Package ingest provides the HTTP handling methods for inbound sensor readings.
|
Package ingest provides the HTTP handling methods for inbound sensor readings. |
|
persist
Package persist provides logic for handling inbound readings and persisting them to a repository.
|
Package persist provides logic for handling inbound readings and persisting them to a repository. |
|
reading
Package reading provides persistence logic for sensor readings.
|
Package reading provides persistence logic for sensor readings. |
|
statistics
Package statistics provides all components required to serve statistical data via an HTTP API.
|
Package statistics provides all components required to serve statistical data via an HTTP API. |
|
pkg
|
|
|
blob
Package blob provides types for interacting with blob storage providers.
|
Package blob provides types for interacting with blob storage providers. |
|
event
Package event provides types used to read events from an event bus.
|
Package event provides types used to read events from an event bus. |
|
postgres
Package postgres provides functions for interacting with postgres-compatible databases and includes migration scripts.
|
Package postgres provides functions for interacting with postgres-compatible databases and includes migration scripts. |
|
testutil
Package testutil provides test helper functions.
|
Package testutil provides test helper functions. |