kollect

README

kollect

Kollect is a Kubernetes Informer implementation that publishes changes in your cluster's resources as events on a configurable event bus.

It uses a dynamic informer that watches over all resource types available to the Service Account provided to the deployment. Message ordering is applied in configurations where it is supported.

Configuration

Kollect's configuration is fairly simple via the use of environment variables:

Variable	Description	Default
NAMESPACE	The namespace to monitor resources in	all
EVENT_WRITER_URL	The URL to configure the event bus, see Event Bus Configuration for more details
KUBECONFIG	The location of the kubeconfig file, uses in-cluster configuration if not provided
CLUSTER_ID	A unique identifier for the cluster the agent is running in.

Event Bus Configuration

Kollect supports the following event buses:

• GCP PubSub
• AWS SQS
• AWS SNS
• Azure Service Bus
• RabbitMQ
• NATS
• Apache Kafka

To select an event bus, the EVENT_WRITER_URL must be set with a URL where the scheme names the event bus driver.

Provider	Sample URL	Authentication
GCP PubSub	gcppubsub://projects/<project>/topics/<topic>	Authenticate using workload identity or by setting the GOOGLE_APPLICATION_CREDENTIALS environment variable
AWS SQS	awssqs://<region_endpoint>/<account_number>/<queue>	Authenticate via the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
AWS SNS	awssns:///arn:aws:sns:<region>:<account_number>:<topic>	Authenticate via the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
Azure Service Bus	azuresb://<topic>	Set the SERVICEBUS_CONNECTION_STRING environment variable to the connection string provided in the Azure console
RabbitMQ	rabbit://<exchange>	Set the RABBIT_SERVER_URL environment variable to the RabbitMQ server host.
NATS	nats://<subject>	Set the NATS_SERVER_URL environment variable to the NATS server host.
Apache Kafka	kafka://<topic>	Set the KAFKA_BROKERS to a comma-separated string of your broker hosts.

Kollect uses gocloud for its event-bus abstration. For more details on how the event bus URLs work, see their documentation

Message Structure

Kollect publishes events using protocol buffers which allows the generation of event handling code for many languages. The definitions are kept within the proto directory. Each event is wrapped within an envelope like so:

{
  "id": "ac3153e2-e9e2-4330-9bbf-52a79fb11214",
  "payload": {},
  "timestamp": "2012-04-23T18:25:43.511Z",
  "applies_at": "2012-04-23T18:25:43.511Z"
}

Below is a description of each field in the envelope:

Field	Description
id	A unique identifier for the event
payload	The message content as an Any type
timestamp	The time the message was published
applies_at	The time in which the process that triggered the message started, this may not always be the same as timestamp

Message Types

Kollect publishes message on resource creation, deletion or updates, each of those has a separate protobuf definition. Below are examples for their structure:

ResourceCreated

{
  "uid": "ac3153e2-e9e2-4330-9bbf-52a79fb11214",
  "cluster_id": "ac3153e2-e9e2-4330-9bbf-52a79fb11214",
  "resource": {} // The JSON encoded resource as an array of bytes.
}

ResourceUpdated

{
  "uid": "ac3153e2-e9e2-4330-9bbf-52a79fb11214",
  "cluster_id": "ac3153e2-e9e2-4330-9bbf-52a79fb11214",
  "old": {}, // The resource's previous state as a JSON encoded array of bytes.
  "new": {}  // The resource's new state as a JSON encoded array of bytes.
}

ResourceDeleted

{
  "uid": "ac3153e2-e9e2-4330-9bbf-52a79fb11214",
  "cluster_id": "ac3153e2-e9e2-4330-9bbf-52a79fb11214",
}

Deployment

This repository provides a Kustomize base for deploying the agent within a cluster in the manifests directory. It is recommended to deploy the agent within its own namespace.

Contributing & Testing

If you are working locally on this project, you can use the docker-compose file located at the root of the repository to spin up a local version of the agent that authenticates via your local kubeconfig file.

The agent exposes prometheus metrics and pprof endpoints on port 8081. You can also enable tracing (provided by OpenTelemetry) via the TELEMETRY_URL environment variable.