dp-filter-api

README

dp-filter-api

Getting started

mongo

• Run brew install mongodb
• Run brew services start mongodb

Kafka

• Run brew install zookeeper
• Download kafka version 0.10.2.1
• Run ./kafka-server-start.sh ../config/server.properties

Scripts for updating and debugging Kafka can be found here(dp-data-tools)

Configuration

• Run brew install neo4j
• Disable authentication in the config
• Run brew services start neo4j

Environment variable	Default	Description
BIND_ADDR	:22100	The host and port to bind to
HOST	http://localhost:22100	The host name used to build URLs
KAFKA_ADDR	localhost:9092	The kafka broker addresses (can be comma separated)
KAFKA_MAX_BYTES	2000000	The maximum permitted size of a message. Should be set equal to or smaller than the broker's message.max.bytes
KAFKA_SEC_PROTO	unset	if set to TLS, kafka connections will use TLS [1]
KAFKA_SEC_CLIENT_KEY	unset	PEM for the client key [1]
KAFKA_SEC_CLIENT_CERT	unset	PEM for the client certificate [1]
KAFKA_SEC_CA_CERTS	unset	CA cert chain for the server cert [1]
KAFKA_SEC_SKIP_VERIFY	false	ignores server certificate issues if true [1]
FILTER_JOB_SUBMITTED_TOPIC	filter-job-submitted	The kafka topic to write messages to
MONGODB_BIND_ADDR	localhost:27017	The MongoDB bind address
MONGODB_USERNAME		The MongoDB Username
MONGODB_PASSWORD		The MongoDB Password
MONGODB_DATABASE	filters	The MongoDB database
MONGODB_COLLECTIONS	FiltersCollection:filters, OutputsCollection:filterOutputs	The MongoDB collections
MONGODB_REPLICA_SET		The name of the MongoDB replica set
MONGODB_ENABLE_READ_CONCERN	false	Switch to use (or not) majority read concern
MONGODB_ENABLE_WRITE_CONCERN	true	Switch to use (or not) majority write concern
MONGODB_CONNECT_TIMEOUT	5s	The timeout when connecting to MongoDB (time.Duration format)
MONGODB_QUERY_TIMEOUT	15s	The timeout for querying MongoDB (time.Duration format)
MONGODB_IS_SSL	false	Switch to use (or not) TLS when connecting to mongodb
SHUTDOWN_TIMEOUT	5s	The graceful shutdown timeout (time.Duration format)
DATASET_API_URL	http://localhost:22000	The URL of the Dataset API
HEALTHCHECK_INTERVAL	30s	Time between self-healthchecks (time.Duration format)
HEALTHCHECK_CRITICAL_TIMEOUT	90s	The time taken for the health changes from warning state to critical due to subsystem check failures
SERVICE_AUTH_TOKEN	FD0108EA-825D-411C-9B1D-41EF7727F465	The token used to identify this service when authenticating
ZEBEDEE_URL	"http://localhost:8082"	Zebedee URL
ENABLE_PRIVATE_ENDPOINTS	false	true if private endpoints should be enabled
DOWNLOAD_SERVICE_URL	http://localhost:23600	The URL of the download service
DOWNLOAD_SERVICE_SECRET_KEY	QB0108EZ-825D-412C-9B1D-41EF7747F462	The service token for the download service

Notes:

1. For more info, see the kafka TLS examples documentation

Healthchecking

Currently checked each $HEALTHCHECK_INTERVAL and reported on endpoint /healthcheck:

• Neo4j
• MongoDB
• Dataset API

Acceptance tests

To run the acceptance tests for this project, use the following commands

• Start Kafka and Mongodb
• Run the dp-filter-api using make acceptance
• Run the tests in dp-api-test

Contributing

See CONTRIBUTING for details.

License

Copyright © 2016-2022, Office for National Statistics (https://www.ons.gov.uk)

Released under MIT license, see LICENSE for details.