🗃️ Indexer

Service for indexing RapidPro/TextIt contacts into Elasticsearch.
Deploying
As it is a Go application, it compiles to a binary and that binary along with the config file is all
you need to run it on your server. You can find bundles for each platform in the
releases directory. You should only run a single
instance for a deployment.
It can run in two modes:
-
the default mode, which simply queries the ElasticSearch database, finds the most recently
modified contact, then on a schedule queries the contacts_contact
table in the
database for contacts to add or delete. You should run this as a long running service which
constantly keeps ElasticSearch in sync with your contacts.
-
a rebuild mode, started with --rebuild
. This builds a brand new index from nothing, querying
all contacts on RapidPro. Once complete, this switches out the alias for the contact index
with the newly build index. This can be run on a cron (in parallel with the mode above) to rebuild
your index occasionally to get rid of bloat.
Configuration
The service uses a tiered configuration system, each option takes precendence over the ones above it:
- The configuration file
- Environment variables starting with
INDEXER_
- Command line parameters
We recommend running it with no changes to the configuration and no parameters, using only
environment variables to configure it. You can use % rp-indexer --help
to see a list of the
environment variables and parameters and for more details on each option.
INDEXER_DB
: a URL connection string for your RapidPro database or read replica
INDEXER_ELASTIC_URL
: the URL for your ElasticSearch endpoint
AWS services:
INDEXER_AWS_ACCESS_KEY_ID
: AWS access key id used to authenticate to AWS
INDEXER_AWS_SECRET_ACCESS_KEY
: AWS secret access key used to authenticate to AWS
INDEXER_AWS_REGION
: AWS region (ex: eu-west-1
)
Logging and error reporting:
INDEXER_DEPLOYMENT_ID
: used for metrics reporting
INDEXER_SENTRY_DSN
: DSN to use when logging errors to Sentry
INDEXER_LOG_LEVEL
: logging level to use (default is info
)
Development
Once you've checked out the code, you can build the service with:
go build github.com/nyaruka/rp-indexer/cmd/rp-indexer
This will create a new executable in $GOPATH/bin called rp-indexer
.
To run the tests you need to create the test database:
$ createdb elastic_test
To run all of the tests:
go test ./... -p=1