indexer

README

Indexer

Service for indexing RapidPro/TextIt contacts into Elasticsearch.

Deploying

As it is a Go application, it compiles to a binary and that binary along with the config file is all you need to run it on your server. You can find bundles for each platform in the releases directory. You should only run a single instance for a deployment.

It can run in two modes:

1. the default mode, which simply queries the ElasticSearch database, finds the most recently modified contact, then on a schedule queries the contacts_contact table in the database for contacts to add or delete. You should run this as a long running service which constantly keeps ElasticSearch in sync with your contacts.

2. a rebuild mode, started with --rebuild. This builds a brand new index from nothing, querying all contacts on RapidPro. Once complete, this switches out the alias for the contact index with the newly build index. This can be run on a cron (in parallel with the mode above) to rebuild your index occasionally to get rid of bloat.

Configuration

The service uses a tiered configuration system, each option takes precendence over the ones above it:

1. The configuration file
2. Environment variables starting with INDEXER_
3. Command line parameters

We recommend running it with no changes to the configuration and no parameters, using only environment variables to configure it. You can use % rp-indexer --help to see a list of the environment variables and parameters and for more details on each option.

RapidPro

For use with RapidPro, you will want to configure these settings:

• INDEXER_DB: a URL connection string for your RapidPro database or read replica
• INDEXER_ELASTIC_URL: the URL for your ElasticSearch endpoint

Recommended settings for error reporting:

• INDEXER_SENTRY_DSN: The DSN to use when logging errors to Sentry

Reference

These are the configuration options that can be provided as parameters or environment variables. If using environment varibles, convert to uppercase, replace dashes with underscores and prefix the name with INDEXER_, e.g. -log-level becomes INDEXER_LOG_LEVEL.

  -cleanup
      whether to remove old indexes after a rebuild
  -db string
      the connection string for our database (default "postgres://localhost/rapidpro?sslmode=disable")
  -debug-conf
      print where config values are coming from
  -elastic-url string
      the url for our elastic search instance (default "http://localhost:9200")
  -help
      print usage information
  -index string
      the alias for our contact index (default "contacts")
  -librato-username
      the Librato username for metrics reporting
  -librato-token
      the Librato token for metrics reporting
  -log-level string
      the log level, one of error, warn, info, debug (default "info")
  -poll int
      the number of seconds to wait between checking for updated contacts (default 5)
  -rebuild
      whether to rebuild the index, swapping it when complete, then exiting (default false)
  -sentry-dsn string
      the sentry configuration to log errors to, if any

Development

Once you've checked out the code, you can build the service with:

go build github.com/nyaruka/rp-indexer/cmd/rp-indexer

This will create a new executable in $GOPATH/bin called rp-indexer.

To run the tests you need to create the test database:

$ createdb elastic_test

To run all of the tests:

go test ./... -p=1

Documentation

Index

• type Config
  • func NewDefaultConfig() *Config
• type Daemon
  • func NewDaemon(cfg *Config, db *sql.DB, ixs []indexers.Indexer, poll time.Duration) *Daemon
  • func (d *Daemon) Start()
  • func (d *Daemon) Stop()

Types

type Config

type Config struct {
	ElasticURL      string `help:"the url for our elastic search instance"`
	DB              string `help:"the connection string for our database"`
	Poll            int    `help:"the number of seconds to wait between checking for database updates"`
	Rebuild         bool   `help:"whether to rebuild the index, swapping it when complete, then exiting (default false)"`
	Cleanup         bool   `help:"whether to remove old indexes after a rebuild"`
	LogLevel        string `help:"the log level, one of error, warn, info, debug"`
	SentryDSN       string `help:"the sentry configuration to log errors to, if any"`
	LibratoUsername string `help:"the username that will be used to authenticate to Librato"`
	LibratoToken    string `help:"the token that will be used to authenticate to Librato"`
	InstanceName    string `help:"the unique name of this instance used for analytics"`

	ContactsIndex    string `help:"the alias to use for the contact index"`
	ContactsShards   int    `help:"the number of shards to use for the contacts index"`
	ContactsReplicas int    `help:"the number of replicas to use for the contacts index"`
}

func NewDefaultConfig

func NewDefaultConfig() *Config

type Daemon

type Daemon struct {
	// contains filtered or unexported fields
}

func NewDaemon

func NewDaemon(cfg *Config, db *sql.DB, ixs []indexers.Indexer, poll time.Duration) *Daemon

NewDaemon creates a new daemon to run the given indexers

func (*Daemon) Start

func (d *Daemon) Start()

Start starts this daemon

func (*Daemon) Stop

func (d *Daemon) Stop()

Stop stops this daemon