bilies-go

README

bilies-go

Bulk Insert Logs Into ElasticSearch - in Go

bilibes-go is designed to bulk-insert log entry into ElasticSearch.

It waits for JSON-encoded log entries on standard input, one per line, stores them in a disk-based queue, then send batchs using the bulk API of ElasticSearch.

Installation

go get github.com/Adirelle/bilies-go

Documentation

Documentation is available on Godoc.

License

See the LICENSE file.

Documentation

Overview

Batchs

Messages are gathered in batchs before being sent to the server. Batchs are sent when the number of messages reachs a defined value or when the delay since the last sent exceeds a defined value.

The following switches control the generation of bulk messages:

-i --index=STRING [default: logs]
	Define the prefix of the index name.

-t --type=STRING [default: log]
	Define the type of messages.

-n --batch-size=INT [default: 500]
	The maximum number of message to send in a single request.

-f --flush-delay=DURATION [default: 1s]
	The maximum delay between two requests.

Internal logging

Internal logging (i.e. messages generated by bilies-go) is done asynchronously. By default, messages of level NOTICE and higher are written do the standard error.

The following switches control logging:

-d --debug [default: false]
	Enable debug logging: write messages of level DEBUG and higher.

-v --verbose [default: false]
	Enable verbose logging: write messages of level INFO and higher.

--log-file=STRING [default: /dev/stderr]
	Write messages into the file PATH, instead of the standard error. CRITICAL messages are still written to standard error.

Sending HUP to bilies-go causes the logfile to be reopened.

PID file

--pid-file=STRING
	Write the PID into  the specified file.

Metrics

bilies-go collects several metrics. They can be written to the log by sending an USR1 signal to the process.

Backend pool

Several hosts can be passed on the command line to create a backend pool.

When a network error occurs while tryng to reach a backend, it is temporarily removed from the pool, using a delay which exponentially increases on consecutive errors. This delay starts at 500ms and is capped at 2 minutes.

Messages queueing

Incoming messages are enqueued into LevelDB database.

The following switch control queueing:

-q --queue-dir=STRING [default: $PWD/.queue]
	Sets the path to the directoy hosting the LevelDB.

Message input

bilies-go waits for JSON messages on its standard input. They should have the following format:

{"date:"YYYY.MM.DD", "id":"some unique idd", "log":{"foo":"bar"}}

The "id" is optional; if missing, a time-based UUID is generated. It is used to identify the document in ElasticSearch. Invalid messages are ignored and logged.

bilies-go expects UTF-8 messages (as JSON). In case the input is not a valid UTF-8 strings, a charset conversion is tried.

The following switch controls input reading:

-c --input-charset=STRING [default: ISO-8859-1]
	Alternative charset, in case the input is a not UTF-8 string.

Requests

bilies-go retries the requests indefinitively on network or 5xx errors. In case of 400 error, batchs are split in smaller parts and send independently to find the culprit.

The following switchs control requests:

-h --host=STRING [default: localhost]
	Hostname of a ElasticSearch servers. This switch can be used multiple times to add more severs.

-P --protocol=(http|https) [default: http]
	Protocol to of the ElasticSearch servers.

-p --port=INT [default: 9200]
	Port of the ElasticSearch servers.

-u --user=STRING [default: none]
	Username for basic authentification.

-w --passwd=STRING [default: none]
	Password for basic authentification.