esbulk

README

esbulk

Fast parallel bulk loading utility for elasticsearch.

Installation

$ go get github.com/miku/esbulk/cmd/esbulk

For deb or rpm packages, see: https://github.com/miku/esbulk/releases

Usage

$ esbulk -h
Usage: esbulk [OPTIONS] JSON
  -cpuprofile="": write cpu profile to file
  -host="localhost": elasticsearch host
  -index="": index name
  -memprofile="": write heap profile to file
  -port=9200: elasticsearch port
  -size=1000: bulk batch size
  -type="default": elasticsearch doc type
  -v=false: prints current program version
  -verbose=false: output basic progress
  -w=4: number of workers to use
  -z=false: unzip gz'd file on the fly

To index a JSON file, that contains one document per line, just run:

$ esbulk -index example file.ldj

Where file.ldj is line delimited JSON, like:

{"name": "esbulk", "version": "0.2.4"}
{"name": "estab", "version": "0.1.3"}
...

By default esbulk will use as many parallel workers, as there are cores. To tweak the indexing process, adjust the -size and -w parameters.

You can index from gzipped files as well, using the -z flag:

$ esbulk -z -index example file.ldj.gz

Documentation

Index

• Constants
• func BulkIndex(docs []string, options Options) error
• func Worker(id string, options Options, lines chan string, wg *sync.WaitGroup)
• type Options

Constants

const Version = "0.3.0"

Application Version

Functions

func BulkIndex

func BulkIndex(docs []string, options Options) error

BulkIndex takes a set of documents as strings and indexes them into elasticsearch

func Worker

func Worker(id string, options Options, lines chan string, wg *sync.WaitGroup)

Worker will batch index documents that come in on the lines channel

Types

type Options

type Options struct {
	Host      string
	Port      int
	Index     string
	DocType   string
	BatchSize int
	Verbose   bool
}

Options represents bulk indexing options