indexer

README

bleve-bench

Detailed background on this code can be found on this blog post.

bleve-bench is a program to test the impact of batch size and sharding on indexing performance of the bleve library.

Building and Running

Building bleve-bench requires Go 1.3 or later. gvm is a great tool for managing your version of Go.

Download and run bleve-bench like so (tested on 64-bit Kubuntu 14.04):

mkdir bleve-bench # Or any directory of your choice.
cd bleve-bench/
export GOPATH=$PWD
go get -v github.com/otoolep/bleve-bench
go install github.com/otoolep/bleve-bench/cmd/bench/.
$GOPATH/bin/bench -h

Executing the last command will show the various options. An example run is shown below.

$ $GOPATH/bin/bench -docs testdata.txt -maxprocs 8 -shards 50 -batchSize 100
Opening docs file testdata.txt
100000 documents read for indexing.
Commencing indexing. GOMAXPROCS: 8, batch size: 100, shards: 50.
Indexing operation took 3.479690221s
100000 documents indexed.
Indexing rate: 28738 docs/sec.

Each line in the test data file is read and indexed as a distinct document. Any previously indexed data is deleted before indexing begins.

Documentation

Index

• type Indexer
  • func New(path string, nShards, batchSz int) *Indexer
  • func (i *Indexer) Count() (uint64, error)
  • func (i *Indexer) Index(docs [][]byte) error
  • func (i *Indexer) Open() error

Types

type Indexer

type Indexer struct {
	// contains filtered or unexported fields
}

Indexer represents the indexing engine.

func New

func New(path string, nShards, batchSz int) *Indexer

New returns a new indexer.

func (*Indexer) Count

func (i *Indexer) Count() (uint64, error)

Count returns the total number of documents indexed.

func (*Indexer) Index

func (i *Indexer) Index(docs [][]byte) error

Index indexes the given docs, dividing the docs evenly across the shards. Blocks until all documents have been indexed.

func (*Indexer) Open

func (i *Indexer) Open() error

Open opens the indexer, preparing it for indexing.