eskeeper

README

eskeeper

Tool managing Elasticsearch Index

eskeeper synchronizes index and alias with configuration files while ensuring idempotency. It still only supports WRITE. DELETE is not yet supported because the operation of deleting persistent data is dangerous and needs to be implemented carefully.

💪 Currently supports

mode

• CLI mode
• Index status(open/close only)
• Reindex
• Agent mode

sync

• index

• create
• status (open or close)
• reindex (only basic parameter)
• status(open/close only)
• lifecycke
• update mapping
• delete

• alias

• create
• update
• delete

🍀 Quick Start

eskeeper recieves yaml format data form stdin.

$ go get -u github.com/po3rin/eskeeper/cmd/eskeeper
$ eskeeper < es.yaml

es.yaml is indices & aliases config file.

index:
  - name: test-v1 # index name
    mapping: testdata/test.json # index setting & mapping (json)

  - name: test-v2
    mapping: testdata/test.json

  - name: close-v1
    mapping: testdata/test.json
    status: close

  # reindex test-v1 -> reindex-v1	
  - name: reindex-v1
    mapping: testdata/test.json
    reindex:
        source: test-v1 
        slices: 3 # default=1
        waitForCompletion: true

        # 'on' field supports 2 hooks.
        # 'firstCreated': only when index is created for the first time.
        # 'always': always exec reindex.
        on: firstCreated


alias:
  - name: alias1
    index:
      - test-v1

  # multi indicies
  - name: alias2
    index:
      - test-v1
      - test-v2

results

curl localhost:9200/_cat/indices
yellow open test-v1 ... 1 1 0 0 208b 208b
yellow open test-v2 ... 1 1 0 0 208b 208b
yellow close close-v1 xxxxxxxxxxxx 1 1
yellow open reindex-v1 ... 1 1 0 0 208b 208b

curl localhost:9200/_cat/aliases
alias2 test-v2 - - - -
alias1 test-v1 - - - -
alias2 test-v1 - - - -

📐 Usage

eskeeper supports flag & environment value.

# use flags
eskeeper -u user -p pass -e=http://localhost:9200,http://localhost9300 < testdata/es.yaml

# use env
ESKEEPER_ES_USER=user ESKEEPER_ES_PASS=pass ESKEEPER_ES_URLS=http://localhost:9200 eskeeper < testdata/es.yaml

🔎 Internal 4 Stages

eskeeper process is divided into four stages. verbose option lets you know eskeeper details.

$ eskeeper < es.yaml
loading config ...

=== validation stage ===
[pass] index: test-v1
[pass] index: test-v2
[pass] index: close-v1
[pass] alias: alias1
[pass] alias: alias2

=== pre-check stage ===
[pass] index: test-v1
[pass] index: test-v2
[pass] index: close-v1
[pass] alias: alias1
[pass] alias: alias2

=== sync stage ===
[synced] index: test-v1
[synced] index: test-v2
[synced] index: close-v1
[synced] alias: alias1
[synced] alias: alias2

=== post-check stage ===
[pass] index: test-v1
[pass] index: test-v2
[pass] index: close-v1
[pass] alias: alias1
[pass] alias: alias2

succeeded

validation stage

• Validates config yaml format

pre-check stage

• Check if mapping file is valid format
• Check if there is an index for alias

sync stage

• Sync indices and aliases with config

The order of synchronization is as follows.

create index
↓
open index
↓
update alias
↓
close index

Index close operation should be done after switching the alias. Because there can be downtime before switching aliases.

post-check stage

• Check if indices & aliases has been created

🚩 Contributing

Did you find something technically wrong, something to fix, or something? Please give me Issue or Pull Request !!

Test

eskeeper's test uses github.com/ory/dockertest. So you need docker to test.

Documentation

Index

• type Eskeeper
  • func New(urls []string, opts ...NewOption) (*Eskeeper, error)
  • func (e *Eskeeper) Sync(ctx context.Context, reader io.Reader) error
  • func (e *Eskeeper) Validate(ctx context.Context, reader io.Reader) error
• type NewOption
  • func Pass(pass string) NewOption
  • func SkipPreCheck(v bool) NewOption
  • func UserName(name string) NewOption
  • func Verbose(v bool) NewOption

Types

type Eskeeper

type Eskeeper struct {
	// contains filtered or unexported fields
}

Eskeeper manages indices & aliases.

func New

func New(urls []string, opts ...NewOption) (*Eskeeper, error)

New inits Eskeeper.

func (*Eskeeper) Sync

func (e *Eskeeper) Sync(ctx context.Context, reader io.Reader) error

Sync synchronizes config & Elasticsearch State.

func (*Eskeeper) Validate

func (e *Eskeeper) Validate(ctx context.Context, reader io.Reader) error

Validate validates cofig.

type NewOption

type NewOption func(*Eskeeper)

NewOption is optional func for eskeeper.New

func Pass

func Pass(pass string) NewOption

Pass is optional func for Elasticsearch password.

func SkipPreCheck

func SkipPreCheck(v bool) NewOption

SkipPreCheck is optional func for skipping pre-check-stage..

func UserName

func UserName(name string) NewOption

UserName is optional func for Elasticsearch user name.

func Verbose

func Verbose(v bool) NewOption

Verbose is optional func for verbose option.