expvastic

README

About

This is an early release and it is private. There will be a lot of changes soon but I'm planning to finalise and make it public as soon as I can. I hope you enjoy it!

Expvastic can read from an endpoint which provides expvar data and ship them to elasticsearch. Please refer to golang's expvar documentation for more information.

Here is a couple of screenshots:

Installing

You need golang 1.7 (I haven't tested it with older versions, but they should be fine) and glide installed. Simply do:

go get github.com/arsham/expvastic/...
cd $GOPATH/src/github.com/arsham/expvastic
glide install

You also need elasticsearch and kibana, here is a couple of docker images for you:

docker run -it --rm --name expvastic --ulimit nofile=98304:98304 -v "/path/to/somewhere/expvastic":/usr/share/elasticsearch/data elasticsearch
docker run -it --rm --name kibana -p 80:5601 --link expvastic:elasticsearch -p 5601:5601 kibana

Running

Here an example configuration, save it somewhere (let's call it expvastic.yml for now):

settings:
    debug_evel: info

readers:
    FirstApp: # service name
        type: expvar
        endpoint: localhost:1234
        routepath: /debug/vars
        interval: 500ms
        timeout: 3s
        log_level: debug
        backoff: 10
    SomeApplication:
        type: expvar
        endpoint: localhost:1235
        routepath: /debug/vars
        interval: 500ms
        timeout: 13s
        log_level: debug
        backoff: 10

recorders:
    main_elasticsearch:
        type: elasticsearch
        endpoint: 127.0.0.1:9200
        index_name: expvastic
        type_name: my_app1
        timeout: 8s
        backoff: 10
    the_other_elasticsearch:
        type: elasticsearch
        endpoint: 127.0.0.1:9200
        index_name: expvastic
        type_name: SomeApplication
        timeout: 18s
        backoff: 10

routes:
    route1:
        readers:
            - FirstApp
        recorders:
            - main_elasticsearch
    route2:
        readers:
            - FirstApp
            - SomeApplication
        recorders:
            - main_elasticsearch
    route3:
        readers:
            - SomeApplication
        recorders:
            - main_elasticsearch
            - the_other_elasticsearch

Then run the application:

expvasyml -c expvastic.yml

Please note that the name of the app will be changed to expvastic.

Running Tests

go test $(glide nv)

Test Coverage

Use this gist. Then run the function, that's all!

TODO

• Decide how to show GC information correctly
• When reader/recorder are not available, don't check right away
• Create UUID for messages in order to log them
• Read from multiple sources
• Record expvastic's own metrics
• Use dates on index names
• Read from other providers; python, JMX etc.
• Read from log files
• Benchmarks
• Create a docker image
• Make a compose file
• Gracefully shutdown
• Share kibana setups
• [=] Read from yaml/toml/json configuration files
• Create different timeouts for each reader/recorder
• Read from etcd/consul for configurations

LICENSE

Use of this source code is governed by the Apache 2.0 license. License that can be found in the LICENSE file.

Thanks!

Documentation

Overview

Package expvastic can read from an endpoint which provides expvar data and ship them to elasticsearch. Please refer to golang's [expvar documentation](https://golang.org/pkg/expvar/) for more information. This is an early release and it is private. There will be a lot of changes soon but I'm planning to finalise and make it public as soon as I can. I hope you enjoy it!

Here is a couple of screenshots: http://i.imgur.com/6kB88g4.png and http://i.imgur.com/0ROSWsM.png

You need golang 1.7 (I haven't tested it with older versions, but they should be fine) and [glide](https://github.com/Masterminds/glide) installed. Simply do:

go get github.com/arsham/expvastic/...
cd $GOPATH/src/github.com/arsham/expvastic
glide install

You also need elasticsearch and kibana, here is a couple of docker images for you:

docker run -it --rm --name expvastic --ulimit nofile=98304:98304 -v "/path/to/somewhere/expvastic":/usr/share/elasticsearch/data elasticsearch
docker run -it --rm --name kibana -p 80:5601 --link expvastic:elasticsearch -p 5601:5601 kibana

At the heart of this package, there is Engine. It acts like a glue between a Reader and a Recorder. Messages are transfered in a package called DataContainer, which is a list of DataType objects.

Here an example configuration, save it somewhere (let's call it expvastic.yml for now):

settings:
    debug_evel: info

readers:
    FirstApp: # service name
        type: expvar
        endpoint: localhost:1234
        routepath: /debug/vars
        interval: 500ms
        timeout: 3s
        log_level: debug
        backoff: 10
    SomeApplication:
        type: expvar
        endpoint: localhost:1235
        routepath: /debug/vars
        interval: 500ms
        timeout: 13s
        log_level: debug
        backoff: 10

recorders:
    main_elasticsearch:
        type: elasticsearch
        endpoint: 127.0.0.1:9200
        index_name: expvastic
        type_name: my_app1
        timeout: 8s
        backoff: 10
    the_other_elasticsearch:
        type: elasticsearch
        endpoint: 127.0.0.1:9200
        index_name: expvastic
        type_name: SomeApplication
        timeout: 18s
        backoff: 10

routes:
    route1:
        readers:
            - FirstApp
        recorders:
            - main_elasticsearch
    route2:
        readers:
            - FirstApp
            - SomeApplication
        recorders:
            - main_elasticsearch
    route3:
        readers:
            - SomeApplication
        recorders:
            - main_elasticsearch
            - the_other_elasticsearch

Then run the application:

expvasyml -c expvastic.yml

Please note that the name of the app will be changed to expvastic.

For running tests, do the following:

go test $(glide nv)

Use of this source code is governed by the Apache 2.0 license. License that can be found in the LICENSE file.

Index

• func StartEngines(ctx context.Context, log logrus.FieldLogger, confMap *config.ConfMap) (chan struct{}, error)
• type Engine
  • func NewEngine(ctx context.Context, log logrus.FieldLogger, reader config.ReaderConf, ...) (*Engine, error)
  • func (c *Engine) Name() string
  • func (c *Engine) Start() chan struct{}
  • func (c *Engine) Stop()

Functions

func StartEngines

func StartEngines(ctx context.Context, log logrus.FieldLogger, confMap *config.ConfMap) (chan struct{}, error)

StartEngines creates some Engines and returns a channel that closes it when it's done its work. For each routes, we need one engine that has one reader and writes to a recorder. This is because:

1 - Readers should not intercept each other by engaging the recorders
2 - When a reader goes out of scope, we can safely stop the recorder.
3 - When a recorder goes out of scope, the reader should be able to stream its data to the rest of recorders.

Types

type Engine

type Engine struct {
	// contains filtered or unexported fields
}

Engine represents an engine that receives information from readers and ships them to recorders. The Engine is allowed to change the index and type names at will. When the context times out or canceled, the engine will close the the job channels by calling the Stop method.

func NewEngine

func NewEngine(ctx context.Context, log logrus.FieldLogger, reader config.ReaderConf, recorder config.RecorderConf) (*Engine, error)

NewEngine copies its configurations from c.

func (*Engine) Name

func (c *Engine) Name() string

Name shows the name identifier for this engine

func (*Engine) Start

func (c *Engine) Start() chan struct{}

Start begins pulling the data from TargetReader and chips them to DataRecorder. When the context cancels or timesout, the engine closes all job channels, causing the readers and recorders to stop.

func (*Engine) Stop

func (c *Engine) Stop()

Stop closes the job channels