expvastic

package module
v0.1.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 21, 2016 License: Apache-2.0 Imports: 11 Imported by: 0

README

About

This is an early release and it is under heavy development. There will be a lot of changes soon but I'm planning to finalise the API as soon as I can. I hope you enjoy it!

Expvastic can read from an endpoint which provides expvar data and ship them to elasticsearch. Please refer to golang's expvar documentation for more information.

Here is a couple of screenshots:

Colored Colored

Installing

You need golang 1.7 (I haven't tested it with older versions, but they should be fine) and glide installed. Simply do:

go get github.com/arsham/expvastic/...
cd $GOPATH/src/github.com/arsham/expvastic
glide install

You also need elasticsearch and kibana, here is a couple of docker images for you:

docker run -d --restart always --name expvastic -p 9200:9200 --ulimit nofile=98304:98304 -v "/path/to/somewhere/expvastic":/usr/share/elasticsearch/data elasticsearch
docker run -d --restart always --name kibana -p 80:5601 --link expvastic:elasticsearch -p 5601:5601 kibana

Usage

With Flags

With this method you can only have one reader and ship to one recorder. Consider the next section for more flexible setup. The defaults are sensible to use, you only need to point the app to two endpoints, and it does the rest for you:

expvastic -reader="localhost:1234/debug/vars" -recorder="localhost:9200"

For more flags run:

expvastic -h
With Configuration File

Here an example configuration, save it somewhere (let's call it expvastic.yml for now):

settings:
    debug_evel: info

readers:
    FirstApp: # service name
        type: expvar
        endpoint: localhost:1234
        routepath: /debug/vars
        interval: 500ms
        timeout: 3s
        log_level: debug
        backoff: 10
    SomeApplication:
        type: expvar
        endpoint: localhost:1235
        routepath: /debug/vars
        interval: 500ms
        timeout: 13s
        log_level: debug
        backoff: 10

recorders:
    main_elasticsearch:
        type: elasticsearch
        endpoint: 127.0.0.1:9200
        index_name: expvastic
        type_name: my_app1
        timeout: 8s
        backoff: 10
    the_other_elasticsearch:
        type: elasticsearch
        endpoint: 127.0.0.1:9200
        index_name: expvastic
        type_name: SomeApplication
        timeout: 18s
        backoff: 10

routes:
    route1:
        readers:
            - FirstApp
        recorders:
            - main_elasticsearch
    route2:
        readers:
            - FirstApp
            - SomeApplication
        recorders:
            - main_elasticsearch
    route3:
        readers:
            - SomeApplication
        recorders:
            - main_elasticsearch
            - the_other_elasticsearch

Then run the application:

expvasyml -c expvastic.yml

You can mix and match the routes, but the engine will choose the best setup to achive your goal without duplicating the results. For instance assume you set the routes like this:

 readers:
     app_0:
     app_1:
     app_2:
 recorders:
     elastic_0:
     elastic_1:
     elastic_2:
     elastic_3:
 routes:
     route1:
         readers:
             - app_0
             - app_2
         recorders:
             - elastic_1
     route2:
         readers:
             - app_0
         recorders:
             - elastic_1
             - elastic_2
             - elastic_3
     route2:
         readers:
             - app_1
             - app_2
         recorders:
             - elastic_1
             - elastic_0

Expvastic creates three engines like so:

    Data from app_0 will be shipped to: elastic_1, elastic_2 and elastic_3
    Data from app_1 will be shipped to: elastic_1 and, elastic_0
    Data from app_2 will be shipped to: elastic_1 and, elastic_0

Running Tests

go test $(glide nv)
Test Coverage

Use this gist. Then run the function, that's all!

TODO

  • Decide how to show GC information correctly
  • When reader/recorder are not available, don't check right away
  • Create UUID for messages in order to log them
  • Read from multiple sources
  • Record expvastic's own metrics
  • Use dates on index names
  • Read from other providers; python, JMX etc.
  • Read from log files
  • Benchmarks
  • Create a docker image
  • Make a compose file
  • Gracefully shutdown
  • Share kibana setups
  • [=] Read from yaml/toml/json configuration files
  • Create different timeouts for each reader/recorder
  • Read from etcd/consul for configurations

LICENSE

Use of this source code is governed by the Apache 2.0 license. License that can be found in the LICENSE file.

Thanks!

Documentation

Overview

Package expvastic can read from an endpoint which provides expvar data and ship them to elasticsearch. Please refer to golang's [expvar documentation](https://golang.org/pkg/expvar/) for more information. This is an early release and it is private. There will be a lot of changes soon but I'm planning to finalise and make it public as soon as I can. I hope you enjoy it!

Here is a couple of screenshots: http://i.imgur.com/6kB88g4.png and http://i.imgur.com/0ROSWsM.png

You need golang 1.7 (I haven't tested it with older versions, but they should be fine) and [glide](https://github.com/Masterminds/glide) installed. Simply do: 

go get github.com/arsham/expvastic/...
cd $GOPATH/src/github.com/arsham/expvastic
glide install

You also need elasticsearch and kibana, here is a couple of docker images for you: 

docker run -it --rm --name expvastic --ulimit nofile=98304:98304 -v "/path/to/somewhere/expvastic":/usr/share/elasticsearch/data elasticsearch
docker run -it --rm --name kibana -p 80:5601 --link expvastic:elasticsearch -p 5601:5601 kibana

At the heart of this package, there is Engine. It acts like a glue between a Reader and a Recorder. Messages are transfered in a package called DataContainer, which is a list of DataType objects.

Here an example configuration, save it somewhere (let's call it expvastic.yml for now): 

settings:
    debug_evel: info

readers:
    FirstApp: # service name
        type: expvar
        endpoint: localhost:1234
        routepath: /debug/vars
        interval: 500ms
        timeout: 3s
        log_level: debug
        backoff: 10
    SomeApplication:
        type: expvar
        endpoint: localhost:1235
        routepath: /debug/vars
        interval: 500ms
        timeout: 13s
        log_level: debug
        backoff: 10

recorders:
    main_elasticsearch:
        type: elasticsearch
        endpoint: 127.0.0.1:9200
        index_name: expvastic
        type_name: my_app1
        timeout: 8s
        backoff: 10
    the_other_elasticsearch:
        type: elasticsearch
        endpoint: 127.0.0.1:9200
        index_name: expvastic
        type_name: SomeApplication
        timeout: 18s
        backoff: 10

routes:
    route1:
        readers:
            - FirstApp
        recorders:
            - main_elasticsearch
    route2:
        readers:
            - FirstApp
            - SomeApplication
        recorders:
            - main_elasticsearch
    route3:
        readers:
            - SomeApplication
        recorders:
            - main_elasticsearch
            - the_other_elasticsearch

Then run the application: 

expvasyml -c expvastic.yml

Please note that the name of the app will be changed to expvastic.

For running tests, do the following: 

go test $(glide nv)

Use of this source code is governed by the Apache 2.0 license. License that can be found in the LICENSE file.

Index

Examples

Constants

This section is empty.

Variables

View Source 
var (
	// ErrEmptyRecName is for when the recorder's name is an empty string.
	ErrEmptyRecName = fmt.Errorf("recorder name empty")

	// ErrDupRecName is for when there are two recorders with the same name.
	ErrDupRecName = fmt.Errorf("recorder name empty")
)

Functions

func StartEngines added in v0.1.0 

func StartEngines(ctx context.Context, log logrus.FieldLogger, confMap *config.ConfMap) (chan struct{}, error)

StartEngines creates some Engines and returns a channel that closes it when it's done its work. For each routes, we need one engine that has one reader and writes to multiple recorders. This is because: 

1 - Readers should not intercept each other by engaging the recorders.
2 - When a reader goes out of scope, we can safely stop the recorders.

When a recorder goes out of scope, the Engine stops sending to that recorder.

Types

type Engine added in v0.0.4 

type Engine struct {
	// contains filtered or unexported fields
}

Engine represents an engine that receives information from readers and ships them to recorders. The Engine is allowed to change the index and type names at will. When the context times out or canceled, the engine will close the the job channels by calling the Stop method. Note that we could create a channel and distribute the recorders payload, but we didn't because there is no way to find out which recorder errors right after the payload has been sent. IMPORTANT: the readers should not close their streams, I am closing them here.

Example (SendJob) 
package main

import (
	"bytes"
	"context"
	"fmt"
	"io"
	"net/http"
	"net/http/httptest"
	"time"

	"github.com/arsham/expvastic"
	"github.com/arsham/expvastic/lib"
	"github.com/arsham/expvastic/reader"
	"github.com/arsham/expvastic/recorder"
)

func main() {
	var res *reader.ReadJobResult
	log := lib.DiscardLogger()
	ctx, cancel := context.WithCancel(context.Background())
	desire := `{"the key": "is the value!"}`

	redTs := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		io.WriteString(w, desire)
	}))
	defer redTs.Close()

	recTs := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		fmt.Println("Job was recorded")
	}))
	defer recTs.Close()

	ctxReader := reader.NewCtxReader(redTs.URL)
	red, _ := reader.NewSimpleReader(log, ctxReader, "reader_example", 10*time.Millisecond, 10*time.Millisecond)
	rec, _ := recorder.NewSimpleRecorder(ctx, log, "reader_example", recTs.URL, "intexName", "typeName", 10*time.Millisecond, 10*time.Millisecond)
	redDone := red.Start(ctx)
	recDone := rec.Start(ctx)

	cl, err := expvastic.NewWithReadRecorder(ctx, log, red, rec)
	fmt.Println("Engine creation success:", err == nil)
	clDone := cl.Start()

	select {
	case red.JobChan() <- ctx:
		fmt.Println("Just sent a job request")
	case <-time.After(time.Second):
		panic("expected the reader to recive the job, but it blocked")
	}

	select {
	case res = <-red.ResultChan():
		fmt.Println("Job operation success:", res.Err == nil)
	case <-time.After(5 * time.Second): // Should be more than the interval, otherwise the response is not ready yet
		panic("expected to recive a data back, nothing recieved")
	}

	buf := new(bytes.Buffer)
	buf.ReadFrom(res.Res)
	fmt.Println("Reader just received payload:", buf.String())

	cancel()

	_, open := <-redDone
	fmt.Println("Reader closure:", !open)

	_, open = <-recDone
	fmt.Println("Recorder closure:", !open)

	_, open = <-clDone
	fmt.Println("Client closure:", !open)

}

Output:

Engine creation success: true
Just sent a job request
Job was recorded
Job operation success: true
Reader just received payload: {"the key": "is the value!"}
Reader closure: true
Recorder closure: true
Client closure: true

func NewWithConfig added in v0.1.1 

func NewWithConfig(ctx context.Context, log logrus.FieldLogger, reader config.ReaderConf, recorders ...config.RecorderConf) (*Engine, error)

NewWithConfig instantiates reader and recorders from the configurations.

func NewWithReadRecorder added in v0.1.1 

func NewWithReadRecorder(ctx context.Context, logger logrus.FieldLogger, red reader.DataReader, recs ...recorder.DataRecorder) (*Engine, error)

NewWithReadRecorder creates an instance with already made reader and recorders. It spawns one reader and streams its payload to all recorders. Returns an error if there are recorders with the same name, or any of them have no name.

func (*Engine) Name added in v0.1.0 

func (e *Engine) Name() string

Name shows the name identifier for this engine

func (*Engine) Start added in v0.0.4 

func (e *Engine) Start() chan struct{}

Start begins pulling the data from DataReader and chips them to DataRecorder. When the context is canceled or timed out, the engine closes all job channels, causing the readers and recorders to stop.

func (*Engine) Stop added in v0.0.4 

func (e *Engine) Stop()

Stop closes the job channels

Source Files

View all Source files

Directories

Path Synopsis
cmd
expvastic
Package lib contains some functionalities needed for expvastic.
 Package lib contains some functionalities needed for expvastic.
expvar
self
Package self contains codes for recording expvars own metrics
 Package self contains codes for recording expvars own metrics
elasticsearch

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.