Documentation
¶
Overview ¶
Package expvastic can read from an endpoint which provides expvar data and ship them to elasticsearch. Please refer to golang's [expvar documentation](https://golang.org/pkg/expvar/) for more information. This is an early release and it is private. There will be a lot of changes soon but I'm planning to finalise and make it public as soon as I can. I hope you enjoy it!
Here is a couple of screenshots: http://i.imgur.com/6kB88g4.png and http://i.imgur.com/0ROSWsM.png
You need golang 1.7 (I haven't tested it with older versions, but they should be fine) and [glide](https://github.com/Masterminds/glide) installed. Simply do:
go get github.com/arsham/expvastic/... cd $GOPATH/src/github.com/arsham/expvastic glide install
You also need elasticsearch and kibana, here is a couple of docker images for you:
docker run -it --rm --name expvastic --ulimit nofile=98304:98304 -v "/path/to/somewhere/expvastic":/usr/share/elasticsearch/data elasticsearch docker run -it --rm --name kibana -p 80:5601 --link expvastic:elasticsearch -p 5601:5601 kibana
At the heart of this package, there is Engine. It acts like a glue between a Reader and a Recorder. Messages are transfered in a package called DataContainer, which is a list of DataType objects.
Here an example configuration, save it somewhere (let's call it expvastic.yml for now):
settings: debug_evel: info readers: FirstApp: # service name type: expvar endpoint: localhost:1234 routepath: /debug/vars interval: 500ms timeout: 3s log_level: debug backoff: 10 SomeApplication: type: expvar endpoint: localhost:1235 routepath: /debug/vars interval: 500ms timeout: 13s log_level: debug backoff: 10 recorders: main_elasticsearch: type: elasticsearch endpoint: 127.0.0.1:9200 index_name: expvastic type_name: my_app1 timeout: 8s backoff: 10 the_other_elasticsearch: type: elasticsearch endpoint: 127.0.0.1:9200 index_name: expvastic type_name: SomeApplication timeout: 18s backoff: 10 routes: route1: readers: - FirstApp recorders: - main_elasticsearch route2: readers: - FirstApp - SomeApplication recorders: - main_elasticsearch route3: readers: - SomeApplication recorders: - main_elasticsearch - the_other_elasticsearch
Then run the application:
expvasyml -c expvastic.yml
Please note that the name of the app will be changed to expvastic.
For running tests, do the following:
go test $(glide nv)
Use of this source code is governed by the Apache 2.0 license. License that can be found in the LICENSE file.
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var ( // ErrEmptyRecName is for when the recorder's name is an empty string. ErrEmptyRecName = fmt.Errorf("recorder name empty") // ErrDupRecName is for when there are two recorders with the same name. ErrDupRecName = fmt.Errorf("recorder name empty") )
Functions ¶
func StartEngines ¶ added in v0.1.0
func StartEngines(ctx context.Context, log logrus.FieldLogger, confMap *config.ConfMap) (chan struct{}, error)
StartEngines creates some Engines and returns a channel that closes it when it's done its work. For each routes, we need one engine that has one reader and writes to multiple recorders. This is because:
1 - Readers should not intercept each other by engaging the recorders. 2 - When a reader goes out of scope, we can safely stop the recorders.
When a recorder goes out of scope, the Engine stops sending to that recorder.
Types ¶
type Engine ¶ added in v0.0.4
type Engine struct {
// contains filtered or unexported fields
}
Engine represents an engine that receives information from readers and ships them to recorders. The Engine is allowed to change the index and type names at will. When the context times out or canceled, the engine will close the the job channels by calling the Stop method. Note that we could create a channel and distribute the recorders payload, but we didn't because there is no way to find out which recorder errors right after the payload has been sent. IMPORTANT: the readers should not close their streams, I am closing them here.
Example (SendJob) ¶
package main import ( "bytes" "context" "fmt" "io" "net/http" "net/http/httptest" "time" "github.com/arsham/expvastic" "github.com/arsham/expvastic/lib" "github.com/arsham/expvastic/reader" "github.com/arsham/expvastic/recorder" ) func main() { var res *reader.ReadJobResult log := lib.DiscardLogger() ctx, cancel := context.WithCancel(context.Background()) desire := `{"the key": "is the value!"}` redTs := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { io.WriteString(w, desire) })) defer redTs.Close() recTs := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { fmt.Println("Job was recorded") })) defer recTs.Close() ctxReader := reader.NewCtxReader(redTs.URL) red, _ := reader.NewSimpleReader(log, ctxReader, "reader_example", 10*time.Millisecond, 10*time.Millisecond) rec, _ := recorder.NewSimpleRecorder(ctx, log, "reader_example", recTs.URL, "intexName", "typeName", 10*time.Millisecond, 10*time.Millisecond) redDone := red.Start(ctx) recDone := rec.Start(ctx) cl, err := expvastic.NewWithReadRecorder(ctx, log, red, rec) fmt.Println("Engine creation success:", err == nil) clDone := cl.Start() select { case red.JobChan() <- ctx: fmt.Println("Just sent a job request") case <-time.After(time.Second): panic("expected the reader to recive the job, but it blocked") } select { case res = <-red.ResultChan(): fmt.Println("Job operation success:", res.Err == nil) case <-time.After(5 * time.Second): // Should be more than the interval, otherwise the response is not ready yet panic("expected to recive a data back, nothing recieved") } buf := new(bytes.Buffer) buf.ReadFrom(res.Res) fmt.Println("Reader just received payload:", buf.String()) cancel() _, open := <-redDone fmt.Println("Reader closure:", !open) _, open = <-recDone fmt.Println("Recorder closure:", !open) _, open = <-clDone fmt.Println("Client closure:", !open) }
Output: Engine creation success: true Just sent a job request Job was recorded Job operation success: true Reader just received payload: {"the key": "is the value!"} Reader closure: true Recorder closure: true Client closure: true
func NewWithConfig ¶ added in v0.1.1
func NewWithConfig(ctx context.Context, log logrus.FieldLogger, reader config.ReaderConf, recorders ...config.RecorderConf) (*Engine, error)
NewWithConfig instantiates reader and recorders from the configurations.
func NewWithReadRecorder ¶ added in v0.1.1
func NewWithReadRecorder(ctx context.Context, logger logrus.FieldLogger, red reader.DataReader, recs ...recorder.DataRecorder) (*Engine, error)
NewWithReadRecorder creates an instance with already made reader and recorders. It spawns one reader and streams its payload to all recorders. Returns an error if there are recorders with the same name, or any of them have no name.