sre

README

SRE framework

Framework for golang applications which helps to send metrics, logs and traces into different monitoring tools or vendors.

Features

• Logging tools (aka logs):
  • Stdout (text, json, template) based on Logrus
  • DataDog based on Logrus over UDP
• Monitoring tools (aka metrics)
  • Prometheus
  • DataDog
• Tracing tools (aka traces)
  • Jaeger
  • DataDog
  • Opentelemetry

Usage

Requirements

• Jaeger works with its Jaeger agent
• DataDog uses self made DataDog agent
• Opentelemetry communicates with its Opentelemetry collector

Envs

Set proper GOROOT and PATH variables

export GOROOT="$HOME/go/root/1.16.4"
export PATH="$PATH:$GOROOT/bin"

Logs usage

Create logs.go file to test logging functionality

package main

import (
  "time"

  "github.com/devopsext/sre/common"
  "github.com/devopsext/sre/provider"
)

var logs = common.NewLogs()

func test() {
  logs.Info("Info message to every log provider...")
  logs.Debug("Debug message to every log provider...")
  logs.Warn("Warn message to every log provider...")
}

func main() {

  // initialize Stdout logger
  stdout := provider.NewStdout(provider.StdoutOptions{
    Format:          "text",
    Level:           "info",
    Template:        "{{.file}} {{.msg}}",
    TimestampFormat: time.RFC3339Nano,
    TextColors:      true,
  })
  // set caller offset for file:line proper usage 
  stdout.SetCallerOffset(2)

  // initialize DataDog logger
  datadog := provider.NewDataDogLogger(provider.DataDogLoggerOptions{
    DataDogOptions: provider.DataDogOptions{
      ServiceName: "some-service",
      Environment: "stage",
    },
    Host:  "localhost", // set DataDog agent UDP logs host
    Port:  10518, // set DataDog agent UDP logs port
    Level: "info",
  }, logs, stdout)

  // add loggers
  logs.Register(stdout)
  logs.Register(datadog)

  test()
}

Collect go modules

go mod init sre
go mod tidy

go: finding module for package github.com/devopsext/sre/provider
go: finding module for package github.com/devopsext/sre/common
go: found github.com/devopsext/sre/common in github.com/devopsext/sre v0.0.4
go: found github.com/devopsext/sre/provider in github.com/devopsext/sre v0.0.4

Run logs example

go run logs.go

INFO[2021-06-17T17:32:30.585651118+03:00] Info message to every log provider...
WARN[2021-06-17T17:32:30.585798024+03:00] Warn message to every log provider...
...

Metrics usage

Create metrics.go file to test metrics functionality

package main

import (
  "sync"
  "time"

  "github.com/devopsext/sre/common"
  "github.com/devopsext/sre/provider"
)

var logs = common.NewLogs()
var metrics = common.NewMetrics()
var mainWG sync.WaitGroup

func test() {
  counter := metrics.Counter("calls", "Calls counter", []string{"time"})
  counter.Inc(time.Now().String())
}

func main() {

  // initialize Stdout logger
  stdout := provider.NewStdout(provider.StdoutOptions{
    Format:          "text",
    Level:           "info",
    Template:        "{{.file}} {{.msg}}",
    TimestampFormat: time.RFC3339Nano,
    TextColors:      true,
  })
  // set caller offset for file:line proper usage
  stdout.SetCallerOffset(2)

  // add Stdout logger
  logs.Register(stdout)

  // initialize Prometheus metricer
  prometheus := provider.NewPrometheus(provider.PrometheusOptions{
    URL:    "/metrics",
    Listen: "127.0.0.1:8080",
    Prefix: "sre",
  }, logs, stdout)
  prometheus.Start(&mainWG)

  // initialize DataDog metricer
  datadog := provider.NewDataDogMetricer(provider.DataDogMetricerOptions{
    DataDogOptions: provider.DataDogOptions{
      ServiceName: "some-service",
      Environment: "stage",
    },
    Host:  "localhost", // set DataDog agent UDP metrics host
    Port:  10518, // set DataDog agent UDP metrics port
  }, logs, stdout)

  // add metricers
  metrics.Register(prometheus)
  metrics.Register(datadog)

  test()

  mainWG.Wait()
}

Collect go modules

go mod init sre
go mod tidy

go: finding module for package github.com/devopsext/sre/provider
go: finding module for package github.com/devopsext/sre/common
go: found github.com/devopsext/sre/common in github.com/devopsext/sre v0.0.4
go: found github.com/devopsext/sre/provider in github.com/devopsext/sre v0.0.4

Run metrcis example

go run metrics.go

INFO[2021-06-17T18:00:30.247316085+03:00] Start prometheus endpoint...
INFO[2021-06-17T18:00:30.247526413+03:00] Prometheus is up. Listening...
INFO[2021-06-17T18:00:30.248965919+03:00] Datadog metrics are up...

Check Prometheus metrics

curl -sk http://127.0.0.1:8080/metrics | grep sre_

# HELP sre_calls Calls counter
# TYPE sre_calls counter
sre_calls{time="2021-06-17 18:00:30.248990729 +0300 EEST m=+0.002878298"} 1

Traces usage

Create traces.go file to test traces functionality

package main

import (
  "time"

  "github.com/devopsext/sre/common"
  "github.com/devopsext/sre/provider"
)

var logs = common.NewLogs()
var traces = common.NewTraces()

func test() {
  rootSpan := traces.StartSpan()

  // emulate delay of 100 msecs
  time.Sleep(time.Duration(100) * time.Millisecond)

  span := traces.StartChildSpan(rootSpan.GetContext())

  // emulate delay of 100 msecs
  time.Sleep(time.Duration(200) * time.Millisecond)
  logs.SpanInfo(span, "Something happened")
  span.Finish()

  rootSpan.Finish()

  // wait for a while to delivery all spans to provider 
  time.Sleep(time.Duration(3000) * time.Millisecond)
}

func main() {

  // initialize Stdout logger
  stdout := provider.NewStdout(provider.StdoutOptions{
    Format:          "text",
    Level:           "info",
    Template:        "{{.file}} {{.msg}}",
    TimestampFormat: time.RFC3339Nano,
    TextColors:      true,
  })
  // set caller offset for file:line proper usage
  stdout.SetCallerOffset(2)

  // add Stdout logger
  logs.Register(stdout)

  // initialize Jaeger tracer
  jaeger := provider.NewJaeger(provider.JaegerOptions{
    ServiceName:         "sre",
    AgentHost:           "localhost", // set Jaeger agent host
    AgentPort:           6831, // set Jaeger agent port
    BufferFlushInterval: 0,
    QueueSize:           0,
    Tags:                "key1=value1",
  }, logs, stdout)

  // initialize DataDog tracer
  datadog := provider.NewDataDogTracer(provider.DataDogTracerOptions{
    DataDogOptions: provider.DataDogOptions{
      ServiceName: "some-service",
      Environment: "stage",
    },
    Host:  "localhost", // set DataDog agent traces host
    Port:  8126, // set DataDog agent traces port
  }, logs, stdout)

  // initialize Opentelemetry tracer
  opentelemetry := provider.NewOpentelemetryTracer(provider.OpentelemetryTracerOptions{
    OpentelemetryOptions: provider.OpentelemetryOptions{
      ServiceName: "some-service",
      Environment: "stage",
    },
    Host:  "localhost", // set Opentelemetry collector traces host
    Port:  4317, // set Opentelemetry collector traces port
  }, logs, stdout)

// add traces
  traces.Register(jaeger)
  traces.Register(datadog)
  traces.Register(opentelemetry)

  test()
}

Collect go modules

go mod init sre
go mod tidy

go: finding module for package github.com/devopsext/sre/provider
go: finding module for package github.com/devopsext/sre/common
go: found github.com/devopsext/sre/common in github.com/devopsext/sre v0.0.4
go: found github.com/devopsext/sre/provider in github.com/devopsext/sre v0.0.4

Run traces example

go run traces.go

...
INFO[2021-06-17T18:28:45.178707109+03:00] Something happened
INFO[2021-06-17T18:28:45.178840198+03:00] Reporting span 10a2beaae092860a:486b0277d5e7ae83:10a2beaae092860a:1
INFO[2021-06-17T18:28:45.178940724+03:00] Reporting span 10a2beaae092860a:10a2beaae092860a:0000000000000000:1

Go to Jaeger UI and there should be seen

Framework in other projects

• devopsext/events Kubernetes & Alertmanager events to Telegram, Slack, Workchat and other messengers.