monkit

package module
v3.0.24 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 6, 2024 License: Apache-2.0 Imports: 21 Imported by: 271

README

monkit

Package monkit is a flexible code instrumenting and data collection library.

See documentation at https://godoc.org/github.com/spacemonkeygo/monkit/v3

Software is hard. Like, really hard. Just the worst. Sometimes it feels like we've constructed a field where the whole point is to see how tangled we can get ourselves before seeing if we can get tangled up more while trying to get untangled.

Many software engineering teams are coming to realize (some slower than others) that collecting data over time about how their systems are functioning is a super power you can't turn back from. Some teams are calling this Telemetry, Observability, or describing it more basically through subcomponents such as distributed tracing, time-series data, or even just metrics. We've been calling it monitoring, but geez, I suppose if trends continue and you want to do this yourself your first step should be to open a thesaurus and pick an unused term.

I'm not here to tell you about our whole platform. Instead, I'm here to explain a redesign of a Go library for instrumenting your Go programs that we rather quietly launched a few years ago. If you are already using version 1 of our old library, we're sorry, but we rewrote it from scratch and renamed it to monkit. This one (this one!) is better - you should switch!

I'm going to try and sell you as fast as I can on this library.

Example usage

package main

import (
  "context"
  "fmt"
  "log"
  "math/rand"
  "net/http"
  "time"

  "github.com/spacemonkeygo/monkit/v3"
  "github.com/spacemonkeygo/monkit/v3/environment"
  "github.com/spacemonkeygo/monkit/v3/present"
)

var mon = monkit.Package()

func main() {
  environment.Register(monkit.Default)

  go http.ListenAndServe("127.0.0.1:9000", present.HTTP(monkit.Default))

  for {
    time.Sleep(time.Second)
    log.Println(DoStuff(context.Background()))
  }
}

func DoStuff(ctx context.Context) (err error) {
  defer mon.Task()(&ctx)(&err)

  result, err := ComputeThing(ctx, 1, 2)
  if err != nil {
    return err
  }

  fmt.Println(result)
  return
}

func ComputeThing(ctx context.Context, arg1, arg2 int) (res int, err error) {
  defer mon.Task()(&ctx)(&err)

  timer := mon.Timer("subcomputation").Start()
  res = arg1 + arg2
  timer.Stop()

  if res == 3 {
    mon.Event("hit 3")
  }

  mon.BoolVal("was-4").Observe(res == 4)
  mon.IntVal("res").Observe(int64(res))
  mon.DurationVal("took").Observe(time.Second + time.Duration(rand.Intn(int(10*time.Second))))
  mon.Counter("calls").Inc(1)
  mon.Gauge("arg1", func() float64 { return float64(arg1) })
  mon.Meter("arg2").Mark(arg2)

  return arg1 + arg2, nil
}

Metrics

We've got tools that capture distribution information (including quantiles) about int64, float64, and bool types. We have tools that capture data about events (we've got meters for deltas, rates, etc). We have rich tools for capturing information about tasks and functions, and literally anything that can generate a name and a number.

Almost just as importantly, the amount of boilerplate and code you have to write to get these features is very minimal. Data that's hard to measure probably won't get measured.

This data can be collected and sent to Graphite or any other time-series database.

Here's a selection of live stats from one of our storage nodes:

env.os.fds      120.000000
env.os.proc.stat.Minflt 81155.000000
env.os.proc.stat.Cminflt        11789.000000
env.os.proc.stat.Majflt 10.000000
env.os.proc.stat.Cmajflt        6.000000
...

env.process.control     1.000000
env.process.crc 3819014369.000000
env.process.uptime      163225.292925
env.runtime.goroutines  52.000000
env.runtime.memory.Alloc        2414080.000000
...

env.rusage.Maxrss       26372.000000
...

sm/flud/csl/client.(*CSLClient).Verify.current  0.000000
sm/flud/csl/client.(*CSLClient).Verify.success  788.000000
sm/flud/csl/client.(*CSLClient).Verify.error volume missing     91.000000
sm/flud/csl/client.(*CSLClient).Verify.error dial error 1.000000
sm/flud/csl/client.(*CSLClient).Verify.panics   0.000000
sm/flud/csl/client.(*CSLClient).Verify.success times min        0.102214
sm/flud/csl/client.(*CSLClient).Verify.success times avg        1.899133
sm/flud/csl/client.(*CSLClient).Verify.success times max        8.601230
sm/flud/csl/client.(*CSLClient).Verify.success times recent     2.673128
sm/flud/csl/client.(*CSLClient).Verify.failure times min        0.682881
sm/flud/csl/client.(*CSLClient).Verify.failure times avg        3.936571
sm/flud/csl/client.(*CSLClient).Verify.failure times max        6.102318
sm/flud/csl/client.(*CSLClient).Verify.failure times recent     2.208020
sm/flud/csl/server.store.avg    710800.000000
sm/flud/csl/server.store.count  271.000000
sm/flud/csl/server.store.max    3354194.000000
sm/flud/csl/server.store.min    467.000000
sm/flud/csl/server.store.recent 1661376.000000
sm/flud/csl/server.store.sum    192626890.000000
...

Call graphs

This library generates call graphs of your live process for you.

These call graphs aren't created through sampling. They're full pictures of all of the interesting functions you've annotated, along with quantile information about their successes, failures, how often they panic, return an error (if so instrumented), how many are currently running, etc.

The data can be returned in dot format, in json, in text, and can be about just the functions that are currently executing, or all the functions the monitoring system has ever seen.

Here's another example of one of our production nodes:

callgraph

Trace graphs

This library generates trace graphs of your live process for you directly, without requiring standing up some tracing system such as Zipkin (though you can do that too).

Inspired by Google's Dapper and Twitter's Zipkin, we have process-internal trace graphs, triggerable by a number of different methods.

You get this trace information for free whenever you use Go contexts and function monitoring. The output formats are svg and json.

Additionally, the library supports trace observation plugins, and we've written a plugin that sends this data to Zipkin.

trace

History

Before our crazy Go rewrite of everything (and before we had even seen Google's Dapper paper), we were a Python shop, and all of our "interesting" functions were decorated with a helper that collected timing information and sent it to Graphite.

When we transliterated to Go, we wanted to preserve that functionality, so the first version of our monitoring package was born.

Over time it started to get janky, especially as we found Zipkin and started adding tracing functionality to it. We rewrote all of our Go code to use Google contexts, and then realized we could get call graph information. We decided a refactor and then an all-out rethinking of our monitoring package was best, and so now we have this library.

Aside about contexts

Sometimes you really want callstack contextual information without having to pass arguments through everything on the call stack. In other languages, many people implement this with thread-local storage.

Example: let's say you have written a big system that responds to user requests. All of your libraries log using your log library. During initial development everything is easy to debug, since there's low user load, but now you've scaled and there's OVER TEN USERS and it's kind of hard to tell what log lines were caused by what. Wouldn't it be nice to add request ids to all of the log lines kicked off by that request? Then you could grep for all log lines caused by a specific request id. Geez, it would suck to have to pass all contextual debugging information through all of your callsites.

Google solved this problem by always passing a context.Context interface through from call to call. A Context is basically just a mapping of arbitrary keys to arbitrary values that users can add new values for. This way if you decide to add a request context, you can add it to your Context and then all callsites that descend from that place will have the new data in their contexts.

It is admittedly very verbose to add contexts to every function call. Painfully so. I hope to write more about it in the future, but Google also wrote up their thoughts about it, which you can go read. For now, just swallow your disgust and let's keep moving.

Motivating program

Let's make a super simple Varnish clone. Open up gedit! (Okay just kidding, open whatever text editor you want.)

For this motivating program, we won't even add the caching, though there's comments for where to add it if you'd like. For now, let's just make a barebones system that will proxy HTTP requests. We'll call it VLite, but maybe we should call it VReallyLite.

package main

import (
  "flag"
  "net/http"
  "net/http/httputil"
  "net/url"
)

type VLite struct {
  target *url.URL
  proxy  *httputil.ReverseProxy
}

func NewVLite(target *url.URL) *VLite {
  return &VLite{
	  target: target,
	  proxy:  httputil.NewSingleHostReverseProxy(target),
  }
}

func (v *VLite) Proxy(w http.ResponseWriter, r *http.Request) {
  r.Host = v.target.Host // let the proxied server get the right vhost
  v.proxy.ServeHTTP(w, r)
}

func (v *VLite) ServeHTTP(w http.ResponseWriter, r *http.Request) {
  // here's where you'd put caching logic
  v.Proxy(w, r)
}

func main() {
  target := flag.String(
	  "proxy",
	  "http://hasthelargehadroncolliderdestroyedtheworldyet.com/",
	  "server to cache")
  flag.Parse()
  targetURL, err := url.Parse(*target)
  if err != nil {
	  panic(err)
  }
  panic(http.ListenAndServe(":8080", NewVLite(targetURL)))
}

Run and build this and open localhost:8080 in your browser. If you use the default proxy target, it should inform you that the world hasn't been destroyed yet.

Adding basic instrumentation

The first thing you'll want to do is add the small amount of boilerplate to make the instrumentation we're going to add to your process observable later.

Import the basic monkit packages:

"github.com/spacemonkeygo/monkit/v3"
"github.com/spacemonkeygo/monkit/v3/environment"
"github.com/spacemonkeygo/monkit/v3/present"

and then register environmental statistics and kick off a goroutine in your main method to serve debug requests:

environment.Register(monkit.Default)
go http.ListenAndServe("localhost:9000", present.HTTP(monkit.Default))

Rebuild, and then check out localhost:9000/stats (or localhost:9000/stats/json, if you prefer) in your browser!

Request contexts

Remember what I said about Google's contexts? It might seem a bit overkill for such a small project, but it's time to add them.

To help out here, I've created a library that constructs contexts for you for incoming HTTP requests. Nothing that's about to happen requires my webhelp library, but here is the code now refactored to receive and pass contexts through our two per-request calls.

package main

import (
  "context"
  "flag"
  "net/http"
  "net/http/httputil"
  "net/url"

  "github.com/jtolds/webhelp"
  "github.com/spacemonkeygo/monkit/v3"
  "github.com/spacemonkeygo/monkit/v3/environment"
  "github.com/spacemonkeygo/monkit/v3/present"
)

type VLite struct {
  target *url.URL
  proxy  *httputil.ReverseProxy
}

func NewVLite(target *url.URL) *VLite {
  return &VLite{
	  target: target,
	  proxy:  httputil.NewSingleHostReverseProxy(target),
  }
}

func (v *VLite) Proxy(ctx context.Context, w http.ResponseWriter, r *http.Request) {
  r.Host = v.target.Host // let the proxied server get the right vhost
  v.proxy.ServeHTTP(w, r)
}

func (v *VLite) HandleHTTP(ctx context.Context, w webhelp.ResponseWriter, r *http.Request) error {
  // here's where you'd put caching logic
  v.Proxy(ctx, w, r)
  return nil
}

func main() {
  target := flag.String(
	  "proxy",
	  "http://hasthelargehadroncolliderdestroyedtheworldyet.com/",
	  "server to cache")
  flag.Parse()
  targetURL, err := url.Parse(*target)
  if err != nil {
	  panic(err)
  }
  environment.Register(monkit.Default)
  go http.ListenAndServe("localhost:9000", present.HTTP(monkit.Default))
  panic(webhelp.ListenAndServe(":8080", NewVLite(targetURL)))
}

You can create a new context for a request however you want. One reason to use something like webhelp is that the cancelation feature of Contexts is hooked up to the HTTP request getting canceled.

Monitor some requests

Let's start to get statistics about how many requests we receive! First, this package (main) will need to get a monitoring Scope. Add this global definition right after all your imports, much like you'd create a logger with many logging libraries:

var mon = monkit.Package()

Now, make the error return value of HandleHTTP named (so, (err error)), and add this defer line as the very first instruction of HandleHTTP:

func (v *VLite) HandleHTTP(ctx context.Context, w webhelp.ResponseWriter, r *http.Request) (err error) {
  defer mon.Task()(&ctx)(&err)

Let's also add the same line (albeit modified for the lack of error) to Proxy, replacing &err with nil:

func (v *VLite) Proxy(ctx context.Context, w http.ResponseWriter, r *http.Request) {
  defer mon.Task()(&ctx)(nil)

You should now have something like:

package main

import (
  "context"
  "flag"
  "net/http"
  "net/http/httputil"
  "net/url"

  "github.com/jtolds/webhelp"
  "github.com/spacemonkeygo/monkit/v3"
  "github.com/spacemonkeygo/monkit/v3/environment"
  "github.com/spacemonkeygo/monkit/v3/present"
)

var mon = monkit.Package()

type VLite struct {
  target *url.URL
  proxy  *httputil.ReverseProxy
}

func NewVLite(target *url.URL) *VLite {
  return &VLite{
	  target: target,
	  proxy:  httputil.NewSingleHostReverseProxy(target),
  }
}

func (v *VLite) Proxy(ctx context.Context, w http.ResponseWriter, r *http.Request) {
  defer mon.Task()(&ctx)(nil)
  r.Host = v.target.Host // let the proxied server get the right vhost
  v.proxy.ServeHTTP(w, r)
}

func (v *VLite) HandleHTTP(ctx context.Context, w webhelp.ResponseWriter, r *http.Request) (err error) {
  defer mon.Task()(&ctx)(&err)
  // here's where you'd put caching logic
  v.Proxy(ctx, w, r)
  return nil
}

func main() {
  target := flag.String(
	  "proxy",
	  "http://hasthelargehadroncolliderdestroyedtheworldyet.com/",
	  "server to cache")
  flag.Parse()
  targetURL, err := url.Parse(*target)
  if err != nil {
	  panic(err)
  }
  environment.Register(monkit.Default)
  go http.ListenAndServe("localhost:9000", present.HTTP(monkit.Default))
  panic(webhelp.ListenAndServe(":8080", NewVLite(targetURL)))
}

We'll unpack what's going on here, but for now:

  • Rebuild and restart!
  • Trigger a full refresh at localhost:8080 to make sure your new HTTP handler runs
  • Visit localhost:9000/stats and then localhost:9000/funcs

For this new funcs dataset, if you want a graph, you can download a dot graph at localhost:9000/funcs/dot and json information from localhost:9000/funcs/json.

You should see something like:

[3693964236144930897] main.(*VLite).HandleHTTP
  parents: entry
  current: 0, highwater: 1, success: 2, errors: 0, panics: 0
  success times:
    0.00: 63.930436ms
    0.10: 70.482159ms
    0.25: 80.309745ms
    0.50: 96.689054ms
    0.75: 113.068363ms
    0.90: 122.895948ms
    0.95: 126.17181ms
    1.00: 129.447675ms
    avg: 96.689055ms
  failure times:
    0.00: 0
    0.10: 0
    0.25: 0
    0.50: 0
    0.75: 0
    0.90: 0
    0.95: 0
    1.00: 0
    avg: 0

with a similar report for the Proxy method, or a graph like:

handlehttp

This data reports the overall callgraph of execution for known traces, along with how many of each function are currently running, the most running concurrently (the highwater), how many were successful along with quantile timing information, how many errors there were (with quantile timing information if applicable), and how many panics there were. Since the Proxy method isn't capturing a returned err value, and since HandleHTTP always returns nil, this example won't ever have failures.

If you're wondering about the success count being higher than you expected, keep in mind your browser probably requested a favicon.ico.

Cool, eh?

How it works

defer mon.Task()(&ctx)(&err)

is an interesting line of code - there's three function calls. If you look at the Go spec, all of the function calls will run at the time the function starts except for the very last one.

The first function call, mon.Task(), creates or looks up a wrapper around a Func. You could get this yourself by requesting mon.Func() inside of the appropriate function or mon.FuncNamed(). Both mon.Task() and mon.Func() are inspecting runtime.Caller to determine the name of the function. Because this is a heavy operation, you can actually store the result of mon.Task() and reuse it somehow else if you prefer, so instead of

func MyFunc(ctx context.Context) (err error) {
  defer mon.Task()(&ctx)(&err)
}

you could instead use

var myFuncMon = mon.Task()

func MyFunc(ctx context.Context) (err error) {
  defer myFuncMon(&ctx)(&err)
}

which is more performant every time after the first time. runtime.Caller only gets called once.

Careful! Don't use the same myFuncMon in different functions unless you want to screw up your statistics!

The second function call starts all the various stop watches and bookkeeping to keep track of the function. It also mutates the context pointer it's given to extend the context with information about what current span (in Zipkin parlance) is active. Notably, you can pass nil for the context if you really don't want a context. You just lose callgraph information.

The last function call stops all the stop watches ad makes a note of any observed errors or panics (it repanics after observing them).

Tracing

Turns out, we don't even need to change our program anymore to get rich tracing information!

Open your browser and go to localhost:9000/trace/svg?regex=HandleHTTP. It won't load, and in fact, it's waiting for you to open another tab and refresh localhost:8080 again. Once you retrigger the actual application behavior, the trace regex will capture a trace starting on the first function that matches the supplied regex, and return an svg. Go back to your first tab, and you should see a relatively uninteresting but super promising svg.

Let's make the trace more interesting. Add a

time.Sleep(200 * time.Millisecond)

to your HandleHTTP method, rebuild, and restart. Load localhost:8080, then start a new request to your trace URL, then reload localhost:8080 again. Flip back to your trace, and you should see that the Proxy method only takes a portion of the time of HandleHTTP!

trace

There's multiple ways to select a trace. You can select by regex using the preselect method (default), which first evaluates the regex on all known functions for sanity checking. Sometimes, however, the function you want to trace may not yet be known to monkit, in which case you'll want to turn preselection off. You may have a bad regex, or you may be in this case if you get the error "Bad Request: regex preselect matches 0 functions."

Another way to select a trace is by providing a trace id, which we'll get to next!

Make sure to check out what the addition of the time.Sleep call did to the other reports.

Plugins

It's easy to write plugins for monkit! Check out our first one that exports data to Zipkin's Scribe API:

We plan to have more (for HTrace, OpenTracing, etc, etc), soon!

License

Copyright (C) 2016 Space Monkey, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Documentation

Overview

Package monkit is a flexible code instrumenting and data collection library.

I'm going to try and sell you as fast as I can on this library.

Example usage 

package main

import (
	"context"
	"fmt"
	"log"
	"net/http"

	"github.com/spacemonkeygo/monkit/v3"
	"github.com/spacemonkeygo/monkit/v3/environment"
	"github.com/spacemonkeygo/monkit/v3/present"
)

var (
	mon = monkit.Package()
)

func ComputeThing(ctx context.Context, arg1, arg2 int) (res int, err error) {
	defer mon.Task()(&ctx)(&err)

	timer := mon.Timer("subcomputation").Start()
	res = arg1 + arg2
	timer.Stop()

	if res == 3 {
		mon.Event("hit 3")
	}

	mon.BoolVal("was-4").Observe(res == 4)
	mon.IntVal("res").Observe(int64(res))
	mon.DurationVal("took").Observe(t2.Sub(t1))
	mon.Counter("calls").Inc(1)
	mon.Gauge("arg1", func() float64 { return float64(arg1) })
	mon.Meter("arg2").Mark(arg2)

	return arg1 + arg2, nil
}

func DoStuff(ctx context.Context) (err error) {
	defer mon.Task()(&ctx)(&err)

	result, err := ComputeThing(ctx, 1, 2)
	if err != nil {
		return err
	}

	fmt.Println(result)
	return
}

func main() {
	environment.Register(monkit.Default)
	go http.ListenAndServe("localhost:9000", present.HTTP(monkit.Default))
	log.Println(DoStuff(context.Background()))
}

Metrics

We've got tools that capture distribution information (including quantiles) about int64, float64, and bool types. We have tools that capture data about events (we've got meters for deltas, rates, etc). We have rich tools for capturing information about tasks and functions, and literally anything that can generate a name and a number.

Almost just as importantly, the amount of boilerplate and code you have to write to get these features is very minimal. Data that's hard to measure probably won't get measured.

This data can be collected and sent to Graphite (http://graphite.wikidot.com/) or any other time-series database.

Here's a selection of live stats from one of our storage nodes: 

env.os.fds      120.000000
env.os.proc.stat.Minflt 81155.000000
env.os.proc.stat.Cminflt        11789.000000
env.os.proc.stat.Majflt 10.000000
env.os.proc.stat.Cmajflt        6.000000
...

env.process.control     1.000000
env.process.crc 3819014369.000000
env.process.uptime      163225.292925
env.runtime.goroutines  52.000000
env.runtime.memory.Alloc        2414080.000000
...

env.rusage.Maxrss       26372.000000
...

sm/flud/csl/client.(*CSLClient).Verify.current  0.000000
sm/flud/csl/client.(*CSLClient).Verify.success  788.000000
sm/flud/csl/client.(*CSLClient).Verify.error volume missing     91.000000
sm/flud/csl/client.(*CSLClient).Verify.error dial error 1.000000
sm/flud/csl/client.(*CSLClient).Verify.panics   0.000000
sm/flud/csl/client.(*CSLClient).Verify.success times min        0.102214
sm/flud/csl/client.(*CSLClient).Verify.success times avg        1.899133
sm/flud/csl/client.(*CSLClient).Verify.success times max        8.601230
sm/flud/csl/client.(*CSLClient).Verify.success times recent     2.673128
sm/flud/csl/client.(*CSLClient).Verify.failure times min        0.682881
sm/flud/csl/client.(*CSLClient).Verify.failure times avg        3.936571
sm/flud/csl/client.(*CSLClient).Verify.failure times max        6.102318
sm/flud/csl/client.(*CSLClient).Verify.failure times recent     2.208020
sm/flud/csl/server.store.avg    710800.000000
sm/flud/csl/server.store.count  271.000000
sm/flud/csl/server.store.max    3354194.000000
sm/flud/csl/server.store.min    467.000000
sm/flud/csl/server.store.recent 1661376.000000
sm/flud/csl/server.store.sum    192626890.000000
...

Call graphs

This library generates call graphs of your live process for you.

These call graphs aren't created through sampling. They're full pictures of all of the interesting functions you've annotated, along with quantile information about their successes, failures, how often they panic, return an error (if so instrumented), how many are currently running, etc.

The data can be returned in dot format, in json, in text, and can be about just the functions that are currently executing, or all the functions the monitoring system has ever seen.

Here's another example of one of our production nodes:

https://raw.githubusercontent.com/spacemonkeygo/monkit/master/images/callgraph2.png

Trace graphs

This library generates trace graphs of your live process for you directly, without requiring standing up some tracing system such as Zipkin (though you can do that too).

Inspired by Google's Dapper (http://research.google.com/pubs/pub36356.html) and Twitter's Zipkin (http://zipkin.io), we have process-internal trace graphs, triggerable by a number of different methods.

You get this trace information for free whenever you use Go contexts (https://blog.golang.org/context) and function monitoring. The output formats are svg and json.

Additionally, the library supports trace observation plugins, and we've written a plugin that sends this data to Zipkin (http://github.com/spacemonkeygo/monkit-zipkin).

https://raw.githubusercontent.com/spacemonkeygo/monkit/master/images/trace.png

History

Before our crazy Go rewrite of everything (https://www.spacemonkey.com/blog/posts/go-space-monkey) (and before we had even seen Google's Dapper paper), we were a Python shop, and all of our "interesting" functions were decorated with a helper that collected timing information and sent it to Graphite.

When we transliterated to Go, we wanted to preserve that functionality, so the first version of our monitoring package was born.

Over time it started to get janky, especially as we found Zipkin and started adding tracing functionality to it. We rewrote all of our Go code to use Google contexts, and then realized we could get call graph information. We decided a refactor and then an all-out rethinking of our monitoring package was best, and so now we have this library.

Aside about contexts

Sometimes you really want callstack contextual information without having to pass arguments through everything on the call stack. In other languages, many people implement this with thread-local storage.

Example: let's say you have written a big system that responds to user requests. All of your libraries log using your log library. During initial development everything is easy to debug, since there's low user load, but now you've scaled and there's OVER TEN USERS and it's kind of hard to tell what log lines were caused by what. Wouldn't it be nice to add request ids to all of the log lines kicked off by that request? Then you could grep for all log lines caused by a specific request id. Geez, it would suck to have to pass all contextual debugging information through all of your callsites.

Google solved this problem by always passing a context.Context interface through from call to call. A Context is basically just a mapping of arbitrary keys to arbitrary values that users can add new values for. This way if you decide to add a request context, you can add it to your Context and then all callsites that decend from that place will have the new data in their contexts.

It is admittedly very verbose to add contexts to every function call. Painfully so. I hope to write more about it in the future, but Google also wrote up their thoughts about it (https://blog.golang.org/context), which you can go read. For now, just swallow your disgust and let's keep moving.

Motivating program

Let's make a super simple Varnish (https://www.varnish-cache.org/) clone. Open up gedit! (Okay just kidding, open whatever text editor you want.)

For this motivating program, we won't even add the caching, though there's comments for where to add it if you'd like. For now, let's just make a barebones system that will proxy HTTP requests. We'll call it VLite, but maybe we should call it VReallyLite. 

  package main

  import (
	  "flag"
	  "net/http"
	  "net/http/httputil"
	  "net/url"
  )

  type VLite struct {
	  target *url.URL
	  proxy  *httputil.ReverseProxy
  }

  func NewVLite(target *url.URL) *VLite {
	  return &VLite{
		  target: target,
		  proxy:  httputil.NewSingleHostReverseProxy(target),
	  }
  }

  func (v *VLite) Proxy(w http.ResponseWriter, r *http.Request) {
	  r.Host = v.target.Host // let the proxied server get the right vhost
	  v.proxy.ServeHTTP(w, r)
  }

  func (v *VLite) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	  // here's where you'd put caching logic
	  v.Proxy(w, r)
  }

  func main() {
	  target := flag.String(
		  "proxy",
		  "http://hasthelargehadroncolliderdestroyedtheworldyet.com/",
		  "server to cache")
	  flag.Parse()
	  targetURL, err := url.Parse(*target)
	  if err != nil {
		  panic(err)
	  }
	  panic(http.ListenAndServe(":8080", NewVLite(targetURL)))
  }

Run and build this and open localhost:8080 in your browser. If you use the default proxy target, it should inform you that the world hasn't been destroyed yet.

Adding basic instrumentation

The first thing you'll want to do is add the small amount of boilerplate to make the instrumentation we're going to add to your process observable later.

Import the basic monkit packages: 

"github.com/spacemonkeygo/monkit/v3"
"github.com/spacemonkeygo/monkit/v3/environment"
"github.com/spacemonkeygo/monkit/v3/present"

and then register environmental statistics and kick off a goroutine in your main method to serve debug requests: 

environment.Register(monkit.Default)
go http.ListenAndServe("localhost:9000", present.HTTP(monkit.Default))

Rebuild, and then check out localhost:9000/stats (or localhost:9000/stats/json, if you prefer) in your browser!

Request contexts

Remember what I said about Google's contexts (https://blog.golang.org/context)? It might seem a bit overkill for such a small project, but it's time to add them.

To help out here, I've created a library that constructs contexts for you for incoming HTTP requests. Nothing that's about to happen requires my webhelp library (https://godoc.org/github.com/jtolds/webhelp), but here is the code now refactored to receive and pass contexts through our two per-request calls. 

  package main

  import (
	  "context"
	  "flag"
	  "net/http"
	  "net/http/httputil"
	  "net/url"

	  "github.com/jtolds/webhelp"
	  "github.com/spacemonkeygo/monkit/v3"
	  "github.com/spacemonkeygo/monkit/v3/environment"
	  "github.com/spacemonkeygo/monkit/v3/present"
  )

  type VLite struct {
	  target *url.URL
	  proxy  *httputil.ReverseProxy
  }

  func NewVLite(target *url.URL) *VLite {
	  return &VLite{
		  target: target,
		  proxy:  httputil.NewSingleHostReverseProxy(target),
	  }
  }

  func (v *VLite) Proxy(ctx context.Context, w http.ResponseWriter, r *http.Request) {
	  r.Host = v.target.Host // let the proxied server get the right vhost
	  v.proxy.ServeHTTP(w, r)
  }

  func (v *VLite) HandleHTTP(ctx context.Context, w webhelp.ResponseWriter, r *http.Request) error {
	  // here's where you'd put caching logic
	  v.Proxy(ctx, w, r)
	  return nil
  }

  func main() {
	  target := flag.String(
		  "proxy",
		  "http://hasthelargehadroncolliderdestroyedtheworldyet.com/",
		  "server to cache")
	  flag.Parse()
	  targetURL, err := url.Parse(*target)
	  if err != nil {
		  panic(err)
	  }
	  environment.Register(monkit.Default)
	  go http.ListenAndServe("localhost:9000", present.HTTP(monkit.Default))
	  panic(webhelp.ListenAndServe(":8080", NewVLite(targetURL)))
  }

You can create a new context for a request however you want. One reason to use something like webhelp is that the cancelation feature of Contexts is hooked up to the HTTP request getting canceled.

Monitor some requests

Let's start to get statistics about how many requests we receive! First, this package (main) will need to get a monitoring Scope. Add this global definition right after all your imports, much like you'd create a logger with many logging libraries: 

var mon = monkit.Package()

Now, make the error return value of HandleHTTP named (so, (err error)), and add this defer line as the very first instruction of HandleHTTP: 

  func (v *VLite) HandleHTTP(ctx context.Context, w webhelp.ResponseWriter, r *http.Request) (err error) {
	  defer mon.Task()(&ctx)(&err)

Let's also add the same line (albeit modified for the lack of error) to Proxy, replacing &err with nil: 

  func (v *VLite) Proxy(ctx context.Context, w http.ResponseWriter, r *http.Request) {
	  defer mon.Task()(&ctx)(nil)

You should now have something like: 

  package main

  import (
	  "context"
	  "flag"
	  "net/http"
	  "net/http/httputil"
	  "net/url"

	  "github.com/jtolds/webhelp"
	  "github.com/spacemonkeygo/monkit/v3"
	  "github.com/spacemonkeygo/monkit/v3/environment"
	  "github.com/spacemonkeygo/monkit/v3/present"
  )

  var mon = monkit.Package()

  type VLite struct {
	  target *url.URL
	  proxy  *httputil.ReverseProxy
  }

  func NewVLite(target *url.URL) *VLite {
	  return &VLite{
		  target: target,
		  proxy:  httputil.NewSingleHostReverseProxy(target),
	  }
  }

  func (v *VLite) Proxy(ctx context.Context, w http.ResponseWriter, r *http.Request) {
    defer mon.Task()(&ctx)(nil)
	  r.Host = v.target.Host // let the proxied server get the right vhost
	  v.proxy.ServeHTTP(w, r)
  }

  func (v *VLite) HandleHTTP(ctx context.Context, w webhelp.ResponseWriter, r *http.Request) (err error) {
    defer mon.Task()(&ctx)(&err)
	  // here's where you'd put caching logic
	  v.Proxy(ctx, w, r)
	  return nil
  }

  func main() {
	  target := flag.String(
		  "proxy",
		  "http://hasthelargehadroncolliderdestroyedtheworldyet.com/",
		  "server to cache")
	  flag.Parse()
	  targetURL, err := url.Parse(*target)
	  if err != nil {
		  panic(err)
	  }
	  environment.Register(monkit.Default)
	  go http.ListenAndServe("localhost:9000", present.HTTP(monkit.Default))
	  panic(webhelp.ListenAndServe(":8080", NewVLite(targetURL)))
  }

We'll unpack what's going on here, but for now:

  • Rebuild and restart!
  • Trigger a full refresh at localhost:8080 to make sure your new HTTP handler runs
  • Visit localhost:9000/stats and then localhost:9000/funcs

For this new funcs dataset, if you want a graph, you can download a dot graph at localhost:9000/funcs/dot and json information from localhost:9000/funcs/json.

You should see something like: 

[3693964236144930897] main.(*VLite).HandleHTTP
  parents: entry
  current: 0, highwater: 1, success: 2, errors: 0, panics: 0
  success times:
    0.00: 63.930436ms
    0.10: 70.482159ms
    0.25: 80.309745ms
    0.50: 96.689054ms
    0.75: 113.068363ms
    0.90: 122.895948ms
    0.95: 126.17181ms
    1.00: 129.447675ms
    avg: 96.689055ms
  failure times:
    0.00: 0
    0.10: 0
    0.25: 0
    0.50: 0
    0.75: 0
    0.90: 0
    0.95: 0
    1.00: 0
    avg: 0

with a similar report for the Proxy method, or a graph like:

https://raw.githubusercontent.com/spacemonkeygo/monkit/master/images/handlehttp.png

This data reports the overall callgraph of execution for known traces, along with how many of each function are currently running, the most running concurrently (the highwater), how many were successful along with quantile timing information, how many errors there were (with quantile timing information if applicable), and how many panics there were. Since the Proxy method isn't capturing a returned err value, and since HandleHTTP always returns nil, this example won't ever have failures.

If you're wondering about the success count being higher than you expected, keep in mind your browser probably requested a favicon.ico.

Cool, eh?

How it works 

defer mon.Task()(&ctx)(&err)

is an interesting line of code - there's three function calls. If you look at the Go spec, all of the function calls will run at the time the function starts except for the very last one.

The first function call, mon.Task(), creates or looks up a wrapper around a Func. You could get this yourself by requesting mon.Func() inside of the appropriate function or mon.FuncNamed(). Both mon.Task() and mon.Func() are inspecting runtime.Caller to determine the name of the function. Because this is a heavy operation, you can actually store the result of mon.Task() and reuse it somehow else if you prefer, so instead of 

  func MyFunc(ctx context.Context) (err error) {
	  defer mon.Task()(&ctx)(&err)
  }

you could instead use 

  var myFuncMon = mon.Task()

  func MyFunc(ctx context.Context) (err error) {
	  defer myFuncMon(&ctx)(&err)
  }

which is more performant every time after the first time. runtime.Caller only gets called once.

Careful! Don't use the same myFuncMon in different functions unless you want to screw up your statistics!

The second function call starts all the various stop watches and bookkeeping to keep track of the function. It also mutates the context pointer it's given to extend the context with information about what current span (in Zipkin parlance) is active. Notably, you *can* pass nil for the context if you really don't want a context. You just lose callgraph information.

The last function call stops all the stop watches ad makes a note of any observed errors or panics (it repanics after observing them).

Tracing

Turns out, we don't even need to change our program anymore to get rich tracing information!

Open your browser and go to localhost:9000/trace/svg?regex=HandleHTTP. It won't load, and in fact, it's waiting for you to open another tab and refresh localhost:8080 again. Once you retrigger the actual application behavior, the trace regex will capture a trace starting on the first function that matches the supplied regex, and return an svg. Go back to your first tab, and you should see a relatively uninteresting but super promising svg.

Let's make the trace more interesting. Add a 

time.Sleep(200 * time.Millisecond)

to your HandleHTTP method, rebuild, and restart. Load localhost:8080, then start a new request to your trace URL, then reload localhost:8080 again. Flip back to your trace, and you should see that the Proxy method only takes a portion of the time of HandleHTTP!

https://cdn.rawgit.com/spacemonkeygo/monkit/master/images/trace.svg

There's multiple ways to select a trace. You can select by regex using the preselect method (default), which first evaluates the regex on all known functions for sanity checking. Sometimes, however, the function you want to trace may not yet be known to monkit, in which case you'll want to turn preselection off. You may have a bad regex, or you may be in this case if you get the error "Bad Request: regex preselect matches 0 functions."

Another way to select a trace is by providing a trace id, which we'll get to next!

Make sure to check out what the addition of the time.Sleep call did to the other reports.

Plugins

It's easy to write plugins for monkit! Check out our first one that exports data to Zipkin (http://zipkin.io/)'s Scribe API:

https://github.com/spacemonkeygo/monkit-zipkin

We plan to have more (for HTrace, OpenTracing, etc, etc), soon!

Index

Constants

View Source 
const (
	ReservoirSize = 64
)

Variables

View Source 
var Default = NewRegistry()

Default is the default Registry

View Source 
var ObservedQuantiles = []float64{0, .1, .25, .5, .75, .9, .95, 1}

ObservedQuantiles is the set of quantiles the internal distribution measurement logic will try to optimize for, if applicable.

View Source 
var (
	// If Window is > 0, the probability of replacing a datapoint will never
	// fall below ReservoirSize/Window instead of continuing to fall over time.
	// Window should be a multiple of ReservoirSize.
	Window int64 = 1024
)

Functions

func AddErrorNameHandler 

func AddErrorNameHandler(f func(error) (string, bool))

AddErrorNameHandler adds an error name handler function that will be consulted every time an error is captured for a task. The handlers will be called in the order they were registered with the most recently added handler first, until a handler returns true for the second return value. If no handler returns true, the error is checked to see if it implements an interface that allows it to name itself, and otherwise, monkit attempts to find a good name for most built in Go standard library errors.

func Collect 

func Collect(mon StatSource) map[string]float64

Collect takes something that implements the StatSource interface and returns a key/value map.

func Funcs 

func Funcs(cb func(f *Func))

Funcs is just a wrapper around Default.Funcs

func NewId 

func NewId() int64

NewId returns a random integer intended for use when constructing new traces. See NewTrace.

func ResetContextSpan added in v3.0.2 

func ResetContextSpan(ctx context.Context) context.Context

ResetContextSpan returns a new context with Span information removed.

func RootSpans 

func RootSpans(cb func(s *Span))

RootSpans is just a wrapper around Default.RootSpans

func Scopes 

func Scopes(cb func(s *Scope))

Scopes is just a wrapper around Default.Scopes

func Stats 

func Stats(cb func(key SeriesKey, field string, val float64))

Stats is just a wrapper around Default.Stats

Types

type Annotation 

type Annotation struct {
	Name  string
	Value string
}

Annotation represents an arbitrary name and value string pair

type BoolVal 

type BoolVal struct {
	// contains filtered or unexported fields
}

BoolVal keeps statistics about boolean values. It keeps the number of trues, number of falses, and the disposition (number of trues minus number of falses). Constructed using NewBoolVal, though its expected usage is like: 

var mon = monkit.Package()

func MyFunc() {
  ...
  mon.BoolVal("flipped").Observe(bool)
  ...
}

func NewBoolVal 

func NewBoolVal(key SeriesKey) *BoolVal

NewBoolVal creates a BoolVal

func (*BoolVal) Observe 

func (v *BoolVal) Observe(val bool)

Observe observes a boolean value

func (*BoolVal) Stats 

func (v *BoolVal) Stats(cb func(key SeriesKey, field string, val float64))

Stats implements the StatSource interface.

type CallbackTransformer added in v3.0.15 

type CallbackTransformer interface {
	Transform(func(SeriesKey, string, float64)) func(SeriesKey, string, float64)
}

CallbackTransformer will take a provided callback and return a transformed one.

type CallbackTransformerFunc added in v3.0.15 

type CallbackTransformerFunc func(func(SeriesKey, string, float64)) func(SeriesKey, string, float64)

CallbackTransformerFunc is a single function that implements CallbackTransformer's Transform.

func (CallbackTransformerFunc) Transform added in v3.0.15 

func (f CallbackTransformerFunc) Transform(cb func(SeriesKey, string, float64)) func(SeriesKey, string, float64)

Transform implements CallbackTransformer.

type Counter 

type Counter struct {
	// contains filtered or unexported fields
}

Counter keeps track of running totals, along with the highest and lowest values seen. The overall value can increment or decrement. Counter implements StatSource. Should be constructed with NewCounter(), though it may be more convenient to use the Counter accessor on a given Scope. Expected creation is like: 

var mon = monkit.Package()

func MyFunc() {
  mon.Counter("beans").Inc(1)
}

func NewCounter 

func NewCounter(key SeriesKey) *Counter

NewCounter constructs a counter

func (*Counter) Current 

func (c *Counter) Current() (cur int64)

Current returns the current value

func (*Counter) Dec 

func (c *Counter) Dec(delta int64) (current int64)

Dec will atomically decrement the counter by delta and return the new value.

func (*Counter) High 

func (c *Counter) High() (h int64)

High returns the highest value seen since construction or the last reset

func (*Counter) Inc 

func (c *Counter) Inc(delta int64) (current int64)

Inc will atomically increment the counter by delta and return the new value.

func (*Counter) Low 

func (c *Counter) Low() (l int64)

Low returns the lowest value seen since construction or the last reset

func (*Counter) Reset 

func (c *Counter) Reset() (val, low, high int64)

Reset resets all values including high/low counters and returns what they were.

func (*Counter) Set 

func (c *Counter) Set(val int64) (former int64)

Set will immediately change the value of the counter to whatever val is. It will appropriately update the high and low values, and return the former value.

func (*Counter) Stats 

func (c *Counter) Stats(cb func(key SeriesKey, field string, val float64))

Stats implements the StatSource interface

type DeltaTransformer added in v3.0.15 

type DeltaTransformer struct {
	// contains filtered or unexported fields
}

DeltaTransformer calculates deltas from any total fields. It keeps internal state to keep track of the previous totals, so care should be taken to use a different DeltaTransformer per output.

func NewDeltaTransformer added in v3.0.15 

func NewDeltaTransformer() *DeltaTransformer

NewDeltaTransformer creates a new DeltaTransformer with its own idea of the last totals seen.

func (*DeltaTransformer) Transform added in v3.0.15 

func (dt *DeltaTransformer) Transform(cb func(SeriesKey, string, float64)) func(SeriesKey, string, float64)

Transform implements CallbackTransformer.

type DiffMeter 

type DiffMeter struct {
	// contains filtered or unexported fields
}

DiffMeter is a StatSource that shows the difference between the rates of two meters. Expected usage like: 

var (
  mon = monkit.Package()
  herps = mon.Meter("herps")
  derps = mon.Meter("derps")
  herpToDerp = mon.DiffMeter("herp_to_derp", herps, derps)
)

func NewDiffMeter 

func NewDiffMeter(key SeriesKey, meter1, meter2 *Meter) *DiffMeter

Constructs a DiffMeter.

func (*DiffMeter) Stats 

func (m *DiffMeter) Stats(cb func(key SeriesKey, field string, val float64))

Stats implements the StatSource interface

type DurationDist 

type DurationDist struct {
	// Low and High are the lowest and highest values observed since
	// construction or the last reset.
	Low, High time.Duration

	// Recent is the last observed value.
	Recent time.Duration

	// Count is the number of observed values since construction or the last
	// reset.
	Count int64

	// Sum is the sum of all the observed values since construction or the last
	// reset.
	Sum time.Duration
	// contains filtered or unexported fields
}

DurationDist keeps statistics about values such as low/high/recent/average/quantiles. Not threadsafe. Construct with NewDurationDist(). Fields are expected to be read from but not written to.

func NewDurationDist 

func NewDurationDist(key SeriesKey) (d *DurationDist)

NewDurationDist creates a distribution of time.Durations.

func (*DurationDist) Copy 

func (d *DurationDist) Copy() *DurationDist

Copy returns a full copy of the entire distribution.

func (*DurationDist) FullAverage 

func (d *DurationDist) FullAverage() time.Duration

FullAverage calculates and returns the average of all inserted values.

func (*DurationDist) Insert 

func (d *DurationDist) Insert(val time.Duration)

Insert adds a value to the distribution, updating appropriate values.

func (*DurationDist) Query 

func (d *DurationDist) Query(quantile float64) time.Duration

Query will return the approximate value at the given quantile from the reservoir, where 0 <= quantile <= 1.

func (*DurationDist) ReservoirAverage 

func (d *DurationDist) ReservoirAverage() time.Duration

ReservoirAverage calculates the average of the current reservoir.

func (*DurationDist) Reset 

func (d *DurationDist) Reset()

func (*DurationDist) Stats 

func (d *DurationDist) Stats(cb func(key SeriesKey, field string, val float64))

type DurationVal added in v3.0.7 

type DurationVal struct {
	// contains filtered or unexported fields
}

DurationVal is a convenience wrapper around an DurationVal. Constructed using NewDurationVal, though its expected usage is like: 

var mon = monkit.Package()

func MyFunc() {
  ...
  mon.DurationVal("time").Observe(val)
  ...
}

func NewDurationVal added in v3.0.7 

func NewDurationVal(key SeriesKey) (v *DurationVal)

NewDurationVal creates an DurationVal

func (*DurationVal) Observe added in v3.0.7 

func (v *DurationVal) Observe(val time.Duration)

Observe observes an integer value

func (*DurationVal) Quantile added in v3.0.7 

func (v *DurationVal) Quantile(quantile float64) (rv time.Duration)

Quantile returns an estimate of the requested quantile of observed values. 0 <= quantile <= 1

func (*DurationVal) Stats added in v3.0.7 

func (v *DurationVal) Stats(cb func(key SeriesKey, field string, val float64))

Stats implements the StatSource interface.

type FloatDist 

type FloatDist struct {
	// Low and High are the lowest and highest values observed since
	// construction or the last reset.
	Low, High float64

	// Recent is the last observed value.
	Recent float64

	// Count is the number of observed values since construction or the last
	// reset.
	Count int64

	// Sum is the sum of all the observed values since construction or the last
	// reset.
	Sum float64
	// contains filtered or unexported fields
}

FloatDist keeps statistics about values such as low/high/recent/average/quantiles. Not threadsafe. Construct with NewFloatDist(). Fields are expected to be read from but not written to.

func NewFloatDist 

func NewFloatDist(key SeriesKey) (d *FloatDist)

NewFloatDist creates a distribution of float64s.

func (*FloatDist) Copy 

func (d *FloatDist) Copy() *FloatDist

Copy returns a full copy of the entire distribution.

func (*FloatDist) FullAverage 

func (d *FloatDist) FullAverage() float64

FullAverage calculates and returns the average of all inserted values.

func (*FloatDist) Insert 

func (d *FloatDist) Insert(val float64)

Insert adds a value to the distribution, updating appropriate values.

func (*FloatDist) Query 

func (d *FloatDist) Query(quantile float64) float64

Query will return the approximate value at the given quantile from the reservoir, where 0 <= quantile <= 1.

func (*FloatDist) ReservoirAverage 

func (d *FloatDist) ReservoirAverage() float64

ReservoirAverage calculates the average of the current reservoir.

func (*FloatDist) Reset 

func (d *FloatDist) Reset()

func (*FloatDist) Stats 

func (d *FloatDist) Stats(cb func(key SeriesKey, field string, val float64))

type FloatVal 

type FloatVal struct {
	// contains filtered or unexported fields
}

FloatVal is a convenience wrapper around an FloatDist. Constructed using NewFloatVal, though its expected usage is like: 

var mon = monkit.Package()

func MyFunc() {
  ...
  mon.FloatVal("size").Observe(val)
  ...
}

func NewFloatVal 

func NewFloatVal(key SeriesKey) (v *FloatVal)

NewFloatVal creates a FloatVal

func (*FloatVal) Observe 

func (v *FloatVal) Observe(val float64)

Observe observes an floating point value

func (*FloatVal) Quantile 

func (v *FloatVal) Quantile(quantile float64) (rv float64)

Quantile returns an estimate of the requested quantile of observed values. 0 <= quantile <= 1

func (*FloatVal) Stats 

func (v *FloatVal) Stats(cb func(key SeriesKey, field string, val float64))

Stats implements the StatSource interface.

type Func 

type Func struct {
	// sync/atomic things
	FuncStats
	// contains filtered or unexported fields
}

Func represents a FuncStats bound to a particular function id, scope, and name. You should create a Func using the Func creation methods (Func/FuncNamed) on a Scope. If you want to manage installation bookkeeping yourself, create a FuncStats directly. Expected Func creation like: 

var mon = monkit.Package()

func MyFunc() {
  f := mon.Func()
  ...
}

func (*Func) FullName 

func (f *Func) FullName() string

FullName returns the name of the function including the package

func (*Func) Id 

func (f *Func) Id() int64

Id returns a unique integer referencing this function

func (*Func) Parents 

func (f *Func) Parents(cb func(f *Func))

Parents will call the given cb with all of the unique Funcs that so far have called this Func.

func (*Func) RemoteTrace 

func (f *Func) RemoteTrace(ctx *context.Context, parentId int64, trace *Trace,
	args ...interface{}) func(*error)

RemoteTrace is like Func.Task, except you can specify the trace and parent span id. Needed for things like the Zipkin plugin.

func (*Func) ResetTrace 

func (f *Func) ResetTrace(ctx *context.Context,
	args ...interface{}) func(*error)

ResetTrace is like Func.Task, except it always creates a new Trace.

func (*Func) Scope 

func (f *Func) Scope() *Scope

Scope references the Scope this Func is bound to

func (*Func) ShortName 

func (f *Func) ShortName() string

ShortName returns the name of the function within the package

func (*Func) Task 

func (f *Func) Task(ctx *context.Context, args ...interface{}) func(*error)

Task returns a new Task for use on this Func. It also adds a new Span to the given ctx during execution. 

var mon = monkit.Package()

func MyFunc(ctx context.Context, arg1, arg2 string) (err error) {
  f := mon.Func()
  defer f.Task(&ctx, arg1, arg2)(&err)
  ...
}

It's more expected for you to use mon.Task directly. See RemoteTrace or ResetTrace if you want greater control over creating new traces.

type FuncStats 

type FuncStats struct {
	// contains filtered or unexported fields
}

FuncStats keeps track of statistics about a possible function's execution. Should be created with NewFuncStats, though expected creation is through a Func object: 

var mon = monkit.Package()

func MyFunc() {
  f := mon.Func()
  ...
}

func NewFuncStats 

func NewFuncStats(key SeriesKey) (f *FuncStats)

NewFuncStats creates a FuncStats

func (*FuncStats) Current 

func (f *FuncStats) Current() int64

Current returns how many concurrent instances of this function are currently being observed.

func (*FuncStats) Errors 

func (f *FuncStats) Errors() (rv map[string]int64)

Errors returns the number of errors observed by error type. The error type is determined by handlers from AddErrorNameHandler, or a default that works with most error types.

func (*FuncStats) FailureTimes 

func (f *FuncStats) FailureTimes() *DurationDist

FailureTimes returns a DurationDist of failures (includes panics and errors)

func (*FuncStats) Highwater 

func (f *FuncStats) Highwater() int64

Highwater returns the highest value Current() would ever return.

func (*FuncStats) Observe 

func (f *FuncStats) Observe() func(errptr *error)

Observe starts the stopwatch for observing this function and returns a function to be called at the end of the function execution. Expected usage like: 

func MyFunc() (err error) {
  defer funcStats.Observe()(&err)
  ...
}

func (*FuncStats) Panics 

func (f *FuncStats) Panics() (rv int64)

Panics returns the number of panics that have been observed

func (*FuncStats) Reset 

func (f *FuncStats) Reset()

Reset resets all recorded data.

func (*FuncStats) Stats 

func (f *FuncStats) Stats(cb func(key SeriesKey, field string, val float64))

Stats implements the StatSource interface

func (*FuncStats) Success 

func (f *FuncStats) Success() (rv int64)

Success returns the number of successes that have been observed

func (*FuncStats) SuccessTimes 

func (f *FuncStats) SuccessTimes() *DurationDist

SuccessTimes returns a DurationDist of successes

type IntDist 

type IntDist struct {
	// Low and High are the lowest and highest values observed since
	// construction or the last reset.
	Low, High int64

	// Recent is the last observed value.
	Recent int64

	// Count is the number of observed values since construction or the last
	// reset.
	Count int64

	// Sum is the sum of all the observed values since construction or the last
	// reset.
	Sum int64
	// contains filtered or unexported fields
}

IntDist keeps statistics about values such as low/high/recent/average/quantiles. Not threadsafe. Construct with NewIntDist(). Fields are expected to be read from but not written to.

func NewIntDist 

func NewIntDist(key SeriesKey) (d *IntDist)

NewIntDist creates a distribution of int64s.

func (*IntDist) Copy 

func (d *IntDist) Copy() *IntDist

Copy returns a full copy of the entire distribution.

func (*IntDist) FullAverage 

func (d *IntDist) FullAverage() int64

FullAverage calculates and returns the average of all inserted values.

func (*IntDist) Insert 

func (d *IntDist) Insert(val int64)

Insert adds a value to the distribution, updating appropriate values.

func (*IntDist) Query 

func (d *IntDist) Query(quantile float64) int64

Query will return the approximate value at the given quantile from the reservoir, where 0 <= quantile <= 1.

func (*IntDist) ReservoirAverage 

func (d *IntDist) ReservoirAverage() int64

ReservoirAverage calculates the average of the current reservoir.

func (*IntDist) Reset 

func (d *IntDist) Reset()

func (*IntDist) Stats 

func (d *IntDist) Stats(cb func(key SeriesKey, field string, val float64))

type IntVal 

type IntVal struct {
	// contains filtered or unexported fields
}

IntVal is a convenience wrapper around an IntDist. Constructed using NewIntVal, though its expected usage is like: 

var mon = monkit.Package()

func MyFunc() {
  ...
  mon.IntVal("size").Observe(val)
  ...
}

func NewIntVal 

func NewIntVal(key SeriesKey) (v *IntVal)

NewIntVal creates an IntVal

func (*IntVal) Observe 

func (v *IntVal) Observe(val int64)

Observe observes an integer value

func (*IntVal) Quantile 

func (v *IntVal) Quantile(quantile float64) (rv int64)

Quantile returns an estimate of the requested quantile of observed values. 0 <= quantile <= 1

func (*IntVal) Stats 

func (v *IntVal) Stats(cb func(key SeriesKey, field string, val float64))

Stats implements the StatSource interface.

type Meter 

type Meter struct {
	// contains filtered or unexported fields
}

Meter keeps track of events and their rates over time. Implements the StatSource interface. You should construct using NewMeter, though expected usage is like: 

var (
  mon   = monkit.Package()
  meter = mon.Meter("meter")
)

func MyFunc() {
  ...
  meter.Mark(4) // 4 things happened
  ...
}

func NewMeter 

func NewMeter(key SeriesKey) *Meter

NewMeter constructs a Meter

func (*Meter) Mark 

func (e *Meter) Mark(amount int)

Mark marks amount events occurring in the current time window.

func (*Meter) Mark64 

func (e *Meter) Mark64(amount int64)

Mark64 marks amount events occurring in the current time window (int64 version).

func (*Meter) Rate 

func (e *Meter) Rate() float64

Rate returns the rate over the internal sliding window

func (*Meter) Reset 

func (e *Meter) Reset(new_total int64)

Reset resets all internal state.

Useful when monitoring a counter that has overflowed.

func (*Meter) SetTotal 

func (e *Meter) SetTotal(total int64)

SetTotal sets the initial total count of the meter.

func (*Meter) Stats 

func (e *Meter) Stats(cb func(key SeriesKey, field string, val float64))

Stats implements the StatSource interface

func (*Meter) Total 

func (e *Meter) Total() float64

Total returns the total over the internal sliding window

type Registry 

type Registry struct {
	// contains filtered or unexported fields
}

Registry encapsulates all of the top-level state for a monitoring system. In general, only the Default registry is ever used.

func NewRegistry 

func NewRegistry() *Registry

NewRegistry creates a NewRegistry, though you almost certainly just want to use Default.

func (*Registry) AllSpans 

func (r *Registry) AllSpans(cb func(s *Span))

AllSpans calls 'cb' on all currently known Spans. See also RootSpans.

func (*Registry) Funcs 

func (r *Registry) Funcs(cb func(f *Func))

Funcs calls 'cb' on all currently known Funcs.

func (*Registry) ObserveTraces 

func (r *Registry) ObserveTraces(cb func(*Trace)) (cancel func())

ObserveTraces lets you observe all traces flowing through the system. The passed in callback 'cb' will be called for every new trace as soon as it starts, until the returned cancel method is called. Note: this only applies to all new traces. If you want to find existing or running traces, please pull them off of live RootSpans.

func (*Registry) Package 

func (r *Registry) Package() *Scope

Package creates a new monitoring Scope, named after the top level package. It's expected that you'll have something like 

var mon = monkit.Package()

at the top of each package.

func (*Registry) RootSpans 

func (r *Registry) RootSpans(cb func(s *Span))

RootSpans will call 'cb' on all currently executing Spans with no live or reachable parent. See also AllSpans.

func (*Registry) ScopeNamed 

func (r *Registry) ScopeNamed(name string) *Scope

ScopeNamed is like Package, but lets you choose the name.

func (*Registry) Scopes 

func (r *Registry) Scopes(cb func(s *Scope))

Scopes calls 'cb' on all currently known Scopes.

func (*Registry) Stats 

func (r *Registry) Stats(cb func(key SeriesKey, field string, val float64))

Stats implements the StatSource interface.

func (*Registry) WithTransformers added in v3.0.15 

func (r *Registry) WithTransformers(t ...CallbackTransformer) *Registry

WithTransformers returns a copy of Registry but with the additional CallbackTransformers applied to the Stats method.

type RunningTimer 

type RunningTimer struct {
	// contains filtered or unexported fields
}

RunningTimer should be constructed from a Timer.

func (*RunningTimer) Elapsed 

func (r *RunningTimer) Elapsed() time.Duration

Elapsed just returns the amount of time since the timer started

func (*RunningTimer) Stop 

func (r *RunningTimer) Stop() time.Duration

Stop stops the timer, adds the duration to the statistics information, and returns the elapsed time.

type Scope 

type Scope struct {
	// contains filtered or unexported fields
}

Scope represents a named collection of StatSources. Scopes are constructed through Registries.

func Package 

func Package() *Scope

Package is just a wrapper around Default.Package

func ScopeNamed 

func ScopeNamed(name string) *Scope

ScopeNamed is just a wrapper around Default.ScopeNamed

func (*Scope) BoolVal 

func (s *Scope) BoolVal(name string, tags ...SeriesTag) *BoolVal

BoolVal retrieves or creates a BoolVal after the given name.

func (*Scope) BoolValf 

func (s *Scope) BoolValf(template string, args ...interface{}) *BoolVal

BoolValf retrieves or creates a BoolVal after the given printf-formatted name.

func (*Scope) Chain 

func (s *Scope) Chain(source StatSource)

Chain registers a full StatSource as the given name in the Scope's StatSource table.

func (*Scope) Counter 

func (s *Scope) Counter(name string, tags ...SeriesTag) *Counter

Counter retrieves or creates a Counter after the given name.

func (*Scope) DiffMeter 

func (s *Scope) DiffMeter(name string, m1, m2 *Meter, tags ...SeriesTag)

DiffMeter retrieves or creates a DiffMeter after the given name and two submeters.

func (*Scope) DurationVal added in v3.0.7 

func (s *Scope) DurationVal(name string, tags ...SeriesTag) *DurationVal

DurationVal retrieves or creates a DurationVal after the given name.

func (*Scope) Event 

func (s *Scope) Event(name string, tags ...SeriesTag)

Event retrieves or creates a Meter named after the given name and then calls Mark(1) on that meter.

func (*Scope) FloatVal 

func (s *Scope) FloatVal(name string, tags ...SeriesTag) *FloatVal

FloatVal retrieves or creates a FloatVal after the given name.

func (*Scope) FloatValf 

func (s *Scope) FloatValf(template string, args ...interface{}) *FloatVal

FloatValf retrieves or creates a FloatVal after the given printf-formatted name.

func (*Scope) Func 

func (s *Scope) Func() *Func

Func retrieves or creates a Func named after the currently executing function name (via runtime.Caller. See FuncNamed to choose your own name.

func (*Scope) FuncNamed 

func (s *Scope) FuncNamed(name string, tags ...SeriesTag) *Func

FuncNamed retrieves or creates a Func named using the given name and SeriesTags. See Func() for automatic name determination.

Each unique combination of keys/values in each SeriesTag will result in a unique Func. SeriesTags are not sorted, so keep the order consistent to avoid unintentionally creating new unique Funcs.

func (*Scope) Funcs 

func (s *Scope) Funcs(cb func(f *Func))

Funcs calls 'cb' for all Funcs registered on this Scope.

func (*Scope) Gauge 

func (s *Scope) Gauge(name string, cb func() float64)

Gauge registers a callback that returns a float as the given name in the Scope's StatSource table.

func (*Scope) IntVal 

func (s *Scope) IntVal(name string, tags ...SeriesTag) *IntVal

IntVal retrieves or creates an IntVal after the given name.

func (*Scope) IntValf 

func (s *Scope) IntValf(template string, args ...interface{}) *IntVal

IntValf retrieves or creates an IntVal after the given printf-formatted name.

func (*Scope) Meter 

func (s *Scope) Meter(name string, tags ...SeriesTag) *Meter

Meter retrieves or creates a Meter named after the given name. See Event.

func (*Scope) Name 

func (s *Scope) Name() string

Name returns the name of the Scope, often the Package name.

func (*Scope) Stats 

func (s *Scope) Stats(cb func(key SeriesKey, field string, val float64))

Stats implements the StatSource interface.

func (*Scope) StructVal 

func (s *Scope) StructVal(name string, tags ...SeriesTag) *StructVal

StructVal retrieves or creates a StructVal after the given name.

func (*Scope) Task 

func (s *Scope) Task(tags ...SeriesTag) Task

Task returns a new Task for use, creating an associated Func if necessary. It also adds a new Span to the given ctx during execution. Expected usage like: 

var mon = monkit.Package()

func MyFunc(ctx context.Context, arg1, arg2 string) (err error) {
  defer mon.Task()(&ctx, arg1, arg2)(&err)
  ...
}

or 

var (
  mon = monkit.Package()
  funcTask = mon.Task()
)

func MyFunc(ctx context.Context, arg1, arg2 string) (err error) {
  defer funcTask(&ctx, arg1, arg2)(&err)
  ...
}

Task allows you to include SeriesTags. WARNING: Each unique tag key/value combination creates a unique Func and a unique series. SeriesTags should only be used for low-cardinality values that you intentionally wish to result in a unique series. Example: 

func MyFunc(ctx context.Context, arg1, arg2 string) (err error) {
  defer mon.Task(monkit.NewSeriesTag("key1", "val1"))(&ctx)(&err)
  ...
}

Task uses runtime.Caller to determine the associated Func name. See TaskNamed if you want to supply your own name. See Func.Task if you already have a Func.

If you want to control Trace creation, see Func.ResetTrace and Func.RemoteTrace

func (*Scope) TaskNamed 

func (s *Scope) TaskNamed(name string, tags ...SeriesTag) Task

TaskNamed is like Task except you can choose the name of the associated Func.

You may also include any SeriesTags which should be included with the Task.

func (*Scope) Timer 

func (s *Scope) Timer(name string, tags ...SeriesTag) *Timer

Timer retrieves or creates a Timer after the given name.

type SeriesKey 

type SeriesKey struct {
	Measurement string
	Tags        *TagSet
}

SeriesKey represents an individual time series for monkit to output.

func NewSeriesKey 

func NewSeriesKey(measurement string) SeriesKey

NewSeriesKey constructs a new series with the minimal fields.

func (SeriesKey) String 

func (s SeriesKey) String() string

String returns a string representation of the series. For example, it returns something like `measurement,tag0=val0,tag1=val1`.

func (SeriesKey) WithField 

func (s SeriesKey) WithField(field string) string

func (SeriesKey) WithTag 

func (s SeriesKey) WithTag(key, value string) SeriesKey

WithTag returns a copy of the SeriesKey with the tag set

func (SeriesKey) WithTags added in v3.0.11 

func (s SeriesKey) WithTags(tags ...SeriesTag) SeriesKey

WithTags returns a copy of the SeriesKey with all of the tags set

type SeriesTag added in v3.0.6 

type SeriesTag struct {
	Key, Val string
}

SeriesTag is a key/value pair. When used with a measurement name, each set of unique key/value pairs represents a new unique series.

func NewSeriesTag added in v3.0.6 

func NewSeriesTag(key, val string) SeriesTag

NewTag creates a new tag

type Span 

type Span struct {
	context.Context
	// contains filtered or unexported fields
}

Span represents a 'span' of execution. A span is analogous to a stack frame. Spans are constructed as a side-effect of Tasks.

func SpanFromCtx 

func SpanFromCtx(ctx context.Context) *Span

SpanFromCtx loads the current Span from the given context. This assumes the context already had a Span created through a Task.

func (*Span) Annotate 

func (s *Span) Annotate(name, val string)

Annotate adds an annotation to the existing Span.

func (*Span) Annotations 

func (s *Span) Annotations() []Annotation

Annotations returns any added annotations created through the Span Annotate method

func (*Span) Args 

func (s *Span) Args() (rv []string)

Args returns the list of strings associated with the args given to the Task that created this Span.

func (*Span) Children 

func (s *Span) Children(cb func(s *Span))

Children returns all known running child Spans.

func (*Span) Duration 

func (s *Span) Duration() time.Duration

Duration returns the current amount of time the Span has been running

func (*Span) Func 

func (s *Span) Func() *Func

Func returns the Func that kicked off this Span.

func (*Span) Id 

func (s *Span) Id() int64

Id returns the Span id.

func (*Span) Orphaned 

func (s *Span) Orphaned() (rv bool)

Orphaned returns true if the Parent span ended before this Span did.

func (*Span) ParentId added in v3.0.18 

func (s *Span) ParentId() (int64, bool)

ParentId returns the id of the parent Span, if it has a parent.

func (*Span) Start 

func (s *Span) Start() time.Time

Start returns the time the Span started.

func (*Span) String 

func (s *Span) String() string

String implements context.Context

func (*Span) Trace 

func (s *Span) Trace() *Trace

Trace returns the Trace this Span is associated with.

func (*Span) Value 

func (s *Span) Value(key interface{}) interface{}

Value implements context.Context

type SpanCtxObserver 

type SpanCtxObserver interface {
	// Start is called when a Span starts. Start should return the context
	// this span should use going forward. ctx is the context it is currently
	// using.
	Start(ctx context.Context, s *Span) context.Context

	// Finish is called when a Span finishes, along with an error if any, whether
	// or not it panicked, and what time it finished.
	Finish(ctx context.Context, s *Span, err error, panicked bool, finish time.Time)
}

SpanCtxObserver is the interface plugins must implement if they want to observe all spans on a given trace as they happen, or add to contexts as they pass through mon.Task()(&ctx)(&err) calls.

type SpanObserver 

type SpanObserver interface {
	// Start is called when a Span starts
	Start(s *Span)

	// Finish is called when a Span finishes, along with an error if any, whether
	// or not it panicked, and what time it finished.
	Finish(s *Span, err error, panicked bool, finish time.Time)
}

SpanObserver is the interface plugins must implement if they want to observe all spans on a given trace as they happen.

type StatSource 

type StatSource interface {
	Stats(cb func(key SeriesKey, field string, val float64))
}

StatSource represents anything that can return named floating point values.

func StatSourceFromStruct 

func StatSourceFromStruct(key SeriesKey, structData interface{}) StatSource

StatSourceFromStruct uses the reflect package to implement the Stats call across all float64-castable and bool-castable fields of the struct.

func TransformStatSource added in v3.0.15 

func TransformStatSource(s StatSource, transformers ...CallbackTransformer) StatSource

TransformStatSource will make sure that a StatSource has the provided CallbackTransformers applied to callbacks given to the StatSource.

type StatSourceFunc 

type StatSourceFunc func(cb func(key SeriesKey, field string, val float64))

func (StatSourceFunc) Stats 

func (f StatSourceFunc) Stats(cb func(key SeriesKey, field string, val float64))

type StructVal 

type StructVal struct {
	// contains filtered or unexported fields
}

StructVal keeps track of a structure of data. Constructed using NewStructVal, though its expected usage is like: 

var mon = monkit.Package()

func MyFunc() {
  ...
  mon.StructVal("stats").Observe(stats)
  ...
}

func NewStructVal 

func NewStructVal(key SeriesKey) *StructVal

NewStructVal creates a StructVal

func (*StructVal) Observe 

func (v *StructVal) Observe(val interface{})

Observe observes a struct value. Only the fields convertable to float64 will be monitored. A reference to the most recently called Observe value is kept for reading when Stats is called.

func (*StructVal) Stats 

func (v *StructVal) Stats(cb func(key SeriesKey, field string, val float64))

Stats implements the StatSource interface.

type TagSet 

type TagSet struct {
	// contains filtered or unexported fields
}

TagSet is an immutible collection of tag, value pairs.

func (*TagSet) All 

func (t *TagSet) All() map[string]string

All returns a map of all the key/value pairs in the tag set. It should not be modified.

func (*TagSet) Get 

func (t *TagSet) Get(key string) string

Get returns the value associated with the key.

func (*TagSet) Len 

func (t *TagSet) Len() int

Len returns the number of tags in the tag set.

func (*TagSet) Set 

func (t *TagSet) Set(key, value string) *TagSet

Set returns a new tag set with the key associated to the value.

func (*TagSet) SetAll 

func (t *TagSet) SetAll(kvs map[string]string) *TagSet

SetAll returns a new tag set with the key value pairs in the map all set.

func (*TagSet) SetTags added in v3.0.11 

func (t *TagSet) SetTags(tags ...SeriesTag) *TagSet

SetTags returns a new tag set with the keys and values set by the tags slice.

func (*TagSet) String 

func (t *TagSet) String() string

String returns a string form of the tag set suitable for sending to influxdb.

type Task 

type Task func(ctx *context.Context, args ...interface{}) func(*error)

Tasks are created (sometimes implicitly) from Funcs. A Task should be called at the start of a monitored task, and its return value should be called at the stop of said task.

func (Task) Func 

func (f Task) Func() (out *Func)

Func returns the Func associated with the Task

type Timer 

type Timer struct {
	// contains filtered or unexported fields
}

Timer is a threadsafe convenience wrapper around a DurationDist. You should construct with NewTimer(), though the expected usage is from a Scope like so: 

var mon = monkit.Package()

func MyFunc() {
  ...
  timer := mon.Timer("event")
  // perform event
  timer.Stop()
  ...
}

Timers implement StatSource.

func NewTimer 

func NewTimer(key SeriesKey) *Timer

NewTimer constructs a new Timer.

func (*Timer) Start 

func (t *Timer) Start() *RunningTimer

Start constructs a RunningTimer

func (*Timer) Stats 

func (t *Timer) Stats(cb func(key SeriesKey, field string, val float64))

Stats implements the StatSource interface

func (*Timer) Values 

func (t *Timer) Values() *DurationDist

Values returns the main timer values

type Trace 

type Trace struct {
	// contains filtered or unexported fields
}

Trace represents a 'trace' of execution. A 'trace' is the collection of all of the 'spans' kicked off from the same root execution context. A trace is a concurrency-supporting analog of a stack trace, where a span is somewhat like a stack frame.

func NewTrace 

func NewTrace(id int64) *Trace

NewTrace creates a new Trace.

func (*Trace) Get 

func (t *Trace) Get(key interface{}) (val interface{})

Get returns a value associated with a key on a trace. See Set.

func (*Trace) GetAll added in v3.0.7 

func (t *Trace) GetAll() (val map[interface{}]interface{})

GetAll returns values associated with a trace. See SetAll.

func (*Trace) Id 

func (t *Trace) Id() int64

Id returns the id of the Trace

func (*Trace) ObserveSpans 

func (t *Trace) ObserveSpans(observer SpanObserver) (cancel func())

ObserveSpans lets you register a SpanObserver for all future Spans on the Trace. The returned cancel method will remove your observer from the trace.

func (*Trace) ObserveSpansCtx 

func (t *Trace) ObserveSpansCtx(observer SpanCtxObserver) (cancel func())

ObserveSpansCtx lets you register a SpanCtxObserver for all future Spans on the Trace. The returned cancel method will remove your observer from the trace.

func (*Trace) Set 

func (t *Trace) Set(key, val interface{})

Set sets a value associated with a key on a trace. See Get.

func (*Trace) Spans added in v3.0.3 

func (t *Trace) Spans() int64

Spans returns the number of spans currently associated with the Trace.

Source Files

View all Source files

Directories

Path Synopsis
examples
minimal
wasm
Package http provides helper to propagate monkit traces via HTTP calls.
 Package http provides helper to propagate monkit traces via HTTP calls.
internal
testpkg1

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.