gostatsd

package module
v0.0.0-...-170def7 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 29, 2018 License: MIT Imports: 8 Imported by: 0

README

gostatsd

Godoc Build Status Coverage Status GitHub tag Docker Pulls Docker Stars MicroBadger Layers Size Go Report Card license

An implementation of Etsy's statsd in Go, based on original code from @kisielk.

The project provides both a server called "gostatsd" which works much like Etsy's version, but also provides a library for developing customized servers.

Backends are pluggable and only need to support the backend interface.

Being written in Go, it is able to use all cores which makes it easy to scale up the server based on load. The server can also be run HA and be scaled out, see Load balancing and scaling out.

Building the server

Gostatsd currently targets Go 1.10.2. There are no known hard dependencies in the code beween 1.9 and 1.10.2, but some may be introduced in future.

From the gostatsd directory run make build. The binary will be built in build/bin/<arch>/gostatsd.

You will need to install the build dependencies by running make setup in the gostatsd directory. This must be done before the first build, and again if the dependencies change.

If you are unable to build gostatsd please try running make setup again before reporting a bug.

Running the server

gostatsd --help gives a complete description of available options and their defaults. You can use make run to run the server with just the stdout backend to display info on screen. You can also run through docker by running make run-docker which will use docker-compose to run gostatsd with a graphite backend and a grafana dashboard.

While not generally tested on Windows, it should work. Maximum throughput is likely to be better on a linux system, however.

Configuring backends and cloud providers

Backends and cloud providers are configured using toml, json or yaml configuration file passed via the --config-path flag. For all configuration options see source code of the backends you are interested in. Configuration file might look like this:

[graphite]
	address = "192.168.99.100:2003"

[datadog]
	api_key = "my-secret-key" # Datadog API key required.

[statsdaemon]
	address = "docker.local:8125"
	disable_tags = false

[aws]
	max_retries = 4

Configuring timer sub-metrics

By default, timer metrics will result in aggregated metrics of the form (exact name varies by backend):

<base>.Count
<base>.CountPerSecond
<base>.Mean
<base>.Median
<base>.Lower
<base>.Upper
<base>.StdDev
<base>.Sum
<base>.SumSquares

In addition, the following aggregated metrics will be emitted for each configured percentile:

<base>.Count_XX
<base>.Mean_XX
<base>.Sum_XX
<base>.SumSquares_XX
<base>.Upper_XX - for positive only
<base>.Lower_-XX - for negative only

These can be controlled through the disabled-sub-metrics configuration section:

[disabled-sub-metrics]
# Regular metrics
count=false
count-per-second=false
mean=false
median=false
lower=false
upper=false
stddev=false
sum=false
sum-squares=false

# Percentile metrics
count-pct=false
mean-pct=false
sum-pct=false
sum-squares-pct=false
lower-pct=false
upper-pct=false

By default (for compatibility), they are all false and the metrics will be emitted.

Sending metrics

The server listens for UDP packets on the address given by the --metrics-addr flag, aggregates them, then sends them to the backend servers given by the --backends flag (space separated list of backend names).

Currently supported backends are:

  • graphite
  • datadog
  • statsdaemon
  • stdout
  • cloudwatch

The format of each metric is:

<bucket name>:<value>|<type>\n
  • <bucket name> is a string like abc.def.g, just like a graphite bucket name
  • <value> is a string representation of a floating point number
  • <type> is one of c, g, or ms for "counter", "gauge", and "timer" respectively.

A single packet can contain multiple metrics, each ending with a newline.

Optionally, gostatsd supports sample rates (for simple counters, and for timer counters) and tags (unused):

  • <bucket name>:<value>|c|@<sample rate>\n where sample rate is a float between 0 and 1
  • <bucket name>:<value>|c|@<sample rate>|#<tags>\n where tags is a comma separated list of tags
  • or <bucket name>:<value>|<type>|#<tags>\n where tags is a comma separated list of tags

Tags format is: simple or key:value.

A simple way to test your installation or send metrics from a script is to use echo and the netcat utility nc:

echo 'abc.def.g:10|c' | nc -w1 -u localhost 8125

Monitoring

Many metrics for the internal processes are emitted. See METRICS.md for details. Go expvar is also exposed if the --profile flag is used.

Memory allocation for read buffers

By default gostatsd will batch read multiple packets to optimise read performance. The amount of memory allocated for these read buffers is determined by the config options:

max-readers * receive-batch-size * 64KB (max packet size)

The metric avg_packets_in_batch can be used to track the average number of datagrams received per batch, and the --receive-batch-size flag used to tune it. There may be some benefit to tuning the --max-readers flag as well.

Using the library

In your source code:

import "github.com/atlassian/gostatsd/pkg/statsd"

Documentation can be found via go doc github.com/atlassian/gostatsd/pkg/statsd or at https://godoc.org/github.com/atlassian/gostatsd/pkg/statsd

Contributors

Pull requests, issues and comments welcome. For pull requests:

  • Add tests for new features and bug fixes
  • Follow the existing style
  • Separate unrelated changes into multiple pull requests

See the existing issues for things to start contributing.

For bigger changes, make sure you start a discussion first by creating an issue and explaining the intended change.

Atlassian requires contributors to sign a Contributor License Agreement, known as a CLA. This serves as a record stating that the contributor is entitled to contribute the code/documentation/translation to the project and is willing to have it used in distributions and derivative works (or is willing to transfer ownership).

Prior to accepting your contributions we ask that you please follow the appropriate link below to digitally sign the CLA. The Corporate CLA is for those who are contributing as a member of an organization and the individual CLA is for those contributing as an individual.

License

Copyright (c) 2012 Kamil Kisiel. Copyright @ 2016-2017 Atlassian Pty Ltd and others.

Licensed under the MIT license. See LICENSE file.

Documentation

Index

Constants

View Source 
const StatsdSourceID = "s"

StatsdSourceID stores the key used to tag metrics with the origin IP address. Should be short to avoid extra hashing and memory overhead for map operations.

Variables

This section is empty.

Functions

func NormalizeTagKey 

func NormalizeTagKey(key string) string

NormalizeTagKey cleans up the key of a tag.

Types

type AggregatedMetrics 

type AggregatedMetrics interface {
	MetricsName() string
	Delete(string)
	DeleteChild(string, string)
	HasChildren(string) bool
}

AggregatedMetrics is an interface for aggregated metrics.

type AlertType 

type AlertType byte

AlertType is the type of alert. 

const (
	// AlertInfo is alert level "info".
	AlertInfo AlertType = iota // Must be zero to work as default
	// AlertWarning is alert level "warning".
	AlertWarning
	// AlertError is alert level "error".
	AlertError
	// AlertSuccess is alert level "success".
	AlertSuccess
)

func (AlertType) String 

func (a AlertType) String() string

func (AlertType) StringWithEmptyDefault 

func (a AlertType) StringWithEmptyDefault() string

StringWithEmptyDefault returns empty string for default alert type.

type Backend 

type Backend interface {
	// Name returns the name of the backend.
	Name() string
	// SendMetricsAsync flushes the metrics to the backend, preparing payload synchronously but doing the send asynchronously.
	// Must not read/write MetricMap asynchronously.
	SendMetricsAsync(context.Context, *MetricMap, SendCallback)
	// SendEvent sends event to the backend.
	SendEvent(context.Context, *Event) error
}

Backend represents a backend.

type BackendFactory 

type BackendFactory func(*viper.Viper) (Backend, error)

BackendFactory is a function that returns a Backend.

type CloudProvider 

type CloudProvider interface {
	// Name returns the name of the cloud provider.
	Name() string
	// Instance returns instances details from the cloud provider.
	// ip -> nil pointer if instance was not found.
	// map is returned even in case of errors because it may contain partial data.
	Instance(context.Context, ...IP) (map[IP]*Instance, error)
	// MaxInstancesBatch returns maximum number of instances that could be requested via the Instance method.
	MaxInstancesBatch() int
	// SelfIP returns host's IPv4 address.
	SelfIP() (IP, error)
	// EstimatedTags returns a guess of how many tags are likely to be added by the CloudProvider
	EstimatedTags() int
}

CloudProvider represents a cloud provider.

type CloudProviderFactory 

type CloudProviderFactory func(v *viper.Viper, logger logrus.FieldLogger) (CloudProvider, error)

CloudProviderFactory is a function that returns a CloudProvider.

type Counter 

type Counter struct {
	PerSecond float64  // The calculated per second rate
	Value     int64    // The numeric value of the metric
	Timestamp Nanotime // Last time value was updated
	Hostname  string   // Hostname of the source of the metric
	Tags      Tags     // The tags for the counter
}

Counter is used for storing aggregated values for counters.

func NewCounter 

func NewCounter(timestamp Nanotime, value int64, hostname string, tags Tags) Counter

NewCounter initialises a new counter.

type Counters 

type Counters map[string]map[string]Counter

Counters stores a map of counters by tags.

func (Counters) Delete 

func (c Counters) Delete(k string)

Delete deletes the metrics from the collection.

func (Counters) DeleteChild 

func (c Counters) DeleteChild(k, t string)

DeleteChild deletes the metrics from the collection for the given tags.

func (Counters) Each 

func (c Counters) Each(f func(string, string, Counter))

Each iterates over each counter.

func (Counters) HasChildren 

func (c Counters) HasChildren(k string) bool

HasChildren returns whether there are more children nested under the key.

func (Counters) MetricsName 

func (c Counters) MetricsName() string

MetricsName returns the name of the aggregated metrics collection.

type Event 

type Event struct {
	// Title of the event.
	Title string
	// Text of the event. Supports line breaks.
	Text string
	// DateHappened of the event. Unix epoch timestamp. Default is now when not specified in incoming metric.
	DateHappened int64
	// Hostname of the event. This field contains information that is received in the body of the event (optional).
	Hostname string
	// AggregationKey of the event, to group it with some other events.
	AggregationKey string
	// SourceTypeName of the event.
	SourceTypeName string
	// Tags of the event.
	Tags Tags
	// IP of the source of the metric
	SourceIP IP
	// Priority of the event.
	Priority Priority
	// AlertType of the event.
	AlertType AlertType
}

Event represents an event, described at http://docs.datadoghq.com/guides/dogstatsd/

type Events 

type Events []Event

Events represents a list of events.

type Gauge 

type Gauge struct {
	Value     float64  // The numeric value of the metric
	Timestamp Nanotime // Last time value was updated
	Hostname  string   // Hostname of the source of the metric
	Tags      Tags     // The tags for the gauge
}

Gauge is used for storing aggregated values for gauges.

func NewGauge 

func NewGauge(timestamp Nanotime, value float64, hostname string, tags Tags) Gauge

NewGauge initialises a new gauge.

type Gauges 

type Gauges map[string]map[string]Gauge

Gauges stores a map of gauges by tags.

func (Gauges) Delete 

func (g Gauges) Delete(k string)

Delete deletes the metrics from the collection.

func (Gauges) DeleteChild 

func (g Gauges) DeleteChild(k, t string)

DeleteChild deletes the metrics from the collection for the given tags.

func (Gauges) Each 

func (g Gauges) Each(f func(string, string, Gauge))

Each iterates over each gauge.

func (Gauges) HasChildren 

func (g Gauges) HasChildren(k string) bool

HasChildren returns whether there are more children nested under the key.

func (Gauges) MetricsName 

func (g Gauges) MetricsName() string

MetricsName returns the name of the aggregated metrics collection.

type IP 

type IP string

IP is a v4/v6 IP address. We do not use net.IP because it will involve conversion to string and back several times. 

const UnknownIP IP = ""

UnknownIP is an IP of an unknown source.

type Instance 

type Instance struct {
	ID   string
	Tags Tags
}

Instance represents a cloud instance.

type Metric 

type Metric struct {
	Name        string     // The name of the metric
	Value       float64    // The numeric value of the metric
	Rate        float64    // The sampling rate of the metric
	Tags        Tags       // The tags for the metric
	TagsKey     string     // The tags rendered as a string to uniquely identify the tagset in a map
	StringValue string     // The string value for some metrics e.g. Set
	Hostname    string     // Hostname of the source of the metric
	SourceIP    IP         // IP of the source of the metric
	Type        MetricType // The type of metric
	DoneFunc    func()     // Returns the metric to the pool. May be nil. Call Metric.Done(), not this.
}

Metric represents a single data collected datapoint.

func (*Metric) Bucket 

func (m *Metric) Bucket(max int) int

Bucket will pick a distribution bucket for this metric to land in. max is exclusive.

func (*Metric) Done 

func (m *Metric) Done()

Done invokes DoneFunc if it's set, returning the metric to the pool.

func (*Metric) Reset 

func (m *Metric) Reset()

Reset is used to reset a metric to as clean state, called on re-use from the pool.

func (*Metric) String 

func (m *Metric) String() string

type MetricMap 

type MetricMap struct {
	Counters Counters
	Timers   Timers
	Gauges   Gauges
	Sets     Sets
}

MetricMap is used for storing aggregated Metric values. The keys of each map are metric names.

func (*MetricMap) String 

func (m *MetricMap) String() string

type MetricType 

type MetricType byte

MetricType is an enumeration of all the possible types of Metric. 

const (

	// COUNTER is statsd counter type
	COUNTER MetricType = iota
	// TIMER is statsd timer type
	TIMER
	// GAUGE is statsd gauge type
	GAUGE
	// SET is statsd set type
	SET
)

func (MetricType) String 

func (m MetricType) String() string

type Nanotime 

type Nanotime int64

Nanotime is the number of nanoseconds elapsed since January 1, 1970 UTC. Get the value with time.Now().UnixNano().

type Percentile 

type Percentile struct {
	Float float64
	Str   string
}

Percentile is used to store the aggregation for a percentile.

func (*Percentile) String 

func (p *Percentile) String() string

String returns the string value of a percentile.

type Percentiles 

type Percentiles []Percentile

Percentiles represents an array of percentiles.

func (*Percentiles) Set 

func (p *Percentiles) Set(s string, f float64)

Set append a percentile aggregation to the percentiles.

func (*Percentiles) String 

func (p *Percentiles) String() string

String returns the string value of percentiles.

type Priority 

type Priority byte

Priority of an event. 

const (
	// PriNormal is normal priority.
	PriNormal Priority = iota // Must be zero to work as default
	// PriLow is low priority.
	PriLow
)

func (Priority) String 

func (p Priority) String() string

func (Priority) StringWithEmptyDefault 

func (p Priority) StringWithEmptyDefault() string

StringWithEmptyDefault returns empty string for default priority.

type RunnableBackend 

type RunnableBackend interface {
	Backend
	// Run executes backend send operations. Should be started in a goroutine.
	Run(context.Context)
}

RunnableBackend represents a backend that needs a Run method to be executed to work.

type SendCallback 

type SendCallback func([]error)

SendCallback is called by Backend.SendMetricsAsync() to notify about the result of operation. A list of errors is passed to the callback. It may be empty or contain nil values. Every non-nil value is an error that happened while sending metrics.

type Set 

type Set struct {
	Values    map[string]struct{}
	Timestamp Nanotime // Last time value was updated
	Hostname  string   // Hostname of the source of the metric
	Tags      Tags     // The tags for the set
}

Set is used for storing aggregated values for sets.

func NewSet 

func NewSet(timestamp Nanotime, values map[string]struct{}, hostname string, tags Tags) Set

NewSet initialises a new set.

type Sets 

type Sets map[string]map[string]Set

Sets stores a map of sets by tags.

func (Sets) Delete 

func (s Sets) Delete(k string)

Delete deletes the metrics from the collection.

func (Sets) DeleteChild 

func (s Sets) DeleteChild(k, t string)

DeleteChild deletes the metrics from the collection for the given tags.

func (Sets) Each 

func (s Sets) Each(f func(string, string, Set))

Each iterates over each set.

func (Sets) HasChildren 

func (s Sets) HasChildren(k string) bool

HasChildren returns whether there are more children nested under the key.

func (Sets) MetricsName 

func (s Sets) MetricsName() string

MetricsName returns the name of the aggregated metrics collection.

type Tags 

type Tags []string

Tags represents a list of tags. Tags can be of two forms: 1. "key:value". "value" may contain column(s) as well. 2. "tag". No column. Each tag's key and/or value may contain characters invalid for a particular backend. Backends are expected to handle them appropriately. Different backends may have different sets of valid characters so it is undesirable to have restrictions on the input side.

func (Tags) Concat 

func (tags Tags) Concat(additional Tags) Tags

Concat returns a new Tags with the additional ones added

func (Tags) Copy 

func (tags Tags) Copy() Tags

Copy returns a copy of the Tags

func (Tags) SortedString 

func (tags Tags) SortedString() string

SortedString sorts the tags alphabetically and returns a comma-separated string representation of the tags. Note that this method may mutate the original object.

func (Tags) String 

func (tags Tags) String() string

String returns a comma-separated string representation of the tags.

type Timer 

type Timer struct {
	Count        int         // The number of timers in the series
	SampledCount float64     // Number of timings received, divided by sampling rate
	PerSecond    float64     // The calculated per second rate
	Mean         float64     // The mean time of the series
	Median       float64     // The median time of the series
	Min          float64     // The minimum time of the series
	Max          float64     // The maximum time of the series
	StdDev       float64     // The standard deviation for the series
	Sum          float64     // The sum for the series
	SumSquares   float64     // The sum squares for the series
	Values       []float64   // The numeric value of the metric
	Percentiles  Percentiles // The percentile aggregations of the metric
	Timestamp    Nanotime    // Last time value was updated
	Hostname     string      // Hostname of the source of the metric
	Tags         Tags        // The tags for the timer
}

Timer is used for storing aggregated values for timers.

func NewTimer 

func NewTimer(timestamp Nanotime, values []float64, hostname string, tags Tags) Timer

NewTimer initialises a new timer.

func NewTimerValues 

func NewTimerValues(values []float64) Timer

NewTimerValues initialises a new timer only from Values array

type TimerSubtypes 

type TimerSubtypes struct {
	Lower          bool
	LowerPct       bool // pct
	Upper          bool
	UpperPct       bool // pct
	Count          bool
	CountPct       bool // pct
	CountPerSecond bool
	Mean           bool
	MeanPct        bool // pct
	Median         bool
	StdDev         bool
	Sum            bool
	SumPct         bool // pct
	SumSquares     bool
	SumSquaresPct  bool // pct
}

func DisabledSubMetrics 

func DisabledSubMetrics(viper *viper.Viper) TimerSubtypes

type Timers 

type Timers map[string]map[string]Timer

Timers stores a map of timers by tags.

func (Timers) Delete 

func (t Timers) Delete(k string)

Delete deletes the metrics from the collection.

func (Timers) DeleteChild 

func (t Timers) DeleteChild(k, tags string)

DeleteChild deletes the metrics from the collection for the given tags.

func (Timers) Each 

func (t Timers) Each(f func(string, string, Timer))

Each iterates over each timer.

func (Timers) HasChildren 

func (t Timers) HasChildren(k string) bool

HasChildren returns whether there are more children nested under the key.

func (Timers) MetricsName 

func (t Timers) MetricsName() string

MetricsName returns the name of the aggregated metrics collection.

type Wait 

type Wait func()

Source Files

View all Source files

Directories

Path Synopsis
cmd
cluster
gostatsd
tester
pkg
backends
backends/cloudwatch
backends/datadog
backends/graphite
backends/null
backends/sender
backends/statsdaemon
backends/stdout
cloudproviders
cloudproviders/aws
cluster/nodes
fakesocket
pool
statsd
Package statsd implements functionality for creating servers compatible with the statsd protocol.
 Package statsd implements functionality for creating servers compatible with the statsd protocol.
statser

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.