datadog

README

Datadog Sink

This sink sends Veneur metrics to Datadog.

Configuration

See the various datadog_* keys in example.yaml for all available configuration options.

Status

This sink is stable.

Capabilities

Veneur began life as an alternative implementation of DogStatsD.

Veneur is a DogStatsD implementation that acts as a local collector and — optionally — as an aggregator for some metric types, such that the metrics are global rather than host-local. This is particularly useful for histograms, timers and sets, as in their normal, per-host configuration the percentiles for histograms can be less effective or even meaningless. Per-host unique sets are also often not what's desired.

Global *StatsD installations can be problematic, as they either require client-side or proxy sharding behavior to prevent an instance being a Single Point of Failure (SPoF) for all metrics. Veneur aims to solve this problem. Non-global metrics like counters and gauges are collected by a local Veneur instance and sent to storage at flush time. Global metrics (histograms and sets) are forwarded to a central Veneur instance for aggregation before being sent to storage.

Replacing Datadog Agent

Veneur can act as a replacement for the stock DogStatsD. In our environment we disable the Datadog DogStatsD port by setting use_dogstatsd to false. We run Veneur on an alternate port (8200) so as to not accidentally ingest metrics from legacy StatsD clients and we configure our Datadog agent to use this port by dogstatsd_port to 8200 to match.

How Veneur Is Different Than Official DogStatsD

Veneur is different for a few reasons. They are enumerated here.

Protocol

Veneur adheres to the official DogStatsD datagram format with the exceptions below:

• The tag veneurlocalonly is stripped and influences forwarding behavior, as discussed below.
• The tag veneurglobalonly is stripped and influences forwarding behavior, as discussed below.

Lack of Host Tags for Aggregated Metrics

By definition the hostname is not applicable to global metrics that Veneur processes. Note that if you do include a hostname tag, Veneur will not strip it for you. Veneur will add its own hostname as configured to metrics sent to Datadog.

Metrics

Enabled if datadog_api_hostname and datadog_api_key are set to non-empty values.

• Counters are converted to second-normalized rates with an interval matching the server's interval.
• Gauges are gauges.

The following tags are mapped to Datadog fields as follows:

• The tag host to hostname
• The tag device to device_name

Compressed, Chunked POST

Datadog's API is tuned for small POST bodies from lots of hosts since they work on a per-host basis. Also there are limits on the size of the body that can be POSTed. As a result Veneur chunks metrics in to smaller bits — governed by datadog_flush_max_per_body — and sends them (compressed) concurrently to Datadog. This is essential for reasonable performance as Datadog's API seems to be somewhat O(n) with the size of the body (which is proportional to the number of metrics).

We've found that our hosts generate around 5k metrics and have reasonable performance, so in our case 5k is used as the datadog_flush_max_per_body.

Spans

Enabled if datadog_trace_api_address and datadog_api_key are set to non-empty values.

Spans are sent to Datadog APM. The following rules manage how SSF spans and tags are mapped to Datadog trace fields:

• The SSF field name is mapped to the trace's name field.
• The tag resource is removed and mapped to the trace's resource field.
• The type field is currently hardcoded to "web".
• The SSF field error is mapped to the trace's error field.
• Remaining tags are mapped to the trace's meta dictionary.

Span Retention

Veneur allocates a ring buffer of datadog_span_buffer_size entries which is flushed every interval. Being a ring buffer, the oldest spans will be overwritten until it interval expires and the ring buffer is reset.

Other

As a side-effect of implementing DogStatsD Veneur parses both Service Checks and Events.

Documentation

Index

• type DDEvent
• type DDMetric
• type DDServiceCheck
• type DatadogMetricSink
  • func NewDatadogMetricSink(interval float64, flushMaxPerBody int, hostname string, tags []string, ...) (*DatadogMetricSink, error)
  • func (dd *DatadogMetricSink) Flush(ctx context.Context, interMetrics []samplers.InterMetric) error
  • func (dd *DatadogMetricSink) FlushOtherSamples(ctx context.Context, samples []ssf.SSFSample)
  • func (dd *DatadogMetricSink) Name() string
  • func (dd *DatadogMetricSink) Start(cl *trace.Client) error
• type DatadogSpanSink
  • func NewDatadogSpanSink(address string, bufferSize int, httpClient *http.Client, log *logrus.Logger) (*DatadogSpanSink, error)
  • func (dd *DatadogSpanSink) Flush()
  • func (dd *DatadogSpanSink) Ingest(span *ssf.SSFSpan) error
  • func (dd *DatadogSpanSink) Name() string
  • func (dd *DatadogSpanSink) Start(cl *trace.Client) error
• type DatadogTraceSpan

Types

type DDEvent

type DDEvent struct {
	Title       string   `json:"msg_title"`
	Text        string   `json:"msg_text"`
	Timestamp   int64    `json:"timestamp,omitempty"` // represented as a unix epoch
	Hostname    string   `json:"host,omitempty"`
	Aggregation string   `json:"aggregation_key,omitempty"`
	Priority    string   `json:"priority,omitempty"`
	Source      string   `json:"source_type_name,omitempty"`
	AlertType   string   `json:"alert_type,omitempty"`
	Tags        []string `json:"tags,omitempty"`
}

DDEvent represents the structure of datadog's undocumented /intake endpoint

type DDMetric

type DDMetric struct {
	Name       string        `json:"metric"`
	Value      [1][2]float64 `json:"points"`
	Tags       []string      `json:"tags,omitempty"`
	MetricType string        `json:"type"`
	Hostname   string        `json:"host,omitempty"`
	DeviceName string        `json:"device_name,omitempty"`
	Interval   int32         `json:"interval,omitempty"`
}

DDMetric is a data structure that represents the JSON that Datadog wants when posting to the API

type DDServiceCheck

type DDServiceCheck struct {
	Name      string   `json:"check"`
	Status    int      `json:"status"`
	Hostname  string   `json:"host_name"`
	Timestamp int64    `json:"timestamp,omitempty"` // represented as a unix epoch
	Tags      []string `json:"tags,omitempty"`
	Message   string   `json:"message,omitempty"`
}

DDServiceCheck is a representation of the service check.

type DatadogMetricSink

type DatadogMetricSink struct {
	HTTPClient *http.Client
	APIKey     string
	DDHostname string
	// contains filtered or unexported fields
}

func NewDatadogMetricSink

func NewDatadogMetricSink(interval float64, flushMaxPerBody int, hostname string, tags []string, ddHostname string, apiKey string, httpClient *http.Client, log *logrus.Logger) (*DatadogMetricSink, error)

NewDatadogMetricSink creates a new Datadog sink for trace spans.

func (*DatadogMetricSink) Flush

func (dd *DatadogMetricSink) Flush(ctx context.Context, interMetrics []samplers.InterMetric) error

Flush sends metrics to Datadog

func (*DatadogMetricSink) FlushOtherSamples

func (dd *DatadogMetricSink) FlushOtherSamples(ctx context.Context, samples []ssf.SSFSample)

FlushOtherSamples serializes Events or Service Checks directly to datadog. May make 2 external calls to the datadog client.

func (*DatadogMetricSink) Name

func (dd *DatadogMetricSink) Name() string

Name returns the name of this sink.

func (*DatadogMetricSink) Start

func (dd *DatadogMetricSink) Start(cl *trace.Client) error

Start sets the sink up.

type DatadogSpanSink

type DatadogSpanSink struct {
	HTTPClient *http.Client
	// contains filtered or unexported fields
}

DatadogSpanSink is a sink for sending spans to a Datadog trace agent.

func NewDatadogSpanSink

func NewDatadogSpanSink(address string, bufferSize int, httpClient *http.Client, log *logrus.Logger) (*DatadogSpanSink, error)

NewDatadogSpanSink creates a new Datadog sink for trace spans.

func (*DatadogSpanSink) Flush

func (dd *DatadogSpanSink) Flush()

Flush signals the sink to send it's spans to their destination. For this sync it means we'll be making an HTTP request to send them along. We assume it's beneficial to performance to defer these until the normal 10s flush.

func (*DatadogSpanSink) Ingest

func (dd *DatadogSpanSink) Ingest(span *ssf.SSFSpan) error

Ingest takes the span and adds it to the ringbuffer.

func (*DatadogSpanSink) Name

func (dd *DatadogSpanSink) Name() string

Name returns the name of this sink.

func (*DatadogSpanSink) Start

func (dd *DatadogSpanSink) Start(cl *trace.Client) error

Start performs final adjustments on the sink.

type DatadogTraceSpan

type DatadogTraceSpan struct {
	Duration int64              `json:"duration"`
	Error    int64              `json:"error"`
	Meta     map[string]string  `json:"meta"`
	Metrics  map[string]float64 `json:"metrics"`
	Name     string             `json:"name"`
	ParentID int64              `json:"parent_id,omitempty"`
	Resource string             `json:"resource,omitempty"`
	Service  string             `json:"service"`
	SpanID   int64              `json:"span_id"`
	Start    int64              `json:"start"`
	TraceID  int64              `json:"trace_id"`
	Type     string             `json:"type"`
}

DatadogTraceSpan represents a trace span as JSON for the Datadog tracing API.