ssf

package
v13.2.0+incompatible Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 1, 2020 License: MIT Imports: 8 Imported by: 169

README

Sensor Sensibility Format

The Sensor Sensibility Format — or SSF for short — is a language agnostic format for transmitting observability data such as trace spans, metrics, events and more.

Why?

SSF is based on prior art of metrics formats and protocols mentioned in the inspiration section. Unlike each of these wonderful formats, which we have used for years and benefited greatly from, SSF is a binary format utilizing Protocol Buffers. It is emission-, collection- and storage-agnostic. It is merely a protocol for transmitting information. It is developed as part of a suite of technologies around Veneur.

Why A New Format?

Because we want to:

  • leverage protobuf so we no longer have to write buggy, string-based marshaling and unmarshaling code in client libraries and server implementations
  • benefit from the efficiency of protobuf
  • collect and combine the great ideas from the tools that inspired us
  • add some of our own ideas

Using SSF

  • You can send SSF as UDP packets to Veneur's ssf_listen_addresses.
  • Veneur can also read SSF as a wire protocol over connection protocols such as TCP or UNIX domain sockets
  • You can use the CLI tool veneur-emit.
  • You can use an SSF trace client in Go or Ruby.

Philosophy

We've got some novel ideas that we've put in to SSF. It might help to be familiar with the concepts in our inspiration. Here is our founding philosophy of SSF:

  • a timer is a span
  • a log line is a span, especially if it's structured
  • events are also spans
  • therefore, the core unit of all observability data is a span, or a unit of a trace.
    • spans might be single units, or units of a larger whole
  • other point metrics (e.g. counters and gauges) can be constituents of a span
    • it's more valuable to know the depth of a queue in the context of a span than alone
    • improve the context of counters and gauges, as they are part of a span
  • provide a format containing the superset of many backends' features

Components

You can examine the protobuf definition, but in a nutshell SSF provides the following:

A metric field describing its type as one of COUNTER, GAUGE, HISTOGRAM (supplanting a timer), SET and STATUS. Rounding out the traditional fields are name, value, sample_rate and timestamp. There is a map of tags string key-value pairs.

Others

Beyond these StatsD-stye fields are also message for including an arbitrary string such as a log message and unit as a string describing the unit of the message such as seconds. Note that SSF does not have defined units at present. Only strings!

STATUS Samples

A Metric of STATUS is most like a Nagios check result.

Log Samples?

Since all fields are optional, one could leave out many fields and represent a log line in SSF by setting timestamp, name with a canonical name and tags for parameters. This is intended to be used in the future for Veneur to unify observability primitives.

Inspiration

We build on the shoulders of giants, and are proud to have used and been inspired by these marvelous tools:

Documentation

Overview

Package ssf provides an implementation of the Sensor Sensibility Format. It consists of two parts: One is the protobuf implementations of the SSF data structures SSFSpan and SSFSample, and the other is a set of helper routines for generating SSFSamples that can be reported on an SSFSpan.

The types in this package are meant to be used together with the neighboring packages trace and trace/metrics.

Index

Examples

Constants

This section is empty.

Variables

View Source 
var (
	ErrInvalidLengthSample = fmt.Errorf("proto: negative length found during unmarshaling")
	ErrIntOverflowSample   = fmt.Errorf("proto: integer overflow")
)
View Source 
var NamePrefix string

NamePrefix is a string prepended to every SSFSample name generated by the constructors in this package. As no separator is added between this prefix and the metric name, users must take care to attach any separators to the prefix themselves.

View Source 
var SSFSample_Metric_name = map[int32]string{
	0: "COUNTER",
	1: "GAUGE",
	2: "HISTOGRAM",
	3: "SET",
	4: "STATUS",
}
View Source 
var SSFSample_Metric_value = map[string]int32{
	"COUNTER":   0,
	"GAUGE":     1,
	"HISTOGRAM": 2,
	"SET":       3,
	"STATUS":    4,
}
View Source 
var SSFSample_Scope_name = map[int32]string{
	0: "DEFAULT",
	1: "LOCAL",
	2: "GLOBAL",
}
View Source 
var SSFSample_Scope_value = map[string]int32{
	"DEFAULT": 0,
	"LOCAL":   1,
	"GLOBAL":  2,
}
View Source 
var SSFSample_Status_name = map[int32]string{
	0: "OK",
	1: "WARNING",
	2: "CRITICAL",
	3: "UNKNOWN",
}
View Source 
var SSFSample_Status_value = map[string]int32{
	"OK":       0,
	"WARNING":  1,
	"CRITICAL": 2,
	"UNKNOWN":  3,
}

Functions

This section is empty.

Types

type SSFSample 

type SSFSample struct {
	// The underlying type of the metric
	Metric SSFSample_Metric `protobuf:"varint,1,opt,name=metric,proto3,enum=ssf.SSFSample_Metric" json:"metric,omitempty"`
	// no spaces, but . is allowed
	// e.g.: veneur.bar.baz
	Name       string            `protobuf:"bytes,2,opt,name=name,proto3" json:"name,omitempty"`
	Value      float32           `protobuf:"fixed32,3,opt,name=value,proto3" json:"value,omitempty"`
	Timestamp  int64             `protobuf:"varint,4,opt,name=timestamp,proto3" json:"timestamp,omitempty"`
	Message    string            `protobuf:"bytes,5,opt,name=message,proto3" json:"message,omitempty"`
	Status     SSFSample_Status  `protobuf:"varint,6,opt,name=status,proto3,enum=ssf.SSFSample_Status" json:"status,omitempty"`
	SampleRate float32           `protobuf:"fixed32,7,opt,name=sample_rate,json=sampleRate,proto3" json:"sample_rate,omitempty"`
	Tags       map[string]string `` /* 149-byte string literal not displayed */
	Unit       string            `protobuf:"bytes,9,opt,name=unit,proto3" json:"unit,omitempty"`
	// scope indicates to an SSF endpoint what it should do with a metric:
	//
	//     - DEFAULT (or absent) - aggregate counters and gauges locally,
	//       handle histograms and sets globally.
	//     - LOCAL - aggregate all metrics locally.
	//     - GLOBAL - aggregate all metrics globally.
	//
	Scope SSFSample_Scope `protobuf:"varint,10,opt,name=scope,proto3,enum=ssf.SSFSample_Scope" json:"scope,omitempty"`
}

SSFSample is similar of a StatsD-style, point in time metric. It has a Metric type, a name, a value and a timestamp. Additionally it can contain a message, a status, a sample rate, a map of tags as string keys and values and a unit type. Note that SSF doesn't understand units, they are just strings!

func Count added in v1.9.0 

func Count(name string, value float32, tags map[string]string, opts ...SampleOption) *SSFSample

Count returns an SSFSample representing an increment / decrement of a counter. It's a convenience wrapper around constructing SSFSample objects.

func Gauge added in v1.9.0 

func Gauge(name string, value float32, tags map[string]string, opts ...SampleOption) *SSFSample

Gauge returns an SSFSample representing a gauge at a certain value. It's a convenience wrapper around constructing SSFSample objects.

func Histogram added in v1.9.0 

func Histogram(name string, value float32, tags map[string]string, opts ...SampleOption) *SSFSample

Histogram returns an SSFSample representing a value on a histogram, like a timer or other range. It's a convenience wrapper around constructing SSFSample objects.

func RandomlySample added in v1.9.0 

func RandomlySample(rate float32, samples ...*SSFSample) []*SSFSample

RandomlySample takes a rate and a set of measurements, and returns a new set of measurements as if sampling had been performed: Each original measurement gets rejected/included in the result based on a random roll of the RNG according to the rate, and each included measurement has its SampleRate field adjusted to be its original SampleRate * rate.

Example 
package main

import (
	"time"

	"github.com/stripe/veneur/ssf"
	"github.com/stripe/veneur/trace"
	"github.com/stripe/veneur/trace/metrics"
)

func main() {
	samples := &ssf.Samples{}
	// Sample some metrics at 50% - each of these metrics, if it
	// gets picked, will report with a SampleRate of 0.5:
	samples.Add(ssf.RandomlySample(0.5,
		ssf.Count("cheap.counter", 1, nil),
		ssf.Timing("cheap.timer", 1*time.Second, time.Nanosecond, nil),
	)...)

	// Sample another metric at 1% - if included, the metric will
	// have a SampleRate of 0.01:
	samples.Add(ssf.RandomlySample(0.01,
		ssf.Count("expensive.counter", 20, nil))...)

	// Report these metrics:
	metrics.Report(trace.DefaultClient, samples)
}

Output:

func Set added in v1.9.0 

func Set(name string, value string, tags map[string]string, opts ...SampleOption) *SSFSample

Set returns an SSFSample representing a value on a set, useful for counting the unique values that occur in a certain time bound.

func Status added in v1.9.0 

func Status(name string, state SSFSample_Status, tags map[string]string, opts ...SampleOption) *SSFSample

Status returns an SSFSample capturing the reported state of a service

func Timing added in v1.9.0 

func Timing(name string, value time.Duration, resolution time.Duration, tags map[string]string, opts ...SampleOption) *SSFSample

Timing returns an SSFSample (really a histogram) representing the timing in the given resolution.

func (*SSFSample) Descriptor 

func (*SSFSample) Descriptor() ([]byte, []int)

func (*SSFSample) GetMessage 

func (m *SSFSample) GetMessage() string

func (*SSFSample) GetMetric 

func (m *SSFSample) GetMetric() SSFSample_Metric

func (*SSFSample) GetName 

func (m *SSFSample) GetName() string

func (*SSFSample) GetSampleRate 

func (m *SSFSample) GetSampleRate() float32

func (*SSFSample) GetScope 

func (m *SSFSample) GetScope() SSFSample_Scope

func (*SSFSample) GetStatus 

func (m *SSFSample) GetStatus() SSFSample_Status

func (*SSFSample) GetTags 

func (m *SSFSample) GetTags() map[string]string

func (*SSFSample) GetTimestamp 

func (m *SSFSample) GetTimestamp() int64

func (*SSFSample) GetUnit 

func (m *SSFSample) GetUnit() string

func (*SSFSample) GetValue added in v1.5.0 

func (m *SSFSample) GetValue() float32

func (*SSFSample) Marshal added in v1.6.0 

func (m *SSFSample) Marshal() (dAtA []byte, err error)

func (*SSFSample) MarshalTo added in v1.6.0 

func (m *SSFSample) MarshalTo(dAtA []byte) (int, error)

func (*SSFSample) ProtoMessage 

func (*SSFSample) ProtoMessage()

func (*SSFSample) Reset 

func (m *SSFSample) Reset()

func (*SSFSample) Size added in v1.6.0 

func (m *SSFSample) Size() (n int)

func (*SSFSample) String 

func (m *SSFSample) String() string

func (*SSFSample) Unmarshal added in v1.6.0 

func (m *SSFSample) Unmarshal(dAtA []byte) error

func (*SSFSample) XXX_DiscardUnknown 

func (m *SSFSample) XXX_DiscardUnknown()

func (*SSFSample) XXX_Marshal 

func (m *SSFSample) XXX_Marshal(b []byte, deterministic bool) ([]byte, error)

func (*SSFSample) XXX_Merge 

func (m *SSFSample) XXX_Merge(src proto.Message)

func (*SSFSample) XXX_Size 

func (m *SSFSample) XXX_Size() int

func (*SSFSample) XXX_Unmarshal 

func (m *SSFSample) XXX_Unmarshal(b []byte) error

type SSFSample_Metric 

type SSFSample_Metric int32
const (
	SSFSample_COUNTER   SSFSample_Metric = 0
	SSFSample_GAUGE     SSFSample_Metric = 1
	SSFSample_HISTOGRAM SSFSample_Metric = 2
	SSFSample_SET       SSFSample_Metric = 3
	SSFSample_STATUS    SSFSample_Metric = 4
)

func (SSFSample_Metric) EnumDescriptor 

func (SSFSample_Metric) EnumDescriptor() ([]byte, []int)

func (SSFSample_Metric) String 

func (x SSFSample_Metric) String() string

type SSFSample_Scope 

type SSFSample_Scope int32
const (
	SSFSample_DEFAULT SSFSample_Scope = 0
	SSFSample_LOCAL   SSFSample_Scope = 1
	SSFSample_GLOBAL  SSFSample_Scope = 2
)

func (SSFSample_Scope) EnumDescriptor 

func (SSFSample_Scope) EnumDescriptor() ([]byte, []int)

func (SSFSample_Scope) String 

func (x SSFSample_Scope) String() string

type SSFSample_Status 

type SSFSample_Status int32
const (
	SSFSample_OK       SSFSample_Status = 0
	SSFSample_WARNING  SSFSample_Status = 1
	SSFSample_CRITICAL SSFSample_Status = 2
	SSFSample_UNKNOWN  SSFSample_Status = 3
)

func (SSFSample_Status) EnumDescriptor 

func (SSFSample_Status) EnumDescriptor() ([]byte, []int)

func (SSFSample_Status) String 

func (x SSFSample_Status) String() string

type SSFSpan added in v1.5.0 

type SSFSpan struct {
	Version int32 `protobuf:"varint,1,opt,name=version,proto3" json:"version,omitempty"`
	// the trace_id is the (span) id of the root span
	TraceId int64 `protobuf:"varint,2,opt,name=trace_id,json=traceId,proto3" json:"trace_id,omitempty"`
	// the id for this span
	Id int64 `protobuf:"varint,3,opt,name=id,proto3" json:"id,omitempty"`
	// the (span) id of the direct parent, if this span is not a root
	// span
	ParentId       int64 `protobuf:"varint,4,opt,name=parent_id,json=parentId,proto3" json:"parent_id,omitempty"`
	StartTimestamp int64 `protobuf:"varint,5,opt,name=start_timestamp,json=startTimestamp,proto3" json:"start_timestamp,omitempty"`
	EndTimestamp   int64 `protobuf:"varint,6,opt,name=end_timestamp,json=endTimestamp,proto3" json:"end_timestamp,omitempty"`
	// This flag being true signals that this span was an error. That definition
	// of error is not implicitly fatal, as a span may error but be fixed by
	// a subsequent retry, etc.
	Error bool `protobuf:"varint,7,opt,name=error,proto3" json:"error,omitempty"`
	// The name of the service
	// e.g. "veneur"
	Service string       `protobuf:"bytes,8,opt,name=service,proto3" json:"service,omitempty"`
	Metrics []*SSFSample `protobuf:"bytes,10,rep,name=metrics,proto3" json:"metrics,omitempty"`
	// Tags are name value pairs that describe a facet of the span. They apply to
	// the *entire* span as opposed to logs which apply to a specific time in
	// the span.
	Tags map[string]string `` /* 150-byte string literal not displayed */
	// An indicator span is one that represents an action that is included in a
	// service's Service Level Indicators (https://en.wikipedia.org/wiki/Service_level_indicator)
	// This is a signal to receivers that this span may be used to compute SLIs.
	// In practice a service's core feature — the thing you would "bill" for, such
	// as an API call or read/write operation — would be flagged as an indicator
	// span, and its child spans would further describe its duration.
	// It's also worth nothing that an indicator need not be the "root" or first
	// span in a trace. You might have various forms of middleware that happen
	// first or you might have multiple services participating in the same trace.
	Indicator bool `protobuf:"varint,12,opt,name=indicator,proto3" json:"indicator,omitempty"`
	// What to call this span. This could take the form of the endpoint
	// (/customer/:id), the function (class::name.method), a friendly name
	// (foo middleware) or whatever makes sense in your context.
	Name string `protobuf:"bytes,13,opt,name=name,proto3" json:"name,omitempty"`
	// Root trace start time. This may be different than the span starting time.
	// This is especially useful for communicating with AWS X-Ray and have a consistent
	// TraceID.
	RootStartTimestamp int64 `protobuf:"varint,14,opt,name=root_start_timestamp,json=rootStartTimestamp,proto3" json:"root_start_timestamp,omitempty"`
}

SSFSpan is the primary unit of reporting in SSF. It embeds a set of SSFSamples, as well as start/stop time stamps and a parent ID (which allows assembling a span lineage for distributed tracing purposes).

Note that since this is protobuf, an SSFSpan does not *have* to include metrics, just as it does not *have* to include information necessary to reconstruct a trace.

Compatibility

On ingestion, an SSFSpan with an empty string for a name field but a tag "name" will have that name field replaced with the name tag, and the tag is removed.

Metric SSFSamples with a zero sample_rate (indicating it was left out) have the sample_rate field set to 1 on ingestion.

Validity Criteria

Programs consuming SSFSpans should take care to only process spans and metrics that fulfill the following criteria:

Metrics are considered valid if they have a name and a value.

SSFSpans are considered valid trace spans if they have non-zero id, trace_id, start_timestamp and end_timestamp fields.

func (*SSFSpan) Descriptor added in v1.5.0 

func (*SSFSpan) Descriptor() ([]byte, []int)

func (*SSFSpan) GetEndTimestamp added in v1.5.0 

func (m *SSFSpan) GetEndTimestamp() int64

func (*SSFSpan) GetError added in v1.5.0 

func (m *SSFSpan) GetError() bool

func (*SSFSpan) GetId added in v1.5.0 

func (m *SSFSpan) GetId() int64

func (*SSFSpan) GetIndicator added in v1.6.0 

func (m *SSFSpan) GetIndicator() bool

func (*SSFSpan) GetMetrics added in v1.5.0 

func (m *SSFSpan) GetMetrics() []*SSFSample

func (*SSFSpan) GetName added in v1.6.0 

func (m *SSFSpan) GetName() string

func (*SSFSpan) GetParentId added in v1.5.0 

func (m *SSFSpan) GetParentId() int64

func (*SSFSpan) GetRootStartTimestamp 

func (m *SSFSpan) GetRootStartTimestamp() int64

func (*SSFSpan) GetService added in v1.5.0 

func (m *SSFSpan) GetService() string

func (*SSFSpan) GetStartTimestamp added in v1.5.0 

func (m *SSFSpan) GetStartTimestamp() int64

func (*SSFSpan) GetTags added in v1.5.0 

func (m *SSFSpan) GetTags() map[string]string

func (*SSFSpan) GetTraceId added in v1.5.0 

func (m *SSFSpan) GetTraceId() int64

func (*SSFSpan) GetVersion added in v1.5.0 

func (m *SSFSpan) GetVersion() int32

func (*SSFSpan) Marshal added in v1.6.0 

func (m *SSFSpan) Marshal() (dAtA []byte, err error)

func (*SSFSpan) MarshalTo added in v1.6.0 

func (m *SSFSpan) MarshalTo(dAtA []byte) (int, error)

func (*SSFSpan) ProtoMessage added in v1.5.0 

func (*SSFSpan) ProtoMessage()

func (*SSFSpan) Reset added in v1.5.0 

func (m *SSFSpan) Reset()

func (*SSFSpan) Size added in v1.6.0 

func (m *SSFSpan) Size() (n int)

func (*SSFSpan) String added in v1.5.0 

func (m *SSFSpan) String() string

func (*SSFSpan) Unmarshal added in v1.6.0 

func (m *SSFSpan) Unmarshal(dAtA []byte) error

func (*SSFSpan) XXX_DiscardUnknown 

func (m *SSFSpan) XXX_DiscardUnknown()

func (*SSFSpan) XXX_Marshal 

func (m *SSFSpan) XXX_Marshal(b []byte, deterministic bool) ([]byte, error)

func (*SSFSpan) XXX_Merge 

func (m *SSFSpan) XXX_Merge(src proto.Message)

func (*SSFSpan) XXX_Size 

func (m *SSFSpan) XXX_Size() int

func (*SSFSpan) XXX_Unmarshal 

func (m *SSFSpan) XXX_Unmarshal(b []byte) error

type SampleOption added in v1.9.0 

type SampleOption func(*SSFSample)

SampleOption is a functional option that can be used for less commonly needed fields in sample creation helper functions. The options are applied by order of arguments (left to right), so when setting multiple of the same option, the rightmost wins.

func SampleRate added in v1.9.0 

func SampleRate(rate float32) SampleOption

SampleRate sets the rate at which a measurement is sampled. The rate is a number on the interval (0..1] (1 means that the value is not sampled). Any numbers outside this interval result in no change to the sample rate (by default, all SSFSamples created with the helpers in this package have a SampleRate=1).

func Scope 

func Scope(scope SampleScope) SampleOption

Scope is a SampleOption that sets the scope of a metric to be either "global" (i.e., aggregated on a central node), or "local" (i.e., aggregated exclusively on the central node).

func TimeUnit added in v1.9.0 

func TimeUnit(resolution time.Duration) SampleOption

TimeUnit sets the unit on a sample to the given resolution's SI unit symbol. Valid resolutions are the time duration constants from Nanosecond through Hour. The non-SI units "minute" and "hour" are represented by "min" and "h" respectively.

If a resolution is passed that does not correspond exactly to the duration constants in package time, this option does not affect the sample at all.

func Timestamp added in v1.9.0 

func Timestamp(ts time.Time) SampleOption

Timestamp is a functional option for creating an SSFSample. It sets the timestamp field on the sample to the timestamp passed.

func Unit added in v1.9.0 

func Unit(name string) SampleOption

Unit is a functional option for creating an SSFSample. It sets the sample's unit name to the name passed.

type SampleScope 

type SampleScope SSFSample_Scope

SampleScope is a slightly more ergonomic representation of the internal type SSFSample_Scope. 

const (
	DefaultScope SampleScope = SampleScope(SSFSample_DEFAULT)
	Local        SampleScope = SampleScope(SSFSample_LOCAL)
	Global       SampleScope = SampleScope(SSFSample_GLOBAL)
)

type Samples added in v1.9.0 

type Samples struct {
	Batch []*SSFSample
}

Samples is a batch of SSFSamples, not attached to an SSF span, that can be submitted with package metrics's Report function.

func (*Samples) Add added in v1.9.0 

func (s *Samples) Add(sample ...*SSFSample)

Add appends a sample to the batch of samples.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.