The highest tagged major version is
README
¶
Sensor Sensibility Format
The Sensor Sensibility Format — or SSF for short — is a language agnostic format for transmitting observability data such as trace spans, metrics, events and more.
Why?
SSF is based on prior art of metrics formats and protocols mentioned in the inspiration section. Unlike each of these wonderful formats, which we have used for years and benefited greatly from, SSF is a binary format utilizing Protocol Buffers. It is emission-, collection- and storage-agnostic. It is merely a protocol for transmitting information. It is developed as part of a suite of technologies around Veneur.
Why A New Format?
Because we want to:
- leverage protobuf so we no longer have to write buggy, string-based marshaling and unmarshaling code in clients libraries and server implementations
- benefit from the efficiency of protobuf
- collect and combine the great ideas from our inspiration
- add some of our own ideas
Using SSF
- You can send SSF as UDP packets to Veneur's
ssf_listen_addresses. - Veneur can also read SSF as a wire protocol over connection protocols such as TCP or UNIX domain sockets
- You can use the CLI tool veneur-emit.
- You can use an SSF trace client in Go or Ruby.
Philosophy
We've got some novel ideas that we've put in to SSF. It might help to be familiar with the concepts in our inspiration. Here is our founding philosophy of SSF:
- a timer is a span
- a log line is a span, especially if it's structured
- events are also spans
- therefore, the core unit of all observability data is a span, or a unit of a trace.
- spans might be single units, or units of a larger whole
- other point metrics (e.g. counters and gauges) can be constituents of a span
- it's more valuable to know the depth of a queue in the context of a span than alone
- improve the context of counters and gauges, as they are part of a span
- provide a format containing the superset of many backend's features
Components
You can examine the protobuf definition, but in a nutshell SSF provides the following:
A metric field describing it's type as one of COUNTER, GAUGE, HISTOGRAM (supplanting a timer), SET and STATUS. Rounding out the traditional fields are name, value, sample_rate and timestamp. There is a map of tags string key-value pairs.
Others
Beyond these StatsD-stye fields are also message for including an arbitrary string such as a log message and unit as a string describing the unit of the message such as seconds. Note that SSF does not have defined units at present. Only strings!
STATUS Samples
A Metric of STATUS is most like a Nagios check result.
Log Samples?
Since all fields are optional, one could leave out many fields and represent a log line in SSF by setting timestamp, name with a canonical name and tags for parameters. This is intended to be used in the future for Veneur to unify observability primitives.
Inspiration
We build on the shoulders of giants, and are proud to have used and been inspired by these marvelous tools:
Documentation
¶
Overview ¶
Package ssf provides an implementation of the Sensor Sensibility Format. It consists of two parts: One is the protobuf implementations of the SSF data structures SSFSpan and SSFSample, and the other is a set of helper routines for generating SSFSamples that can be reported on an SSFSpan.
The types in this package is meant to be used together with the neighboring packages trace and trace/metrics.
Package ssf is a generated protocol buffer package. It is generated from these files: ssf/sample.proto It has these top-level messages: SSFSample SSFSpan
Index ¶
- Variables
- type SSFSample
- func Count(name string, value float32, tags map[string]string, opts ...SampleOption) *SSFSample
- func Gauge(name string, value float32, tags map[string]string, opts ...SampleOption) *SSFSample
- func Histogram(name string, value float32, tags map[string]string, opts ...SampleOption) *SSFSample
- func RandomlySample(rate float32, samples ...*SSFSample) []*SSFSample
- func Set(name string, value string, tags map[string]string, opts ...SampleOption) *SSFSample
- func Timing(name string, value time.Duration, resolution time.Duration, ...) *SSFSample
- func (*SSFSample) Descriptor() ([]byte, []int)
- func (m *SSFSample) GetMessage() string
- func (m *SSFSample) GetMetric() SSFSample_Metric
- func (m *SSFSample) GetName() string
- func (m *SSFSample) GetSampleRate() float32
- func (m *SSFSample) GetStatus() SSFSample_Status
- func (m *SSFSample) GetTags() map[string]string
- func (m *SSFSample) GetTimestamp() int64
- func (m *SSFSample) GetUnit() string
- func (m *SSFSample) GetValue() float32
- func (m *SSFSample) Marshal() (dAtA []byte, err error)
- func (m *SSFSample) MarshalTo(dAtA []byte) (int, error)
- func (*SSFSample) ProtoMessage()
- func (m *SSFSample) Reset()
- func (m *SSFSample) Size() (n int)
- func (m *SSFSample) String() string
- func (m *SSFSample) Unmarshal(dAtA []byte) error
- type SSFSample_Metric
- type SSFSample_Status
- type SSFSpan
- func (*SSFSpan) Descriptor() ([]byte, []int)
- func (m *SSFSpan) GetEndTimestamp() int64
- func (m *SSFSpan) GetError() bool
- func (m *SSFSpan) GetId() int64
- func (m *SSFSpan) GetIndicator() bool
- func (m *SSFSpan) GetMetrics() []*SSFSample
- func (m *SSFSpan) GetName() string
- func (m *SSFSpan) GetParentId() int64
- func (m *SSFSpan) GetService() string
- func (m *SSFSpan) GetStartTimestamp() int64
- func (m *SSFSpan) GetTags() map[string]string
- func (m *SSFSpan) GetTraceId() int64
- func (m *SSFSpan) GetVersion() int32
- func (m *SSFSpan) Marshal() (dAtA []byte, err error)
- func (m *SSFSpan) MarshalTo(dAtA []byte) (int, error)
- func (*SSFSpan) ProtoMessage()
- func (m *SSFSpan) Reset()
- func (m *SSFSpan) Size() (n int)
- func (m *SSFSpan) String() string
- func (m *SSFSpan) Unmarshal(dAtA []byte) error
- type SampleOption
- type Samples
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var ( ErrInvalidLengthSample = fmt.Errorf("proto: negative length found during unmarshaling") ErrIntOverflowSample = fmt.Errorf("proto: integer overflow") )
var NamePrefix string
NamePrefix is a string prepended to every SSFSample name generated by the constructors in this package. As no separator is added between this prefix and the metric name, users must take care to attach any separators to the prefix themselves.
var SSFSample_Metric_name = map[int32]string{
0: "COUNTER",
1: "GAUGE",
2: "HISTOGRAM",
3: "SET",
4: "STATUS",
}
var SSFSample_Metric_value = map[string]int32{
"COUNTER": 0,
"GAUGE": 1,
"HISTOGRAM": 2,
"SET": 3,
"STATUS": 4,
}
var SSFSample_Status_name = map[int32]string{
0: "OK",
1: "WARNING",
2: "CRITICAL",
3: "UNKNOWN",
}
var SSFSample_Status_value = map[string]int32{
"OK": 0,
"WARNING": 1,
"CRITICAL": 2,
"UNKNOWN": 3,
}
Functions ¶
This section is empty.
Types ¶
type SSFSample ¶
type SSFSample struct {
// The underlying type of the metric
Metric SSFSample_Metric `protobuf:"varint,1,opt,name=metric,proto3,enum=ssf.SSFSample_Metric" json:"metric,omitempty"`
// no spaces, but . is allowed
// e.g.: veneur.bar.baz
Name string `protobuf:"bytes,2,opt,name=name,proto3" json:"name,omitempty"`
Value float32 `protobuf:"fixed32,3,opt,name=value,proto3" json:"value,omitempty"`
Timestamp int64 `protobuf:"varint,4,opt,name=timestamp,proto3" json:"timestamp,omitempty"`
Message string `protobuf:"bytes,5,opt,name=message,proto3" json:"message,omitempty"`
Status SSFSample_Status `protobuf:"varint,6,opt,name=status,proto3,enum=ssf.SSFSample_Status" json:"status,omitempty"`
SampleRate float32 `protobuf:"fixed32,7,opt,name=sample_rate,json=sampleRate,proto3" json:"sample_rate,omitempty"`
Tags map[string]string `` /* 142-byte string literal not displayed */
Unit string `protobuf:"bytes,9,opt,name=unit,proto3" json:"unit,omitempty"`
}
SSFSample is similar of a StatsD-style, point in time metric. It has a Metric type, a name, a value and a timestamp. Additionally it can contain a message, a status, a sample rate, a map of tags as string keys and values and a unit type. Note that SSF doesn't understand units, they are just strings!
func Count ¶ added in v1.9.0
Count returns an SSFSample representing an increment / decrement of a counter. It's a convenience wrapper around constructing SSFSample objects.
func Gauge ¶ added in v1.9.0
Gauge returns an SSFSample representing a gauge at a certain value. It's a convenience wrapper around constructing SSFSample objects.
func Histogram ¶ added in v1.9.0
Histogram returns an SSFSample representing a value on a histogram, like a timer or other range. It's a convenience wrapper around constructing SSFSample objects.
func RandomlySample ¶ added in v1.9.0
RandomlySample takes a rate and a set of measurements, and returns a new set of measurements as if sampling had been performed: Each original measurement gets rejected/included in the result based on a random roll of the RNG according to the rate, and each included measurement has its SampleRate field adjusted to be its original SampleRate * rate.
Example ¶
package main
import (
"time"
"github.com/stripe/veneur/ssf"
"github.com/stripe/veneur/trace"
"github.com/stripe/veneur/trace/metrics"
)
func main() {
samples := &ssf.Samples{}
// Sample some metrics at 50% - each of these metrics, if it
// gets picked, will report with a SampleRate of 0.5:
samples.Add(ssf.RandomlySample(0.5,
ssf.Count("cheap.counter", 1, nil),
ssf.Timing("cheap.timer", 1*time.Second, time.Nanosecond, nil),
)...)
// Sample another metric at 1% - if included, the metric will
// have a SampleRate of 0.01:
samples.Add(ssf.RandomlySample(0.01,
ssf.Count("expensive.counter", 20, nil))...)
// Report these metrics:
metrics.Report(trace.DefaultClient, samples)
}
Output:
func Set ¶ added in v1.9.0
Set returns an SSFSample representing a value on a set, useful for counting the unique values that occur in a certain time bound.
func Timing ¶ added in v1.9.0
func Timing(name string, value time.Duration, resolution time.Duration, tags map[string]string, opts ...SampleOption) *SSFSample
Timing returns an SSFSample (really a histogram) representing the timing in the given resolution.
func (*SSFSample) Descriptor ¶
func (*SSFSample) GetMessage ¶
func (*SSFSample) GetMetric ¶
func (m *SSFSample) GetMetric() SSFSample_Metric
func (*SSFSample) GetSampleRate ¶
func (*SSFSample) GetStatus ¶
func (m *SSFSample) GetStatus() SSFSample_Status
func (*SSFSample) GetTimestamp ¶
func (*SSFSample) ProtoMessage ¶
func (*SSFSample) ProtoMessage()
type SSFSample_Metric ¶
type SSFSample_Metric int32
const ( SSFSample_COUNTER SSFSample_Metric = 0 SSFSample_GAUGE SSFSample_Metric = 1 SSFSample_HISTOGRAM SSFSample_Metric = 2 SSFSample_SET SSFSample_Metric = 3 SSFSample_STATUS SSFSample_Metric = 4 )
func (SSFSample_Metric) EnumDescriptor ¶
func (SSFSample_Metric) EnumDescriptor() ([]byte, []int)
func (SSFSample_Metric) String ¶
func (x SSFSample_Metric) String() string
type SSFSample_Status ¶
type SSFSample_Status int32
const ( SSFSample_OK SSFSample_Status = 0 SSFSample_WARNING SSFSample_Status = 1 SSFSample_CRITICAL SSFSample_Status = 2 SSFSample_UNKNOWN SSFSample_Status = 3 )
func (SSFSample_Status) EnumDescriptor ¶
func (SSFSample_Status) EnumDescriptor() ([]byte, []int)
func (SSFSample_Status) String ¶
func (x SSFSample_Status) String() string
type SSFSpan ¶ added in v1.5.0
type SSFSpan struct {
Version int32 `protobuf:"varint,1,opt,name=version,proto3" json:"version,omitempty"`
// the trace_id is the (span) id of the root span
TraceId int64 `protobuf:"varint,2,opt,name=trace_id,json=traceId,proto3" json:"trace_id,omitempty"`
// the id for this span
Id int64 `protobuf:"varint,3,opt,name=id,proto3" json:"id,omitempty"`
// the (span) id of the direct parent, if this span is not a root
// span
ParentId int64 `protobuf:"varint,4,opt,name=parent_id,json=parentId,proto3" json:"parent_id,omitempty"`
StartTimestamp int64 `protobuf:"varint,5,opt,name=start_timestamp,json=startTimestamp,proto3" json:"start_timestamp,omitempty"`
EndTimestamp int64 `protobuf:"varint,6,opt,name=end_timestamp,json=endTimestamp,proto3" json:"end_timestamp,omitempty"`
// This flag being true signals that this span was an error. That definition
// of error is not implicitly fatal, as a span may error but be fixed by
// a subsequent retry, etc.
Error bool `protobuf:"varint,7,opt,name=error,proto3" json:"error,omitempty"`
// The name of the service
// e.g. "veneur"
Service string `protobuf:"bytes,8,opt,name=service,proto3" json:"service,omitempty"`
Metrics []*SSFSample `protobuf:"bytes,10,rep,name=metrics" json:"metrics,omitempty"`
// Tags are name value pairs that describe a facet of the span. They apply to
// the *entire* span as opposed to logs which apply to a specific time in
// the span.
Tags map[string]string `` /* 143-byte string literal not displayed */
// An indicator span is one that represents an action that is included in a
// service's Service Level Indicators (https://en.wikipedia.org/wiki/Service_level_indicator)
// This is a signal to receivers that this span may be used to compute SLIs.
// In practice a service's core feature — the thing you would "bill" for, such
// as an API call or read/write operation — would be flagged as an indicator
// span, and its child spans would further describe its duration.
// It's also worth nothing that an indicator need not be the "root" or first
// span in a trace. You might have various forms of middleware that happen
// first or you might have multiple services participating in the same trace.
Indicator bool `protobuf:"varint,12,opt,name=indicator,proto3" json:"indicator,omitempty"`
// What to call this span. This could take the form of the endpoint
// (/customer/:id), the function (class::name.method), a friendly name
// (foo middleware) or whatever makes sense in your context.
Name string `protobuf:"bytes,13,opt,name=name,proto3" json:"name,omitempty"`
}
SSFSpan is the primary unit of reporting in SSF. It embeds a set of SSFSamples, as well as start/stop time stamps and a parent ID (which allows assembling a span lineage for distributed tracing purposes).
Note that since this is protobuf, an SSFSpan does not *have* to include metrics, just as it does not *have* to include information necessary to reconstruct a trace.
Compatibility ¶
On ingestion, an SSFSpan with an empty string for a name field but a tag "name" will have that name field replaced with the name tag, and the tag is removed.
Metric SSFSamples with a zero sample_rate (indicating it was left out) have the sample_rate field set to 1 on ingestion.
Validity Criteria ¶
Programs consuming SSFSpans should take care to only process spans and metrics that fulfill the following criteria:
Metrics are considered valid if they have a name and a value.
SSFSpans are considered valid trace spans if they have non-zero id, trace_id, start_timestamp and end_timestamp fields.
func (*SSFSpan) Descriptor ¶ added in v1.5.0
func (*SSFSpan) GetEndTimestamp ¶ added in v1.5.0
func (*SSFSpan) GetIndicator ¶ added in v1.6.0
func (*SSFSpan) GetMetrics ¶ added in v1.5.0
func (*SSFSpan) GetParentId ¶ added in v1.5.0
func (*SSFSpan) GetService ¶ added in v1.5.0
func (*SSFSpan) GetStartTimestamp ¶ added in v1.5.0
func (*SSFSpan) GetTraceId ¶ added in v1.5.0
func (*SSFSpan) GetVersion ¶ added in v1.5.0
func (*SSFSpan) ProtoMessage ¶ added in v1.5.0
func (*SSFSpan) ProtoMessage()
type SampleOption ¶ added in v1.9.0
type SampleOption func(*SSFSample)
SampleOption is a functional option that can be used for less commonly needed fields in sample creation helper functions. The options are applied by order of arguments (left to right), so when setting multiple of the same option, the rightmost wins.
func SampleRate ¶ added in v1.9.0
func SampleRate(rate float32) SampleOption
SampleRate sets the rate at which a measurement is sampled. The rate is a number on the interval (0..1] (1 means that the value is not sampled). Any numbers outside this interval result in no change to the sample rate (by default, all SSFSamples created with the helpers in this package have a SampleRate=1).
func TimeUnit ¶ added in v1.9.0
func TimeUnit(resolution time.Duration) SampleOption
TimeUnit sets the unit on a sample to the given resolution's SI unit symbol. Valid resolutions are the time duration constants from Nanosecond through Hour. The non-SI units "minute" and "hour" are represented by "min" and "h" respectively.
If a resolution is passed that does not correspond exactly to the duration constants in package time, this option does not affect the sample at all.
func Timestamp ¶ added in v1.9.0
func Timestamp(ts time.Time) SampleOption
Timestamp is a functional option for creating an SSFSample. It sets the timestamp field on the sample to the timestamp passed.
func Unit ¶ added in v1.9.0
func Unit(name string) SampleOption
Unit is a functional option for creating an SSFSample. It sets the sample's unit name to the name passed.
Source Files