tailsamplingprocessor

README

Tail Sampling Processor

Supported pipeline types: traces

The tail sampling processor samples traces based on a set of defined policies. Today, this processor only works with a single instance of the collector. Technically, trace ID aware load balancing could be used to support multiple collector instances, but this configuration has not been tested. Please refer to config.go for the config spec.

The following configuration options are required:

• policies (no default): Policies used to make a sampling decision

Multiple policies exist today and it is straight forward to add more. These include:

• always_sample: Sample all traces
• latency: Sample based on the duration of the trace. The duration is determined by looking at the earliest start time and latest end time, without taking into consideration what happened in between.
• numeric_attribute: Sample based on number attributes
• probabilistic: Sample a percentage of traces. Read a comparison with the Probabilistic Sampling Processor.
• status_code: Sample based upon the status code (OK, ERROR or UNSET)
• string_attribute: Sample based on string attributes value matches, both exact and regex value matches are supported
• rate_limiting: Sample based on rate
• and: Sample based on multiple policies, creates an AND policy
• composite: Sample based on a combination of above samplers, with ordering and rate allocation per sampler. Rate allocation allocates certain percentages of spans per policy order. For example if we have set max_total_spans_per_second as 100 then we can set rate_allocation as follows
  1. test-composite-policy-1 = 50 % of max_total_spans_per_second = 50 spans_per_second
  2. test-composite-policy-2 = 25 % of max_total_spans_per_second = 25 spans_per_second
  3. To ensure remaining capacity is filled use always_sample as one of the policies

The following configuration options can also be modified:

• decision_wait (default = 30s): Wait time since the first span of a trace before making a sampling decision
• num_traces (default = 50000): Number of traces kept in memory
• expected_new_traces_per_sec (default = 0): Expected number of new traces (helps in allocating data structures)

Examples:

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 100
    expected_new_traces_per_sec: 10
    policies:
      [
          {
            name: test-policy-1,
            type: always_sample
          },
          {
            name: test-policy-2,
            type: latency,
            latency: {threshold_ms: 5000}
          },
          {
            name: test-policy-3,
            type: numeric_attribute,
            numeric_attribute: {key: key1, min_value: 50, max_value: 100}
          },
          {
            name: test-policy-4,
            type: probabilistic,
            probabilistic: {sampling_percentage: 10}
          },
          {
            name: test-policy-5,
            type: status_code,
            status_code: {status_codes: [ERROR, UNSET]}
          },
          {
            name: test-policy-6,
            type: string_attribute,
            string_attribute: {key: key2, values: [value1, value2]}
          },
          {
            name: test-policy-7,
            type: string_attribute,
            string_attribute: {key: key2, values: [value1, val*], enabled_regex_matching: true, cache_max_size: 10}
          },
          {
            name: test-policy-8,
            type: rate_limiting,
            rate_limiting: {spans_per_second: 35}
         },
         {
            name: test-policy-9,
            type: string_attribute,
            string_attribute: {key: http.url, values: [\/health, \/metrics], enabled_regex_matching: true, invert_match: true}
         },
         {
            name: and-policy-1,
            type: and,
            and: {
              and_sub_policy: 
              [
                {
                  name: test-and-policy-1,
                  type: numeric_attribute,
                  numeric_attribute: { key: key1, min_value: 50, max_value: 100 }
                },
                {
                    name: test-and-policy-2,
                    type: string_attribute,
                    string_attribute: { key: key2, values: [ value1, value2 ] }
                },
              ]
            }
         },
         {
            name: composite-policy-1,
            type: composite,
            composite:
              {
                max_total_spans_per_second: 1000,
                policy_order: [test-composite-policy-1, test-composite-policy-2, test-composite-policy-3],
                composite_sub_policy:
                  [
                    {
                      name: test-composite-policy-1,
                      type: numeric_attribute,
                      numeric_attribute: {key: key1, min_value: 50, max_value: 100}
                    },
                    {
                      name: test-composite-policy-2,
                      type: string_attribute,
                      string_attribute: {key: key2, values: [value1, value2]}
                    },
                    {
                      name: test-composite-policy-3,
                      type: always_sample
                    }
                  ],
                rate_allocation:
                  [
                    {
                      policy: test-composite-policy-1,
                      percent: 50
                    },
                    {
                      policy: test-composite-policy-2,
                      percent: 25
                    }
                  ]
              }
          },
        ]

Refer to tail_sampling_config.yaml for detailed examples on using the processor.

Probabilistic Sampling Processor compared to the Tail Sampling Processor with the Probabilistic policy

The probabilistic sampling processor and the probabilistic tail sampling processor policy work very similar: based upon a configurable sampling percentage they will sample a fixed ratio of received traces. But depending on the overall processing pipeline you should prefer using one over the other.

As a rule of thumb, if you want to add probabilistic sampling and...

...you are not using the tail sampling processor already: use the probabilistic sampling processor. Running the probabilistic sampling processor is more efficient than the tail sampling processor. The probabilistic sampling policy makes decision based upon the trace ID, so waiting until more spans have arrived will not influence its decision.

...you are already using the tail sampling processor: add the probabilistic sampling policy. You are already incurring the cost of running the tail sampling processor, adding the probabilistic policy will be negligible. Additionally, using the policy within the tail sampling processor will ensure traces that are sampled by other policies will not be dropped.

Documentation

Index

• func NewFactory() component.ProcessorFactory
• func SamplingProcessorMetricViews(level configtelemetry.Level) []*view.View
• type AndCfg
• type AndSubPolicyCfg
• type CompositeCfg
• type Config
• type LatencyCfg
• type NumericAttributeCfg
• type PolicyCfg
• type PolicyType
• type ProbabilisticCfg
• type RateAllocationCfg
• type RateLimitingCfg
• type StatusCodeCfg
• type StringAttributeCfg
• type SubPolicyCfg

Functions

func NewFactory

func NewFactory() component.ProcessorFactory

NewFactory returns a new factory for the Tail Sampling processor.

func SamplingProcessorMetricViews

func SamplingProcessorMetricViews(level configtelemetry.Level) []*view.View

SamplingProcessorMetricViews return the metrics views according to given telemetry level.

Types

type AndCfg

type AndCfg struct {
	SubPolicyCfg []AndSubPolicyCfg `mapstructure:"and_sub_policy"`
}

type AndSubPolicyCfg

type AndSubPolicyCfg struct {
	// Name given to the instance of the policy to make easy to identify it in metrics and logs.
	Name string `mapstructure:"name"`
	// Type of the policy this will be used to match the proper configuration of the policy.
	Type PolicyType `mapstructure:"type"`
	// Configs for numeric attribute filter sampling policy evaluator.
	NumericAttributeCfg NumericAttributeCfg `mapstructure:"numeric_attribute"`
	// Configs for probabilistic sampling policy evaluator.
	ProbabilisticCfg ProbabilisticCfg `mapstructure:"probabilistic"`
	// Configs for string attribute filter sampling policy evaluator.
	StringAttributeCfg StringAttributeCfg `mapstructure:"string_attribute"`
	// Configs for rate limiting filter sampling policy evaluator.
	RateLimitingCfg RateLimitingCfg `mapstructure:"rate_limiting"`
	// Configs for latency filter sampling policy evaluator.
	LatencyCfg LatencyCfg `mapstructure:"latency"`
	// Configs for status code filter sampling policy evaluator.
	StatusCodeCfg StatusCodeCfg `mapstructure:"status_code"`
}

type CompositeCfg

type CompositeCfg struct {
	MaxTotalSpansPerSecond int64               `mapstructure:"max_total_spans_per_second"`
	PolicyOrder            []string            `mapstructure:"policy_order"`
	SubPolicyCfg           []SubPolicyCfg      `mapstructure:"composite_sub_policy"`
	RateAllocation         []RateAllocationCfg `mapstructure:"rate_allocation"`
}

CompositeCfg holds the configurable settings to create a composite sampling policy evaluator.

type Config

type Config struct {
	config.ProcessorSettings `mapstructure:",squash"` // squash ensures fields are correctly decoded in embedded struct
	// DecisionWait is the desired wait time from the arrival of the first span of
	// trace until the decision about sampling it or not is evaluated.
	DecisionWait time.Duration `mapstructure:"decision_wait"`
	// NumTraces is the number of traces kept on memory. Typically most of the data
	// of a trace is released after a sampling decision is taken.
	NumTraces uint64 `mapstructure:"num_traces"`
	// ExpectedNewTracesPerSec sets the expected number of new traces sending to the tail sampling processor
	// per second. This helps with allocating data structures with closer to actual usage size.
	ExpectedNewTracesPerSec uint64 `mapstructure:"expected_new_traces_per_sec"`
	// PolicyCfgs sets the tail-based sampling policy which makes a sampling decision
	// for a given trace when requested.
	PolicyCfgs []PolicyCfg `mapstructure:"policies"`
}

Config holds the configuration for tail-based sampling.

type LatencyCfg

type LatencyCfg struct {
	// ThresholdMs in milliseconds.
	ThresholdMs int64 `mapstructure:"threshold_ms"`
}

LatencyCfg holds the configurable settings to create a latency filter sampling policy evaluator

type NumericAttributeCfg

type NumericAttributeCfg struct {
	// Tag that the filter is going to be matching against.
	Key string `mapstructure:"key"`
	// MinValue is the minimum value of the attribute to be considered a match.
	MinValue int64 `mapstructure:"min_value"`
	// MaxValue is the maximum value of the attribute to be considered a match.
	MaxValue int64 `mapstructure:"max_value"`
}

NumericAttributeCfg holds the configurable settings to create a numeric attribute filter sampling policy evaluator.

type PolicyCfg

type PolicyCfg struct {
	// Name given to the instance of the policy to make easy to identify it in metrics and logs.
	Name string `mapstructure:"name"`
	// Type of the policy this will be used to match the proper configuration of the policy.
	Type PolicyType `mapstructure:"type"`
	// Configs for latency filter sampling policy evaluator.
	LatencyCfg LatencyCfg `mapstructure:"latency"`
	// Configs for numeric attribute filter sampling policy evaluator.
	NumericAttributeCfg NumericAttributeCfg `mapstructure:"numeric_attribute"`
	// Configs for probabilistic sampling policy evaluator.
	ProbabilisticCfg ProbabilisticCfg `mapstructure:"probabilistic"`
	// Configs for status code filter sampling policy evaluator.
	StatusCodeCfg StatusCodeCfg `mapstructure:"status_code"`
	// Configs for string attribute filter sampling policy evaluator.
	StringAttributeCfg StringAttributeCfg `mapstructure:"string_attribute"`
	// Configs for rate limiting filter sampling policy evaluator.
	RateLimitingCfg RateLimitingCfg `mapstructure:"rate_limiting"`
	// Configs for defining composite policy
	CompositeCfg CompositeCfg `mapstructure:"composite"`
	// Configs for defining and policy
	AndCfg AndCfg `mapstructure:"and"`
}

PolicyCfg holds the common configuration to all policies.

type PolicyType

type PolicyType string

PolicyType indicates the type of sampling policy.

const (
	// AlwaysSample samples all traces, typically used for debugging.
	AlwaysSample PolicyType = "always_sample"
	// Latency sample traces that are longer than a given threshold.
	Latency PolicyType = "latency"
	// NumericAttribute sample traces that have a given numeric attribute in a specified
	// range, e.g.: attribute "http.status_code" >= 399 and <= 999.
	NumericAttribute PolicyType = "numeric_attribute"
	// Probabilistic samples a given percentage of traces.
	Probabilistic PolicyType = "probabilistic"
	// StatusCode sample traces that have a given status code.
	StatusCode PolicyType = "status_code"
	// StringAttribute sample traces that a attribute, of type string, matching
	// one of the listed values.
	StringAttribute PolicyType = "string_attribute"
	// RateLimiting allows all traces until the specified limits are satisfied.
	RateLimiting PolicyType = "rate_limiting"
	// Composite allows defining a composite policy, combining the other policies in one
	Composite PolicyType = "composite"
	// And allows defining a And policy, combining the other policies in one
	And PolicyType = "and"
)

type ProbabilisticCfg

type ProbabilisticCfg struct {
	// HashSalt allows one to configure the hashing salts. This is important in scenarios where multiple layers of collectors
	// have different sampling rates: if they use the same salt all passing one layer may pass the other even if they have
	// different sampling rates, configuring different salts avoids that.
	HashSalt string `mapstructure:"hash_salt"`
	// SamplingPercentage is the percentage rate at which traces are going to be sampled. Defaults to zero, i.e.: no sample.
	// Values greater or equal 100 are treated as "sample all traces".
	SamplingPercentage float64 `mapstructure:"sampling_percentage"`
}

ProbabilisticCfg holds the configurable settings to create a probabilistic sampling policy evaluator.

type RateAllocationCfg

type RateAllocationCfg struct {
	Policy  string `mapstructure:"policy"`
	Percent int64  `mapstructure:"percent"`
}

RateAllocationCfg used within composite policy

type RateLimitingCfg

type RateLimitingCfg struct {
	// SpansPerSecond sets the limit on the maximum nuber of spans that can be processed each second.
	SpansPerSecond int64 `mapstructure:"spans_per_second"`
}

RateLimitingCfg holds the configurable settings to create a rate limiting sampling policy evaluator.

type StatusCodeCfg

type StatusCodeCfg struct {
	StatusCodes []string `mapstructure:"status_codes"`
}

StatusCodeCfg holds the configurable settings to create a status code filter sampling policy evaluator.

type StringAttributeCfg

type StringAttributeCfg struct {
	// Tag that the filter is going to be matching against.
	Key string `mapstructure:"key"`
	// Values indicate the set of values or regular expressions to use when matching against attribute values.
	// StringAttribute Policy will apply exact value match on Values unless EnabledRegexMatching is true.
	Values []string `mapstructure:"values"`
	// EnabledRegexMatching determines whether match attribute values by regexp string.
	EnabledRegexMatching bool `mapstructure:"enabled_regex_matching"`
	// CacheMaxSize is the maximum number of attribute entries of LRU Cache that stores the matched result
	// from the regular expressions defined in Values.
	// CacheMaxSize will not be used if EnabledRegexMatching is set to false.
	CacheMaxSize int `mapstructure:"cache_max_size"`
	// InvertMatch indicates that values or regular expressions must not match against attribute values.
	// If InvertMatch is true and Values is equal to 'acme', all other values will be sampled except 'acme'.
	// Also, if the specified Key does not match on any resource or span attributes, data will be sampled.
	InvertMatch bool `mapstructure:"invert_match"`
}

StringAttributeCfg holds the configurable settings to create a string attribute filter sampling policy evaluator.

type SubPolicyCfg

type SubPolicyCfg struct {
	// Name given to the instance of the policy to make easy to identify it in metrics and logs.
	Name string `mapstructure:"name"`
	// Type of the policy this will be used to match the proper configuration of the policy.
	Type PolicyType `mapstructure:"type"`
	// Configs for latency filter sampling policy evaluator.
	LatencyCfg LatencyCfg `mapstructure:"latency"`
	// Configs for numeric attribute filter sampling policy evaluator.
	NumericAttributeCfg NumericAttributeCfg `mapstructure:"numeric_attribute"`
	// Configs for probabilistic sampling policy evaluator.
	ProbabilisticCfg ProbabilisticCfg `mapstructure:"probabilistic"`
	// Configs for status code filter sampling policy evaluator.
	StatusCodeCfg StatusCodeCfg `mapstructure:"status_code"`
	// Configs for string attribute filter sampling policy evaluator.
	StringAttributeCfg StringAttributeCfg `mapstructure:"string_attribute"`
	// Configs for rate limiting filter sampling policy evaluator.
	RateLimitingCfg RateLimitingCfg `mapstructure:"rate_limiting"`
	// Configs for and policy evaluator.
	AndCfg AndCfg `mapstructure:"and"`
}

SubPolicyCfg holds the common configuration to all policies under composite policy.