transparencyprocessor

README

otel-transparency-collector

This repo is for the Custom OpenTelemetry Collector Distro integrated with the custom transparency processor. The builder manifest that declares which components (receivers, processors, exporters, and connectors) are available in this custom collector can be found in builder.yaml

The custom transparency processor is an extension of the existing tailsampling processor created by OpenTelemetry authors. It enables sampling of the traces based on the accuracy of the transparency information coming from manually instrumented traces against the static privacy declarations.

The files regarding the configuration of the processor, processor metrics generation and the core tail sampling logic are taken from the tailsampling processor.

Internal

This package contains some packages that were originally contributed to opentelemetry-collector-contrib under ./coreinternal and to tailsampling processor under ./internal.

Since the internal packages from other projects can’t be imported directly, they are copied under this project’s internal package.

Sampling

This package consists of most of the sampling policies already available in the existing tailsampling processor, with the new transparency attribute sampling policy added.

Deployment

This project has a built-in CI/CD setup with GitHub Actions to build and publish the custom OpenTelemetry Collector Distro as a docker image. They are stored as public packages in Github Container Registry.

The custom collector image can be downloaded and installed in the development and production environments with the instructions specified on the package page of the project.

With Helm

The custom collector can also be deployed into Kubernetes cluster with a Helm chart. The modified Helm chart to deploy this custom collector can be found here.

Documentation

Index

• Constants
• func NewFactory() processor.Factory
• func SamplingProcessorMetricViews(level configtelemetry.Level) []*view.View
• type AndCfg
• type AndSubPolicyCfg
• type BooleanAttributeCfg
• type CompositeCfg
• type CompositeSubPolicyCfg
• type Config
• type NumericAttributeCfg
• type OTTLConditionCfg
• type PolicyCfg
• type PolicyType
• type ProbabilisticCfg
• type RateAllocationCfg
• type RateLimitingCfg
• type StatusCodeCfg
• type StringAttributeCfg
• type TraceStateCfg

Constants

const (
	Type            = "transparency"
	TracesStability = component.StabilityLevelBeta
)

Functions

func NewFactory

func NewFactory() processor.Factory

NewFactory creates a factory for the transparency processor

func SamplingProcessorMetricViews

func SamplingProcessorMetricViews(level configtelemetry.Level) []*view.View

SamplingProcessorMetricViews return the metrics views according to given telemetry level.

Types

type AndCfg

type AndCfg struct {
	SubPolicyCfg []AndSubPolicyCfg `mapstructure:"and_sub_policy"`
}

type AndSubPolicyCfg

type AndSubPolicyCfg struct {
	// contains filtered or unexported fields
}

AndSubPolicyCfg holds the common configuration to all policies under and policy.

type BooleanAttributeCfg

type BooleanAttributeCfg struct {
	// Tag that the filter is going to be matching against.
	Key string `mapstructure:"key"`
	// Value indicate the bool value, either true or false to use when matching against attribute values.
	// BooleanAttribute Policy will apply exact value match on Value
	Value bool `mapstructure:"value"`
}

BooleanAttributeCfg holds the configurable settings to create a boolean attribute filter sampling policy evaluator.

type CompositeCfg

type CompositeCfg struct {
	MaxTotalSpansPerSecond int64                   `mapstructure:"max_total_spans_per_second"`
	PolicyOrder            []string                `mapstructure:"policy_order"`
	SubPolicyCfg           []CompositeSubPolicyCfg `mapstructure:"composite_sub_policy"`
	RateAllocation         []RateAllocationCfg     `mapstructure:"rate_allocation"`
}

CompositeCfg holds the configurable settings to create a composite sampling policy evaluator.

type CompositeSubPolicyCfg

type CompositeSubPolicyCfg struct {

	// Configs for and policy evaluator.
	AndCfg AndCfg `mapstructure:"and"`
	// contains filtered or unexported fields
}

CompositeSubPolicyCfg holds the common configuration to all policies under composite policy.

type Config

type Config struct {
	// DecisionWait is the desired wait time from the arrival of the first span of
	// trace until the decision about sampling it or not is evaluated.
	DecisionWait time.Duration `mapstructure:"decision_wait"`
	// NumTraces is the number of traces kept on memory. Typically most of the data
	// of a trace is released after a sampling decision is taken.
	NumTraces uint64 `mapstructure:"num_traces"`
	// ExpectedNewTracesPerSec sets the expected number of new traces sending to the tail sampling processor
	// per second. This helps with allocating data structures with closer to actual usage size.
	ExpectedNewTracesPerSec uint64 `mapstructure:"expected_new_traces_per_sec"`
	// PolicyCfgs sets the tail-based sampling policy which makes a sampling decision
	// for a given trace when requested.
	PolicyCfgs []PolicyCfg `mapstructure:"policies"`
}

Config holds the configuration for tail-based sampling.

type NumericAttributeCfg

type NumericAttributeCfg struct {
	// Tag that the filter is going to be matching against.
	Key string `mapstructure:"key"`
	// MinValue is the minimum value of the attribute to be considered a match.
	MinValue int64 `mapstructure:"min_value"`
	// MaxValue is the maximum value of the attribute to be considered a match.
	MaxValue int64 `mapstructure:"max_value"`
}

NumericAttributeCfg holds the configurable settings to create a numeric attribute filter sampling policy evaluator.

type OTTLConditionCfg

type OTTLConditionCfg struct {
	ErrorMode           ottl.ErrorMode `mapstructure:"error_mode"`
	SpanConditions      []string       `mapstructure:"span"`
	SpanEventConditions []string       `mapstructure:"spanevent"`
}

OTTLConditionCfg holds the configurable setting to create a OTTL condition filter sampling policy evaluator.

type PolicyCfg

type PolicyCfg struct {

	// Configs for defining composite policy
	CompositeCfg CompositeCfg `mapstructure:"composite"`
	// Configs for defining and policy
	AndCfg AndCfg `mapstructure:"and"`
	// contains filtered or unexported fields
}

PolicyCfg holds the common configuration to all policies.

type PolicyType

type PolicyType string

PolicyType indicates the type of sampling policy.

const (
	// AlwaysSample samples all traces, typically used for debugging.
	AlwaysSample PolicyType = "always_sample"
	// Probabilistic samples a given percentage of traces.
	Probabilistic PolicyType = "probabilistic"
	// StatusCode sample traces that have a given status code.
	StatusCode PolicyType = "status_code"
	// StringAttribute sample traces that an attribute, of type string, matching
	// one of the listed values.
	StringAttribute PolicyType = "string_attribute"
	// RateLimiting allows all traces until the specified limits are satisfied.
	RateLimiting PolicyType = "rate_limiting"
	// Composite allows defining a composite policy, combining the other policies in one
	Composite PolicyType = "composite"
	// And allows defining a And policy, combining the other policies in one
	And PolicyType = "and"
	// TraceState sample traces with specified values by the given key
	TraceState PolicyType = "trace_state"
	// BooleanAttribute sample traces having an attribute, of type bool, that matches
	// the specified boolean value [true|false].
	BooleanAttribute PolicyType = "boolean_attribute"
	// OTTLCondition sample traces which match user provided OpenTelemetry Transformation Language
	// conditions.
	OTTLCondition PolicyType = "ottl_condition"

	// TransparencyAttribute sample traces based on their tilt checks and service dependency paths
	TransparencyAttribute PolicyType = "transparency_attribute"
)

type ProbabilisticCfg

type ProbabilisticCfg struct {
	// HashSalt allows one to configure the hashing salts. This is important in scenarios where multiple layers of collectors
	// have different sampling rates: if they use the same salt all passing one layer may pass the other even if they have
	// different sampling rates, configuring different salts avoids that.
	HashSalt string `mapstructure:"hash_salt"`
	// SamplingPercentage is the percentage rate at which traces are going to be sampled. Defaults to zero, i.e.: no sample.
	// Values greater or equal 100 are treated as "sample all traces".
	SamplingPercentage float64 `mapstructure:"sampling_percentage"`
}

ProbabilisticCfg holds the configurable settings to create a probabilistic sampling policy evaluator.

type RateAllocationCfg

type RateAllocationCfg struct {
	Policy  string `mapstructure:"policy"`
	Percent int64  `mapstructure:"percent"`
}

RateAllocationCfg used within composite policy

type RateLimitingCfg

type RateLimitingCfg struct {
	// SpansPerSecond sets the limit on the maximum nuber of spans that can be processed each second.
	SpansPerSecond int64 `mapstructure:"spans_per_second"`
}

RateLimitingCfg holds the configurable settings to create a rate limiting sampling policy evaluator.

type StatusCodeCfg

type StatusCodeCfg struct {
	StatusCodes []string `mapstructure:"status_codes"`
}

StatusCodeCfg holds the configurable settings to create a status code filter sampling policy evaluator.

type StringAttributeCfg

type StringAttributeCfg struct {
	// Tag that the filter is going to be matching against.
	Key string `mapstructure:"key"`
	// Values indicate the set of values or regular expressions to use when matching against attribute values.
	// StringAttribute Policy will apply exact value match on Values unless EnabledRegexMatching is true.
	Values []string `mapstructure:"values"`
	// EnabledRegexMatching determines whether match attribute values by regexp string.
	EnabledRegexMatching bool `mapstructure:"enabled_regex_matching"`
	// CacheMaxSize is the maximum number of attribute entries of LRU Cache that stores the matched result
	// from the regular expressions defined in Values.
	// CacheMaxSize will not be used if EnabledRegexMatching is set to false.
	CacheMaxSize int `mapstructure:"cache_max_size"`
	// InvertMatch indicates that values or regular expressions must not match against attribute values.
	// If InvertMatch is true and Values is equal to 'acme', all other values will be sampled except 'acme'.
	// Also, if the specified Key does not match on any resource or span attributes, data will be sampled.
	InvertMatch bool `mapstructure:"invert_match"`
}

StringAttributeCfg holds the configurable settings to create a string attribute filter sampling policy evaluator.

type TraceStateCfg

type TraceStateCfg struct {
	// Tag that the filter is going to be matching against.
	Key string `mapstructure:"key"`
	// Values indicate the set of values to use when matching against trace_state values.
	Values []string `mapstructure:"values"`
}