sampler

README

Go sampler module

The Go Sampler module allows you to get Data Samples from your Go services.

gRPC interceptors

The Go library also provides a package to easily add Samplers as gRPC interceptors. This will automatically add Samplers in each gRPC method that will intercept all requests and responses.

Supported encodings

Encoding	Description
Protobuf	It can efficiently process proto.Message objects if the Protobuf message definition is provided.
Go native object	Only capable of accessing exported fields. This is a limitation of the Go language.
JSON	A string containing a JSON object.

Warning The next section is better read in the documentation page since it is post-processed to include code snippets from tests

Usage

!!! note All code snippets have error handling omitted for brevity

To add Samplers in your code that export Data Samples you first need to initialize a Provider. A Provider receives a configuration that is then used to configure and initialize all the Samplers in your application.

import (
--8<-- "./sampler/test/docs/provider_test.go:ProviderInitImport"
)

--8<-- "./sampler/test/docs/provider_test.go:ProviderInit"

To see details about the required settings and available options, see this page.

Once the Provider is initialized, you can use it to initialize Samplers. If you have registered the provider as global, you can simply call the Sampler method from anywhere in your application. If not, you will need to call the Sampler method of your provider. They both have the same signature so the following explanation works for both options.

!!! Info It is not required to first initialize and register the Provider as global before creating Samplers. If a Sampler is initialized using the global provider before a Provider is registered, it will return a stubbed Sampler with no-op methods. Once the Provider is registered, it will internally replace the no-op stub with the real Sampler. This happens transparently for the application.

import (
--8<-- "./sampler/test/docs/sampler_test.go:SamplerInitImport"
)

--8<-- "./sampler/test/docs/sampler_test.go:SamplerInit"

To see what other schemas the Go Sampler supports, check this Godoc page.

Once you have initialized the Sampler, you can call any of its methods to have it evaluate a Data Sample. It will then be evaluated by any configured Sampling Rule and exported if there is a match.

!!! Warning You need to be mindful of what methods you use to sample data. Depending on the schema provided when the Sampler is initialized, some methods will work better or faster than others.

As a rule of thumb, you want to provide a schema if you have it since this allows the `Sampler` to internally optimize how it evaluates the `Sampling Rules`. If you do not have it, a sampler configured with a `DynamicSchema` is capable of processing any type data using any of the sampling methods. See the [Godoc](https://pkg.go.dev/github.com/neblic/platform/sampler/defs) documentation for details.

--8<-- "./sampler/test/docs/sampler_test.go:SampleData"

In this example, since the Sampler was initialized with a DynamicSchema, it is best to use the method SampleJson() or SampleNative(). These sampling methods are designed to work with samples that do not have a fixed or known schema.

gRPC interceptor

If you use gRPC servers or clients in your services, you can make use of a gRPC interceptor. They will automatically create Samplers that will efficiently intercept all requests and responses.

Internally, they create Samplers with a ProtoSchema so they do not need to deserialize the Protobuf message to evaluate its contents.

To use it, you need to initialize the interceptor and provide it when initializing the gRPC connection

import (
--8<-- "./sampler/test/docs/interceptor_test.go:InterceptorInitImport"
)

--8<-- "./sampler/test/docs/interceptor_test.go:InterceptorInit"

Documentation

Overview

It implements a sampler and a global provider placeholder so samplers can be created before setting the global provider. Once the global provider is set, any prob ebuilt before setting it is initialized with the new global provider.

Index

• Constants
• func Sample(ctx context.Context, name string, schema sample.Schema, sample sample.Sample) bool
• func SetProvider(pp Provider) error
• type Option
  • func WithInitalValueDigest(location control.ComputationLocation) Option
  • func WithInitialDeterministicSamplingIn(samplingRate int32) Option
  • func WithInitialLimiterInLimit(l int32) Option
  • func WithInitialLimiterOutLimit(l int32) Option
  • func WithInitialStructDigest(location control.ComputationLocation) Option
  • func WithTags(tags ...string) Option
  • func WithUpdateStatsPeriod(p time.Duration) Option
  • func WithoutDefaultInitialConfig() Option
• type Provider
  • func NewProvider(ctx context.Context, settings Settings, opts ...ProviderOption) (Provider, error)
• type ProviderOption
  • func WithBearerAuth(token string) ProviderOption
  • func WithLogger(l logging.Logger) ProviderOption
  • func WithSamplerErrorChannel(errCh chan error) ProviderOption
  • func WithTLS() ProviderOption
• type Sampler
  • func New(name string, schema sample.Schema, opts ...Option) (Sampler, error)
• type Settings

Constants

const (
	ProducerTag = control.ProducerTag
	ConsumerTag = control.ConsumerTag
	RequestTag  = control.RequestTag
	ResponseTag = control.ResponseTag
	// DLQTag is used to tag samples that are sent to a dead letter queue
	// Automatically enables an event that notifies when a sample is sent to a DLQ
	DLQTag = control.DLQTag
)

re-exported known tags

const DLQEventName = "sample_sent_to_dlq"

Functions

func Sample

func Sample(ctx context.Context, name string, schema sample.Schema, sample sample.Sample) bool

Sample samples the given data sample using the given sampler name and schema. If a sampler with the given name is not registered, it will be created. Note that if there is any error creating the sampler, it wont't be reported and the sample will be silently discarded.

func SetProvider

func SetProvider(pp Provider) error

SetProvider sets the global provider. It can only be set once. It may return sampler initialization errors because the creation of samplers created with the default global provider are deferred until it is set

Types

type Option

type Option interface {
	// contains filtered or unexported methods
}

func WithInitalValueDigest

func WithInitalValueDigest(location control.ComputationLocation) Option

WithInitalValueDigest enables the computation of value digest. Location defines where this computation must take place (in the sampler of in the collector). In case of computing the digest in the collector, the raw samples are exported (that can have an impact in the sampler performance)

func WithInitialDeterministicSamplingIn

func WithInitialDeterministicSamplingIn(samplingRate int32) Option

WithInitialDeterministicSamplingIn defines a deterministic sampling strategy which will be applied when a sample is received and before processing it in any way (e.g. before determining if a sample belongs to a stream which would require parsing it and evaluating the stream rules). Sampling is performed after the input limiter has been applied. This configuration is only used the first time a sampler is registered with a server, posterior executions will use the configuration stored in the server and the provided configuration will be ignored.

func WithInitialLimiterInLimit

func WithInitialLimiterInLimit(l int32) Option

WithInitialLimiterInLimit sets the initial limiter in rate limit. This configuration is only used the first time a sampler is registered with a server, posterior executions will use the configuration stored in the server and the provided configuration will be ignored.

func WithInitialLimiterOutLimit

func WithInitialLimiterOutLimit(l int32) Option

WithInitialLimiterInLimit sets the initial limiter in rate limit. This configuration is only used the first time a sampler is registered with a server, posterior executions will use the configuration stored in the server and the provided configuration will be ignored.

func WithInitialStructDigest

func WithInitialStructDigest(location control.ComputationLocation) Option

WithInitalStructDigest enables the computation of struct digest. Location defines where this computation must take place (in the sampler of in the collector). In case of computing the digest in the collector, the raw samples are exported (that can have an impact in the sampler performance)

func WithTags

func WithTags(tags ...string) Option

WithTags can be used to specify tags to classify the Sampler or to specify its type. There are known tags that can be used to identify the Sampler type and may be used by the platform to provide additional functionality automnatically. See the tags constants defined in this file.

func WithUpdateStatsPeriod

func WithUpdateStatsPeriod(p time.Duration) Option

WithUpdateStatsPeriod specifies the period to send sampler stats to server If the provided period is less than a second, it will be set to 1 second.

func WithoutDefaultInitialConfig

func WithoutDefaultInitialConfig() Option

WithoutDefaultInitialConfig avoids setting the default 'all' stream and digest. This configuration is only used the first time a sampler is registered with a server, posterior executions will use the configuration stored in the server and the provided configuration will be ignored.

type Provider

type Provider interface {
	// Sampler creates a new sampler with the specified schema. It currently supports Dynamic and Proto schemas.
	// * A Dynamic schema does not enforce any structure to the sampled data and is compatible with all the Sample*()
	// methods. The downside, is that it is slower than the Proto schema since it needs to determine at runtime the sampled
	// data format.
	// * A Proto schema requires the caller to provide a proto message (type proto.Message) to define the sampler schema.
	// All sampled data is expected to be provided as proto messages with the sampler.SampleProto() method, and it should
	// be the same type as the one provided when defining the sampler schema.
	Sampler(name string, schema sample.Schema, opts ...Option) (Sampler, error)
}

Provider defines a sampler provider object capable of creating new samplers.

func NewProvider

func NewProvider(ctx context.Context, settings Settings, opts ...ProviderOption) (Provider, error)

NewProvider creates a new sampler provider capable of creating new samplers. After initializing a provider, it is recommended to set it as the global provider for easier access.

provider := NewProvider(...)
sampler.SetProvider(provider)

Then, to create new samplers do:

sampler, err := sampler.New(...)

which is equivalent to creating a sampler using the global provider:

sampler, err := provider.Sampler(...)

type ProviderOption

type ProviderOption interface {
	// contains filtered or unexported methods
}

func WithBearerAuth

func WithBearerAuth(token string) ProviderOption

WithBearerAuth sets authorization based on a Bearer token

func WithLogger

func WithLogger(l logging.Logger) ProviderOption

WithLogger provides a logger instance to log the Provider and Sampler activity

func WithSamplerErrorChannel

func WithSamplerErrorChannel(errCh chan error) ProviderOption

WithErrorChannel received a channel where Sampler errors will be sent. The avoid blocking the Sampler, it won't block if the channel is full so it is responsibility of the provider to ensure the channel has enough buffer to avoid losing errors.

func WithTLS

func WithTLS() ProviderOption

WithTLS enables TLS connections with the server

type Sampler

type Sampler interface {
	// Sample samples the given data sample. Returns true if the sample has been exported.
	Sample(ctx context.Context, sample sample.Sample) bool
	// Close closes all Sampler connections with the Control and Data planes. Once closed,
	// the Sampler can't be reused and none of its methods can be called.
	Close() error
}

Sampler defines the sampler public interface

func New

func New(name string, schema sample.Schema, opts ...Option) (Sampler, error)

New creates a new sampler using the global provider. Since it uses the global sampler provider, it is necessary to set one for the samplers to work. See sampler.SetProvider(...) method for more details.

type Settings

type Settings struct {
	// ResourceName sets the resource name common to all samplers created by this provider
	// For example, the service or the data pipeline operator name
	ResourceName string
	// ControlServerAddr specifies the address where the control server is listening at.
	// Format addr:port
	ControlServerAddr string
	// ControlServerAddr specifies the address where the data server is listening at.
	// By default, it sends the data samples encoded as OTLP logs following the OTLP gRPC
	// protocol, so it works with any OTLP gRPC logs compatible collector.
	// Format addr:port
	DataServerAddr string
}