activity

README

Events telemetry

Types of telemetry in Rill

In Rill, we have two kinds of telemetry:

• OpenTelemetry: Used for emitting and collecting traditional logging, tracing, and metrics in a hosted setting. Encapsulated in the runtime/pkg/observability package.
• Events telemetry: Used for event telemetry, such as user behavior, browser errors, system metrics, billing events, etc. Encapsulated in runtime/pkg/activity (this package).

Why do we have a separate implementation for events telemetry? Two reasons:

1. The event telemetry is collected from all combinations of frontend/CLI/backend services and local/cloud-hosted environments. At the time of writing, OpenTelemetry is not well-suited for capturing data across such disparate environments.
2. At the time of writing, OpenTelemetry does not have mature abstractions for capturing event data.

Sources of events telemetry and how it propagates

Event telemetry propagate in various ways depending on the source service and environment:

1. The cloud admin server sends events directly to Kafka.
2. The cloud runtime sends events directly to Kafka.
3. The cloud admin UI sends events to the cloud admin server, which proxies them to Kafka.
4. The local CLI sends events to Rill's intake API.
5. The local UI sends events to the local server hosted by the CLI, which proxies them to Rill's intake API.
6. The local runtime sends events to Rill's intake API (using the same client as the CLI).

Required event format

We do not enforce strict schemas for different event types, but we do require all events to have the following format:

{
    "event_id": "<generated unique ID for deduplication>",
    "event_time": "<timestamp in ISO8601 format>",
    "event_type": "<high-level event type, such as 'behavioral', 'metric', ...>",
    "event_name": "<unique identifer of the event within the event type, such as 'login_start' or 'duckdb_estimated_size'>"
    // Other event-specific attributes
}

Events that do not contain these fields cannot be generated or proxied through the client.

Common event types, names and attributes

The event.go file in this package contains constants for common event types/names and common attributes.

Documentation

Index

• Constants
• func SetAttributes(ctx context.Context, attrs ...attribute.KeyValue) context.Context
• type Client
  • func NewClient(sink Sink, logger *zap.Logger) *Client
  • func NewNoopClient() *Client
  • func (c *Client) Close(ctx context.Context) error
  • func (c *Client) Record(ctx context.Context, typ, name string, extraAttrs ...attribute.KeyValue)
  • func (c *Client) RecordBehavioralLegacy(name string, extraAttrs ...attribute.KeyValue)
  • func (c *Client) RecordMetric(ctx context.Context, name string, value float64, attrs ...attribute.KeyValue)
  • func (c *Client) RecordRaw(data map[string]any) error
  • func (c *Client) With(attrs ...attribute.KeyValue) *Client
  • func (c *Client) WithInstallID(installID string) *Client
  • func (c *Client) WithIsDev() *Client
  • func (c *Client) WithServiceName(serviceName string) *Client
  • func (c *Client) WithServiceVersion(number, commit string) *Client
  • func (c *Client) WithUserID(userID string) *Client
• type Event
  • func (e Event) MarshalJSON() ([]byte, error)
• type IntakeSinkOptions
• type Sink
  • func NewFilterSink(sink Sink, fn func(Event) bool) Sink
  • func NewIntakeSink(logger *zap.Logger, opts IntakeSinkOptions) Sink
  • func NewKafkaSink(brokers, topic string, logger *zap.Logger) (Sink, error)
  • func NewLoggerSink(logger *zap.Logger, level zapcore.Level) Sink
  • func NewNoopSink() Sink

Constants

const (
	EventTypeLog        = "log"
	EventTypeMetric     = "metric"
	EventTypeBehavioral = "behavioral"
)

Constants for common event types.

const (
	AttrKeyServiceName    = "service_name"
	AttrKeyServiceVersion = "service_version"
	AttrKeyServiceCommit  = "service_commit"
	AttrKeyIsDev          = "is_dev"
	AttrKeyInstallID      = "install_id"
	AttrKeyUserID         = "user_id"
)

Constants for common event attribute keys.

const (
	BehavioralEventInstallSuccess         = "install-success"
	BehavioralEventAppStart               = "app-start"
	BehavioralEventLoginStart             = "login-start"
	BehavioralEventLoginSuccess           = "login-success"
	BehavioralEventDeployStart            = "deploy-start"
	BehavioralEventDeploySuccess          = "deploy-success"
	BehavioralEventGithubConnectedStart   = "ghconnected-start"
	BehavioralEventGithubConnectedSuccess = "ghconnected-success"
	BehavioralEventDataAccessStart        = "dataaccess-start"
	BehavioralEventDataAccessSuccess      = "dataaccess-success"
)

Constants for event names of type EventTypeBehavioral. Note: This list is not exhaustive. Proxied events may contain other names.

Functions

func SetAttributes

func SetAttributes(ctx context.Context, attrs ...attribute.KeyValue) context.Context

Types

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client constructs telemetry events and sends them to a sink.

func NewClient

func NewClient(sink Sink, logger *zap.Logger) *Client

NewClient creates a base telemetry client that sends events to the provided sink. The client will close the sink when Close is called.

func NewNoopClient

func NewNoopClient() *Client

NewNoopClient creates a client that discards all events.

func (*Client) Close

func (c *Client) Close(ctx context.Context) error

Close the client. Also closes the sink passed to NewClient.

func (*Client) Record

func (c *Client) Record(ctx context.Context, typ, name string, extraAttrs ...attribute.KeyValue)

Record sends a generic telemetry event with the provided event type and name.

func (*Client) RecordBehavioralLegacy

func (c *Client) RecordBehavioralLegacy(name string, extraAttrs ...attribute.KeyValue)

EmitBehavioral sends a telemetry event of type "behavioral" with the provided name and attributes. The event additionally has all the attributes associated with out legacy behavioral events. It will panic if all of WithServiceName, WithServiceVersion, WithInstallID and WithUserID have not been called on the client.

func (*Client) RecordMetric

func (c *Client) RecordMetric(ctx context.Context, name string, value float64, attrs ...attribute.KeyValue)

RecordMetric sends a telemetry event of type "metric" with the provided name and value.

func (*Client) RecordRaw

func (c *Client) RecordRaw(data map[string]any) error

RecordRaw proxies a raw event represented as a map to the client's sink. It does not enrich the provided event with any of the client's contextual attributes. It returns an error if the event does not contain the required fields (see the Event type for required fields).

func (*Client) With

func (c *Client) With(attrs ...attribute.KeyValue) *Client

With returns a copy of the client that will set the provided attributes on all events.

func (*Client) WithInstallID

func (c *Client) WithInstallID(installID string) *Client

WithInstallID returns a copy of the client with an attribute set for AttrKeyInstallID.

func (*Client) WithIsDev

func (c *Client) WithIsDev() *Client

WithIsDev returns a copy of the client with an attribute set for AttrKeyIsDev.

func (*Client) WithServiceName

func (c *Client) WithServiceName(serviceName string) *Client

WithServiceName returns a copy of the client with an attribute set for AttrKeyServiceName.

func (*Client) WithServiceVersion

func (c *Client) WithServiceVersion(number, commit string) *Client

WithServiceVersion returns a copy of the client with attributes set for AttrKeyServiceVersion and AttrKeyServiceCommit.

func (*Client) WithUserID

func (c *Client) WithUserID(userID string) *Client

WithUserID returns a copy of the client with an attribute set for AttrKeyUserID.

type Event

type Event struct {
	EventID   string
	EventTime time.Time
	EventType string
	EventName string
	Data      map[string]any
}

Event is a telemetry event. It consists of a few required fields that are common to all events and a payload of type-specific data. All the common fields are prefixed with "event_" to avoid conflicts with the payload data.

func (Event) MarshalJSON

func (e Event) MarshalJSON() ([]byte, error)

type IntakeSinkOptions

type IntakeSinkOptions struct {
	IntakeURL      string
	IntakeUser     string
	IntakePassword string
	BufferSize     int
	SinkInterval   time.Duration
}

IntakeSinkOptions provides options for NewIntakeSink.

type Sink

type Sink interface {
	Emit(event Event) error
	Close()
}

Sink is a destination for sending telemetry events.

func NewFilterSink

func NewFilterSink(sink Sink, fn func(Event) bool) Sink

NewFilterSink returns a sink that filters events based on the provided filter function. Only events for which the filter function returns true will be emitted to the wrapped sink.

func NewIntakeSink

func NewIntakeSink(logger *zap.Logger, opts IntakeSinkOptions) Sink

NewIntakeSink creates a new sink that sends events to the Rill intake API.

func NewKafkaSink

func NewKafkaSink(brokers, topic string, logger *zap.Logger) (Sink, error)

NewKafkaSink returns a sink that sends events to a Kafka topic.

func NewLoggerSink

func NewLoggerSink(logger *zap.Logger, level zapcore.Level) Sink

NewLoggerSink returns a sink that logs events to the given logger.

func NewNoopSink

func NewNoopSink() Sink

NewNoopSink returns a sink that drops all events.