tagger

README

package tagger

The Tagger is the central source of truth for client-side entity tagging. It runs Collectors that detect entities and collect their tags. Tags are then stored in memory (by the TagStore) and can be queried by the tagger.Tag() method. Calling once tagger.Init() after the config package is ready is needed to enable collection.

The package methods use a common defaultTagger object, but we can create a custom Tagger object for testing.

The package will implement an IPC mechanism (a server and a client) to allow other agents to query the DefaultTagger and avoid duplicating the information in their process. Switch between local and client mode will be done via a build flag.

The tagger is also available to python checks via the tagger module exporting the get_tags() function. This function accepts the same arguments as the Go Tag() function, and returns an empty list on errors.

Collector

A Collector connects to a single information source and pushes TagInfo structs to a channel, towards the Tagger. It can either run in streaming mode, pull or fetchonly mode, depending of what's most efficient for the data source:

Streamer

The DockerCollector runs in stream mode as it collects events from the docker daemon and reacts to them, sending updates incrementally.

Puller

The KubernetesCollector will run in pull mode as it needs to query and filter a full entity list every time. It will only push updates to the store though, by keeping an internal state of the latest revision.

FetchOnly

The ECSCollector does not push updates to the Store by itself, but is only triggered on cache misses. As tasks don't change after creation, there's no need for periodic pulling. It is designed to run alongside DockerCollector, that will trigger deletions in the store.

TagStore

The TagStore reads TagInfo structs and stores them in a in-memory cache. Cache invalidation is triggered by the collectors (or source) by either:

• sending new tags for the same Entity, all the tags from this Source will be removed and replaced by the new tags
• sending a TagInfo with DeleteEntity set, all the tags collected for this entity by the specified source (but not others) will be deleted when prune() is called.

TagCardinality

TagInfo accepts and store tags that have different cardinality. TagCardinality can be:

• LowCardinality: in the host count order of magnitude
• OrchestratorCardinality: tags that change value for each pod or task
• HighCardinality: typically tags that change value for each web request, user agent, container, etc.

Tagger

The Tagger handles the glue between Collectors and TagStore and the cache miss logic. If the tags from the TagStore are missing some sources, they will be manually queried in a block way, and the cache will be updated.

For convenience, the package creates a defaultTagger object that is used when calling the tagger.Tag() method.

               +-----------+
               | Collector |
               +---+-------+
                   |
                   |
+--------+      +--+-------+       +-------------+
|  User  <------+  Tagger  +-------> IPC handler |
|packages|      +--+-----^-+       +-------------+
+--------+         |     |
                   |     |
                +--v-----+-+
                | TagStore |
                +----------+

Documentation

Index

• Variables
• func AccumulateTagsFor(entity string, cardinality collectors.TagCardinality, tb tagset.TagAccumulator) error
• func AgentTags(cardinality collectors.TagCardinality) ([]string, error)
• func EnrichTags(tb tagset.TagAccumulator, origin string, k8sOriginID string, ...)
• func GetEntity(entityID string) (*types.Entity, error)
• func GetEntityHash(entity string, cardinality collectors.TagCardinality) string
• func Init() error
• func List(cardinality collectors.TagCardinality) response.TaggerListResponse
• func OrchestratorScopeTag() ([]string, error)
• func OrchestratorScopeTagBuilder(tb tagset.TagAccumulator) error
• func ResetCaptureTagger()
• func SetCaptureTagger(tagger Tagger)
• func SetDefaultTagger(tagger Tagger)
• func StandardTags(entity string) ([]string, error)
• func Stop() error
• func Tag(entity string, cardinality collectors.TagCardinality) ([]string, error)
• func TagWithHash(entity string, cardinality collectors.TagCardinality) ([]string, string, error)
• type Tagger
  • func GetDefaultTagger() Tagger

Variables

var ChecksCardinality collectors.TagCardinality

ChecksCardinality defines the cardinality of tags we should send for check metrics this can still be overridden when calling get_tags in python checks.

var DogstatsdCardinality collectors.TagCardinality

DogstatsdCardinality defines the cardinality of tags we should send for metrics from dogstatsd.

Functions

func AccumulateTagsFor

func AccumulateTagsFor(entity string, cardinality collectors.TagCardinality, tb tagset.TagAccumulator) error

AccumulateTagsFor queries the defaultTagger to get entity tags from cache or sources and appends them to the TagAccumulator. It can return tags at high cardinality (with tags about individual containers), or at orchestrator cardinality (pod/task level).

func AgentTags

func AgentTags(cardinality collectors.TagCardinality) ([]string, error)

AgentTags returns the agent tags It relies on the container provider utils to get the Agent container ID

func EnrichTags

func EnrichTags(tb tagset.TagAccumulator, origin string, k8sOriginID string, cardinalityName string)

EnrichTags extends a tag list with origin detection tags NOTE(remy): it is not needed to sort/dedup the tags anymore since after the enrichment, the metric and its tags is sent to the context key generator, which is taking care of deduping the tags while generating the context key.

func GetEntity

func GetEntity(entityID string) (*types.Entity, error)

GetEntity returns the hash for the provided entity id.

func GetEntityHash

func GetEntityHash(entity string, cardinality collectors.TagCardinality) string

GetEntityHash returns the hash for the tags associated with the given entity Returns an empty string if the tags lookup fails

func Init

func Init() error

Init must be called once config is available, call it in your cmd

func List

func List(cardinality collectors.TagCardinality) response.TaggerListResponse

List the content of the defaulTagger

func OrchestratorScopeTag

func OrchestratorScopeTag() ([]string, error)

OrchestratorScopeTag queries tags for orchestrator scope (e.g. task_arn in ECS Fargate)

func OrchestratorScopeTagBuilder

func OrchestratorScopeTagBuilder(tb tagset.TagAccumulator) error

OrchestratorScopeTagBuilder queries tags for orchestrator scope (e.g. task_arn in ECS Fargate) and appends them to the TagAccumulator

func ResetCaptureTagger

func ResetCaptureTagger()

ResetCaptureTagger resets the capture tagger to nil

func SetCaptureTagger

func SetCaptureTagger(tagger Tagger)

SetCaptureTagger sets the tagger to be used when replaying a capture

func SetDefaultTagger

func SetDefaultTagger(tagger Tagger)

SetDefaultTagger sets the global Tagger instance

func StandardTags

func StandardTags(entity string) ([]string, error)

StandardTags queries the defaultTagger to get entity standard tags (env, version, service) from cache or sources.

func Stop

func Stop() error

Stop queues a stop signal to the defaultTagger

func Tag

func Tag(entity string, cardinality collectors.TagCardinality) ([]string, error)

Tag queries the captureTagger (for replay scenarios) or the defaultTagger. It can return tags at high cardinality (with tags about individual containers), or at orchestrator cardinality (pod/task level).

func TagWithHash

func TagWithHash(entity string, cardinality collectors.TagCardinality) ([]string, string, error)

TagWithHash is similar to Tag but it also computes and returns the hash of the tags found

Types

type Tagger

type Tagger interface {
	Init() error
	Stop() error

	Tag(entity string, cardinality collectors.TagCardinality) ([]string, error)
	AccumulateTagsFor(entity string, cardinality collectors.TagCardinality, tb tagset.TagAccumulator) error
	Standard(entity string) ([]string, error)
	List(cardinality collectors.TagCardinality) response.TaggerListResponse
	GetEntity(entityID string) (*types.Entity, error)

	Subscribe(cardinality collectors.TagCardinality) chan []types.EntityEvent
	Unsubscribe(ch chan []types.EntityEvent)
}

Tagger is an interface for transparent access to both localTagger and remoteTagger.

func GetDefaultTagger

func GetDefaultTagger() Tagger

GetDefaultTagger returns the global Tagger instance