tagger

README

package tagger

The Tagger is the central source of truth for client-side entity tagging. It runs Collectors that detect entities and collect their tags. Tags are then stored in memory (by the TagStore) and can be queried by the tagger.Tag() method. Calling once tagger.Init() after the config package is ready is needed to enable collection.

The package methods use a common defaultTagger object, but we can create a custom Tagger object for testing.

The package will implement an IPC mechanism (a server and a client) to allow other agents to query the DefaultTagger and avoid duplicating the information in their process. Switch between local and client mode will be done via a build flag.

The tagger is also available to python checks via the tagger module exporting the get_tags() function. This function accepts the same arguments as the Go Tag() function, and returns an empty list on errors.

Collector

A Collector connects to a single information source and pushes TagInfo structs to a channel, towards the Tagger. It can either run in streaming mode, pull or fetchonly mode, depending of what's most efficient for the data source:

Streamer

The DockerCollector runs in stream mode as it collects events from the docker daemon and reacts to them, sending updates incrementally.

Puller

The KubernetesCollector will run in pull mode as it needs to query and filter a full entity list every time. It will only push updates to the store though, by keeping an internal state of the latest revision.

FetchOnly

The ECSCollector does not push updates to the Store by itself, but is only triggered on cache misses. As tasks don't change after creation, there's no need for periodic pulling. It is designed to run alongside DockerCollector, that will trigger deletions in the store.

TagStore

The TagStore reads TagInfo structs and stores them in a in-memory cache. Cache invalidation is triggered by the collectors (or source) by either:

• sending new tags for the same Entity, all the tags from this Source will be removed and replaced by the new tags
• sending a TagInfo with DeleteEntity set, all the entries for this entity (including from other sources) will be deleted when prune() is called.

The deletions are batched so that if two sources send coliding add and delete messages, the delete eventually wins.

TagCardinality

TagInfo accepts and store tags that have different cardinality. TagCardinality can be:

• *LowCardinality: in the host count order of magnitude
• *OrchestratorCardinality:tags that change value for each pod or task
• *HighCardinality: typically tags that change value for each web request, user agent, container, etc.

Tagger

The Tagger handles the glue between Collectors and TagStore and the cache miss logic. If the tags from the TagStore are missing some sources, they will be manually queried in a block way, and the cache will be updated.

For convenience, the package creates a defaultTagger object that is used when calling the tagger.Tag() method.

               +-----------+
               | Collector |
               +---+-------+
                   |
                   |
+--------+      +--+-------+       +-------------+
|  User  <------+  Tagger  +-------> IPC handler |
|packages|      +--+-----^-+       +-------------+
+--------+         |     |
                   |     |
                +--v-----+-+
                | TagStore |
                +----------+

Documentation

Index

• Variables
• func GetEntityHash(entity string) string
• func Init()
• func List(cardinality collectors.TagCardinality) response.TaggerListResponse
• func Stop() error
• func Tag(entity string, cardinality collectors.TagCardinality) ([]string, error)
• type Tagger
  • func (t *Tagger) GetEntityHash(entity string) string
  • func (t *Tagger) Init(catalog collectors.Catalog)
  • func (t *Tagger) List(cardinality collectors.TagCardinality) response.TaggerListResponse
  • func (t *Tagger) Stop() error
  • func (t *Tagger) Tag(entity string, cardinality collectors.TagCardinality) ([]string, error)

Variables

var ChecksCardinality collectors.TagCardinality

ChecksCardinality defines the cardinality of tags we should send for check metrics this can still be overridden when calling get_tags in python checks.

var DogstatsdCardinality collectors.TagCardinality

DogstatsdCardinality defines the cardinality of tags we should send for metrics from dogstatsd.

Functions

func GetEntityHash

func GetEntityHash(entity string) string

GetEntityHash returns the hash for the tags associated with the given entity

func Init

func Init()

Init must be called once config is available, call it in your cmd

func List

func List(cardinality collectors.TagCardinality) response.TaggerListResponse

List the content of the defaulTagger

func Stop

func Stop() error

Stop queues a stop signal to the defaultTagger

func Tag

func Tag(entity string, cardinality collectors.TagCardinality) ([]string, error)

Tag queries the defaultTagger to get entity tags from cache or sources. It can return tags at high cardinality (with tags about individual containers), or at orchestrator cardinality (pod/task level)

Types

type Tagger

type Tagger struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

Tagger is the entry class for entity tagging. It holds collectors, memory store and handles the query logic. One can use the package methods to use the default tagger instead of instantiating one.

func (*Tagger) GetEntityHash

func (t *Tagger) GetEntityHash(entity string) string

GetEntityHash returns the tags hash of an entity

func (*Tagger) Init

func (t *Tagger) Init(catalog collectors.Catalog)

Init goes through a catalog and tries to detect which are relevant for this host. It then starts the collection logic and is ready for requests.

func (*Tagger) List

func (t *Tagger) List(cardinality collectors.TagCardinality) response.TaggerListResponse

List the content of the tagger

func (*Tagger) Stop

func (t *Tagger) Stop() error

Stop queues a shutdown of Tagger

func (*Tagger) Tag

func (t *Tagger) Tag(entity string, cardinality collectors.TagCardinality) ([]string, error)

Tag returns tags for a given entity