tagger

README

package tagger

The Tagger is the central source of truth for client-side entity tagging. It runs Collectors that detect entities and collect their tags. Tags are then stored in memory (by the TagStore) and can be queried by the tagger.Tag() method. Calling once tagger.Init() after the config package is ready is needed to enable collection.

The package methods use a common defaultTagger object, but we can create a custom Tagger object for testing.

The package will implement an IPC mechanism (a server and a client) to allow other agents to query the DefaultTagger and avoid duplicating the information in their process. Switch between local and client mode will be done via a build flag.

The tagger is also available to python checks via the tagger module exporting the get_tags() function. This function accepts the same arguments as the Go Tag() function, and returns an empty list on errors.

Collector

A Collector connects to a single information source and pushes TagInfo structs to a channel, towards the Tagger. It can either run in streaming mode, pull or fetchonly mode, depending of what's most efficient for the data source:

Streamer

The DockerCollector runs in stream mode as it collects events from the docker daemon and reacts to them, sending updates incrementally.

Puller

The KubernetesCollector will run in pull mode as it needs to query and filter a full entity list every time. It will only push updates to the store though, by keeping an internal state of the latest revision.

FetchOnly

The ECSCollector does not push updates to the Store by itself, but is only triggered on cache misses. As tasks don't change after creation, there's no need for periodic pulling. It is designed to run alongside DockerCollector, that will trigger deletions in the store.

TagStore

The TagStore reads TagInfo structs and stores them in a in-memory cache. Cache invalidation is triggered by the collectors (or source) by either:

• sending new tags for the same Entity, all the tags from this Source will be removed and replaced by the new tags
• sending a TagInfo with DeleteEntity set, all the entries for this entity (including from other sources) will be deleted when prune() is called.

The deletions are batched so that if two sources send coliding add and delete messages, the delete eventually wins.

Tagger

The Tagger handles the glue between Collectors and TagStore and the cache miss logic. If the tags from the TagStore are missing some sources, they will be manually queried in a block way, and the cache will be updated.

For convenience, the package creates a defaultTagger object that is used when calling the tagger.Tag() method.

               +-----------+
               | Collector |
               +---+-------+
                   |
                   |
+--------+      +--+-------+       +-------------+
|  User  <------+  Tagger  +-------> IPC handler |
|packages|      +--+-----^-+       +-------------+
+--------+         |     |
                   |     |
                +--v-----+-+
                | TagStore |
                +----------+

Documentation

Index

• func Init() error
• func Stop() error
• func Tag(entity string, highCard bool) ([]string, error)
• type Tagger
  • func (t *Tagger) Init(catalog collectors.Catalog) error
  • func (t *Tagger) Stop() error
  • func (t *Tagger) Tag(entity string, highCard bool) ([]string, error)

Functions

func Init

func Init() error

Init must be called once config is available, call it in your cmd

func Stop

func Stop() error

Stop queues a stop signal to the defaultTagger

func Tag

func Tag(entity string, highCard bool) ([]string, error)

Tag queries the defaultTagger to get entity tags from cache or sources

Types

type Tagger

type Tagger struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

Tagger is the entry class for entity tagging. It holds collectors, memory store and handles the query logic. One can use the package methods to use the default tagger instead of instanciating one.

func (*Tagger) Init

func (t *Tagger) Init(catalog collectors.Catalog) error

Init goes through a catalog and tries to detect which are relevant for this host. It then starts the collection logic and is ready for requests.

func (*Tagger) Stop

func (t *Tagger) Stop() error

Stop queues a shutdown of Tagger

func (*Tagger) Tag

func (t *Tagger) Tag(entity string, highCard bool) ([]string, error)

Tag returns tags for a given entity. If highCard is false, high cardinality tags are left out.