turtlefinder

README

Turtlefinder Container Engine Discovery

🐢🐘 "Turtles all the way down" (Wikipedia)

turtlefinder is a Go module that discovers various container engines in a Linux host, including container engines that have been put into containers. If you consider such configurations to be rarer than rare, then please take a look at KinD (Kubernetes-in-Docker) as well as Docker Desktop on WSL2 (Windows Subsystem for Linux).

It supports the following container engines:

• Docker
• containerd (both native API as well as CRI Event PLEG API)
• CRI-O (CRI Event PLEG API)
• podman (via Docker-compatible API only)

The turtlefinder package originates from Ghostwire (part of the Edgeshark project) and has been carved out in order to foster easy reuse in other projects without the need for importing the full Ghostwire module.

Usage: One to Bind Find Them All

Simply create a single turtlefinder. No need to create individual “containerizers” (lxkns' word for things that discover the current container workload of an container engine) and then stitching them together, such as when dealing with Docker and containerd simultaneously.

enginectx, cancel := context.WithCancel(context.Background())
containerizer := turtlefinder.New(
  func() context.Context { return enginectx },
  /* options... */
)

Whenever a turtlefinder finds a new container engine process, it tries to talk sense to it and discover and track its container workload. In order to shut down such engine workload background tracking (watching) a turtlefinder expects us to supply it with a suitable “background” context; one we preferably have control over. So this is, what the first parameter to New is.

For further options, please refer to the module documentation.

Project Structure

The "Edgeshark" project consist of several repositories:

• Edgeshark Hub repository
• G(h)ostwire discovery service
• Packetflix packet streaming service
• Containershark Extcap plugin for Wireshark
• support modules:
  • 🖝 Turtlefinder 🖜
  • csharg (CLI)
  • mobydig
  • ieddata

Mode of Operation

Finding container engines works in principle as follows:

1. detect long-running engines (also commonly refered to as "demons"):
   1. scan the process tree for processes with known names (such as dockerd, containerd, cri-o, et cetera). The well-known process names are supplied by a set of built-in "detectors" in form of sub-packages of the github.com/siemens/turtlefinder/detector package.
   2. scan matching processes for file descriptors referencing listening unix domain sockets: we assume them to be potential container engine API endpoints.
2. detect socket-activated engines:
   1. scan the process tree for socket-activating processes with known names, especially systemd. Again, the well-known process names are supplied by a set of build-in socket-activator detectors in form of sub-packages of the github.com/siemens/turtlefinder/activator package.
   2. scan matching processes for file descriptors referencing listening unix domain sockets with well-known suffixes, such as podman.sock. While this is slightly less efficient compared with directly using well-known absolute socket API paths, our approach is much more powerful as it finds suffix-matching API endpoints even in containers. This scanning happens only for a newly found socket activator or when we detect a change in the socket activator's socket configuration (such as after a configuration reload).
   3. activate the services ("don't call them demons") behind the API sockets and wait for the demonsservice processes to appear, before proceeding with talking to these endpoints. (The rationale is that we need the engine PIDs for the turtlefinder hierarchy detection to work.)
3. try to talk sense to the API endpoints found; this won't be always the case, such as when we trip on metrics endpoints, and other strange endpoints. Where we succeed, we add the engine found to our list of engines to watch.

Additionally, we also do engine pruning when we can't find a particular engine process anymore after a more recent process tree scan.

Furthermore, we do some fancy things during workload discovery in order to figure out how container engines might have been stuck into containers of another container engine: that is, the hierarchy of container engines. This is especially useful for such system configurations as KinD clusters and Docker Desktop on WSL2.

VSCode Tasks

The included turtlefinder.code-workspace defines the following tasks:

• View Go module documentation task: installs pkgsite, if not done already so, then starts pkgsite and opens VSCode's integrated ("simple") browser to show the csharg documentation.

Aux Tasks

• pksite service: auxilliary task to run pkgsite as a background service using scripts/pkgsite.sh. The script leverages browser-sync and nodemon to hot reload the Go module documentation on changes; many thanks to @mdaverde's Build your Golang package docs locally for paving the way. scripts/pkgsite.sh adds automatic installation of pkgsite, as well as the browser-sync and nodemon npm packages for the local user.
• view pkgsite: auxilliary task to open the VSCode-integrated "simple" browser and pass it the local URL to open in order to show the module documentation rendered by pkgsite. This requires a detour via a task input with ID "pkgsite".

Make Targets

• make: lists all targets.
• make test: runs all tests – please note that this strictly requires a genuine Docker (moby) container demon to be present. Trying to substitute a dockerd with podman will make tests fail for good reason, as podman isn't Docker for our purposes.
• make pkgsite: installs x/pkgsite, as well as the browser-sync and nodemon npm packages first, if not already done so. Then runs the pkgsite and hot reloads it whenever the documentation changes.
• make report: installs @gojp/goreportcard if not yet done so and then runs it on the code base.
• make vuln: install (or updates) govuln and then checks the Go sources.

Contributing

Please see CONTRIBUTING.md.

License and Copyright

(c) Siemens AG 2023

SPDX-License-Identifier: MIT

Documentation

Overview

Package turtlefinder provides a Containerizer that auto-detects container engines and automatically creates workload watchers for them. It supports both “permanent” daemons as well as socket-activated “don't-call-them-daemons”. Additionally, it also detects the hierarchy of container engines, such as containerd-in-Docker and podman-in-Docker.

Supported Container Engines

The following container engines are supported:

• Docker/Moby
• containerd
• CRI-O
• podman (via Docker-compatible API only)

Supported Socket Activators

The following socket activators are supported:

• systemd

Quick Start

That's all that is necessary:

containerizer := turtlefinder.New()

Boringly simple, right?

The turtlefinder containerizer.Containerizer is safe to be used in concurrent discoveries.

Principles of “Turtle” Discovery

A turtlefinder supports two different container engine discovery mechanisms:

• based on well-known engine process names; please note that this works for “always on” engine daemons, such as “dockerd” (even if this is initially socket-activated on systemd installations).
• based on well-known socket activators, such as “systemd”, in combination with well-known API socket names (rather, suffixes). This method is much more involved when compared to the well-known always-on process name method, but allows discovering especially usually short-lived “podman” services.

The turtlefinder then spins up background watchers as required that synchronize with the workload state of the detected container engines. Also, old engine watchers get retired as their engine processes die. This workload state information is then returned as the list of discovered containers, including the hierarchy of container engines, based on which engine is placed inside a container managed by another engine.

Well-Known Process Name Discovery

Basically, upon a container query the turtlefinder containerizer first looks for any newly seen container engines, based on container engine process names. The engine discovery can be extended by pluging in new engine detectors (and adaptors).

Well-known Socket Activation Name Discovery

For “short-lived” container engine services that terminate themselves whenever they go idle, we unfortunately need a more involved discovery mechanism. More involved, as we don't want to slow down discovery by constantly looking for something that isn't even installed in the system, so we need to do some optimization.

The general idea is to look for well-known socket activators, namely “systemd”. If found (even multiple times!), we scan such an activator for its listening unix domain sockets and determine their file system paths. If we find a matching path (rather, a matching suffix, such as “podman.sock”) we spin up a suitable background watcher. Of course, this background watcher will keep the container engine alive, but then we also need this service in constant monitoring.

The difficult part here is to avoid repeated unnecessary costly socket activator discoveries. We thus keep some state information about a socket activator's socket-related setup and only rediscover upon noticing changes in its socket configuration (which rarely if ever occurs).

Engines in Engines

A defining feature of the turtlefinder is that it additionally determines the hierarchy of container engines, such as when a container engine is hosted inside a container managed by a (parent) container engine. This hierarchy later gets propagated to the individual containers in form of a so-called “prefix”, attached in form of a special container label.

Such engine-in-engine configurations are actually not so unknown:

• Docker Desktop
• Kubernetes in Docker (KinD)
• podman in Docker

Decoration

Finally, the decoration of the discovered containers uses the usual (extensible) lxkns github.com/thediveo/lxkns/decorator.Decorator mechanism as part of the overall discovery.

Index

• Constants
• type Contexter
• type Engine
  • func NewEngine(ctx context.Context, w watcher.Watcher, ppidhint model.PIDType) *Engine
  • func (e *Engine) Containers(ctx context.Context) []*model.Container
  • func (e *Engine) IsAlive() bool
• type NewOption
  • func WithGettingOnlineWait(d time.Duration) NewOption
  • func WithWorkers(num int) NewOption
• type Overseer
• type TurtleFinder
  • func New(contexter Contexter, opts ...NewOption) *TurtleFinder
  • func (f *TurtleFinder) Close()
  • func (f *TurtleFinder) Containers(ctx context.Context, procs model.ProcessTable, pidmap model.PIDMapper) []*model.Container
  • func (f *TurtleFinder) EngineCount() int
  • func (f *TurtleFinder) Engines() []*model.ContainerEngine

Constants

const PrefixSeparator = "/"

PrefixSeparator is the separator used in hierarchical prefixes.

const TurtlefinderContainerPrefixLabelName = "turtlefinder/container/prefix"

TurtlefinderContainerPrefixLabelName defines the label name for attaching prefix information about the engine-hierarchy to containers. Discovery client can use these container labels to find out the hierarchy of containers. For instance, if container "A" is managed by a container engine hosted inside container "B", then container "A" is labelled with prefix "B".

Types

type Contexter

type Contexter func() context.Context

Contexter supplies a TurtleFinder with a suitable context for long-running container engine workload watching.

type Engine

type Engine struct {
	watcher.Watcher               // engine watcher (doubles as engine adapter).
	ID              string        // engine ID.
	Version         string        // engine version.
	Done            chan struct{} // closed when watch is done/has terminated.
	PPIDHint        model.PIDType // PID of engine's process; for container PID translation.
}

Engine watches a single container engine process for signs of container workload life, using the supplied "whale watcher".

Engine objects then can be queried for their workload, that is, the list of currently alive (running/paused) containers they manage.

An Engine can be “done” at any time when the container engine process terminates or otherwise disconnects the watcher. In this case, the Done channel will be closed.

func NewEngine

func NewEngine(ctx context.Context, w watcher.Watcher, ppidhint model.PIDType) *Engine

NewEngine returns a new Engine given the specified watcher. As NewEngine returns, the Engine is already "warming up" and has started watching (using the given context).

ppidhint optionally specifies a container engine's immediate parent process. This information is later necessary for lxkns to correctly translate container PIDs. When activating a socket-activated engine, the process tree scan does never include the engine, as this is only activated after the scan. In order to still allow lxkns to translate container PIDs related to newly socket-activated engines, we assume that the engine's parent process PID is in the same PID namespace, so we can also use that for correct PID translation.

func (*Engine) Containers

func (e *Engine) Containers(ctx context.Context) []*model.Container

Containers returns the alive containers managed by this engine, using the associated watcher.

The containers returned will reference a model.ContainerEngine and thus are decoupled from a turtlefinder's (container) Engine object.

func (*Engine) IsAlive

func (e *Engine) IsAlive() bool

IsAlive returns true as long as the engine watcher is operational and hasn't permanently failed/terminated.

type NewOption

type NewOption func(*TurtleFinder)

NewOption represents options to New when creating a new turtle finder.

func WithGettingOnlineWait

func WithGettingOnlineWait(d time.Duration) NewOption

WithGettingOnlineWait sets the maximum duration to wait for our workload view of a newly discovered container engine to become synchronized before proceeding with a container discovery. If the initial synchronisation phase takes longer, it won't be aborted. This option instead controls the maximum wait before proceeding with discovering containers from the already known engine workloads.

func WithWorkers

func WithWorkers(num int) NewOption

WithWorkers sets the maximum number of parallel container engine queries on the same TurtleFinder. A maximum number of zero or less is taken as GOMAXPROCS instead. Please note that this maximum applies to all concurrent TurtleFinder.Containers calls, and not to individual TurtleFinder.Containers calls separately.

type Overseer

type Overseer interface {
	Engines() []*model.ContainerEngine
}

Overseer gives access to information about container engines currently monitored.

turtlefinder.Turtlefinder objects implement the Overseer interface to allow code given only a containerizer.Containerizer to query the currently monitored container engine instances.

	var c containerizer.Containerizer
	o, ok := c.(turtlefinder.Overseer)
 	if ok {
	    engines := o.Engines()
 	}

type TurtleFinder

type TurtleFinder struct {
	// contains filtered or unexported fields
}

TurtleFinder implements the lxkns Containerizer interface to discover alive containers from one or more container engines. It can be safely used from multiple goroutines.

On demand, a TurtleFinder scans a process list for signs of container engines and then tries to contact the potential engines in order to watch their containers.

func New

func New(contexter Contexter, opts ...NewOption) *TurtleFinder

New returns a TurtleFinder object for further use. The supplied contexter is called whenever a new container engine has been found and its workload is to be watched: this contexter should return a suitable (long-running) context it preferably has control over, in order to properly shut down the "background" goroutine resources (indirectly) used by a TurtleFinder.

Further options (NewOption, such as WithWorkers and WithGettingOnlineWait) allow to customize the TurtleFinder object returned.

func (*TurtleFinder) Close

func (f *TurtleFinder) Close()

Close closes all resources associated with this turtle finder. This is an asynchronous process. Make sure to also cancel or have already cancelled the context

func (*TurtleFinder) Containers

func (f *TurtleFinder) Containers(
	ctx context.Context, procs model.ProcessTable, pidmap model.PIDMapper,
) []*model.Container

Containers returns the current container state of (alive) containers from all discovered container engines.

func (*TurtleFinder) EngineCount

func (f *TurtleFinder) EngineCount() int

EngineCount returns the number of container engines currently under watch. Callers might want to use the Engines method instead as EngineCount bases on it (because we don't store an explicit engine count anywhere).

func (*TurtleFinder) Engines

func (f *TurtleFinder) Engines() []*model.ContainerEngine

Engines returns information about the container engines currently being monitored.