whalewatcher

README

Whalewatcher

🔭🐋 whalewatcher is a simple Golang module that relieves applications from the tedious task of constantly monitoring "alive" container workloads: no need to watching boring event streams or alternatively polling to have the accurate picture. Never worry about how to properly synchronize to a changing workload at startup, this is all taken care of for you.

Instead, using whalewatcher an application simply asks for the current state of affairs at any time when it needs to do so. The workload state then is directly answered from whalewatcher's trackers without causing container engine load: which containers are alive right now? And what composer projects are in use?

Oh, whalewatcher isn't limited to just Docker, it also supports nerdctl with plain containerd. Wait, there's even more: any container engine supporting the CRI pod event API can be tracked, too.

Stayin' Alive

This module watches Docker and plain containerd containers becoming "alive" with processes and later die, keeping track of only the "alive" containers. On purpose, whalewatcher focuses solely on running and paused containers, so those only that have at least an initial container process running (and thus a PID).

Thus, use cases for whalewatcher are container-aware tools that seemingly randomly need the current state of affairs for all running containers – such as lxkns. These tools themselves now don't need anymore to do the ugly lifting of container engine event tracking, engine state resynchronization after reconnects, et cetera. Here, the whalewatcher module reduces system load especially when state is requested in bursts, as it offers a load-optimized kind of "cache". Yet this cache is always closely synchronized to the container engine state.

ℹ️ This module now optionally supports receiving container lifecycle events by requesting a lifecycle event stream from a watcher.Watcher. Only the lifecycle events are supported for when a container becomes alive or exists, or it pauses or unpauses.

Features

• tracks container information with respect to a container's ID/name, PID, labels, (un)pausing state, and optional (composer) project. See the whalewatcher.Container type for details.
• supports multiple types of container engines:
  • Docker/Moby.
  • plain containerd using containerd's native API.
  • cri-o and containerd via the generic CRI pod event API. In principle, other container engines implementing the CRI pod event API should also work:
    • sandbox container lifecycle events must be reported and not suppressed.
    • sandbox and container PIDs must be reported by the verbose variant of the container status API call in the PID field of the JSON info object.
  • Podman: use the Docker/Moby watcher. Due to several severe issues we're not supporting Podman's own API any longer and have archived the sealwatcher experiment. More background information can be found in alias podman=p.o.'d.man.
• composer project-aware:
  • docker-compose
  • nerdctl
• optional configurable automatic retries using backoffs (with different strategies as supported by the external backoff module).
• documentation ... please see:

Example Usage

From example/main.go:

package main

import (
    "context"
    "fmt"
    "sort"

    "github.com/thediveo/whalewatcher/watcher/moby"
)

func main() {
    // connect to the Docker engine; configure no backoff.
    whalewatcher, err := moby.New("unix:///var/run/docker.sock", nil)
    if err != nil {
        panic(err)
    }
    ctx, cancel := context.WithCancel(context.Background())
    fmt.Printf("watching engine ID: %s\n", whalewatcher.ID(ctx))

    // run the watch in a separate go routine.
    done := make(chan struct{})
    go func() {
        if err := whalewatcher.Watch(ctx); ctx.Err() != context.Canceled {
            panic(err)
        }
        close(done)
    }()

    // depending on application you don't need to wait for the first results to
    // become ready; in this example we want to wait for results.
    <-whalewatcher.Ready()

    // get list of projects; we add the unnamed "" project which automatically
    // contains all non-project (standalone) containers.
    projectnames := append(whalewatcher.Portfolio().Names(), "")
    sort.Strings(projectnames)
    for _, projectname := range projectnames {
        containers := whalewatcher.Portfolio().Project(projectname)
        if containers == nil {
            continue // doh ... gone!
        }
        fmt.Printf("project %q:\n", projectname)
        for _, container := range containers.Containers() {
            fmt.Printf("  container %q with PID %d\n", container.Name, container.PID)
        }
        fmt.Println()
    }

    // finally stop the watch
    cancel()
    <-done
    whalewatcher.Close()
}

Hacking It

This project comes with comprehensive unit tests, including (albeit limited) mocking of Docker clients to the small extend required for whale watching.

• unit tests require Docker CE in a moderately recent version. Debian users are advised to install Docker CE from Docker's package, as Debian's own packages tend to completely outdate function-wise over the lifespan of a particular Debian release.

Fun Fact: the tests covering containerd and CRI-O use a dockerized container/cri-o image, leveraging the kindest/base image by the KinD SIG, in ways the SIG surely didn't envision.

The tests come with integrated leak checks:

• goroutine leak checking courtesy of Gomega's gleak package.

• file descriptor leak checking courtesy of the @thediveo/fdooze module.

Note: do not run parallel tests for multiple packages. make test ensures to run all package tests always sequentially, but in case you run go test yourself, please don't forget -p 1 when testing multiple packages in one, erm, go.

Unit tests about interfacing with and tracking containerd and CRI container engines use Docker containers with containerized containerd and cri-o engines. The corresponding test images base on kindest/base Docker images, courtesy of the KinD k8s SIG. Now, we fully understand that we're on our own here with no guarantees given by the KinD k8s SIG. However, their kindest/base images are really helpful in coming up with containerizing a containerd engine to a Docker container that we cannot simply pass by them.

All we're adding is some slim configuration so that we can create some pods and/or containers. The cri-o bases on some instructions about how to modify the KinD images to use cri-o instead of containerd; but in our case we install them both side-by-side. And as it happens, they seem to somehow get along with each other when confined to the same Docker container.

VSCode Tasks

The included go-plugger.code-workspace defines the following tasks:

• View Go module documentation task: installs pkgsite, if not done already so, then starts pkgsite and opens VSCode's integrated ("simple") browser to show the go-plugger/v2 documentation.

• Build workspace task: builds all, including the shared library test plugin.

• Run all tests with coverage task: does what it says on the tin and runs all tests with coverage.

Aux Tasks

• pksite service: auxilliary task to run pkgsite as a background service using scripts/pkgsite.sh. The script leverages browser-sync and nodemon to hot reload the Go module documentation on changes; many thanks to @mdaverde's Build your Golang package docs locally for paving the way. scripts/pkgsite.sh adds automatic installation of pkgsite, as well as the browser-sync and nodemon npm packages for the local user.
• view pkgsite: auxilliary task to open the VSCode-integrated "simple" browser and pass it the local URL to open in order to show the module documentation rendered by pkgsite. This requires a detour via a task input with ID "pkgsite".

Make Targets

• make: lists all targets.
• make coverage: runs all tests with coverage and then updates the coverage badge in README.md.
• make pkgsite: installs x/pkgsite, as well as the browser-sync and nodemon npm packages first, if not already done so. Then runs the pkgsite and hot reloads it whenever the documentation changes.
• make report: installs @gojp/goreportcard if not yet done so and then runs it on the code base.
• make test: runs all tests (including dynamic plugins).

Contributing

Please see CONTRIBUTING.md.

Copyright and License

whalewatcher is Copyright 2021, 2023 Harald Albrecht, licensed under the Apache License, Version 2.0.

Documentation

Overview

Package whalewatcher watches Docker and containerd containers as they come and go from the perspective of containers that are "alive", that is, only those containers with actual processes. In contrast, freshly created or "dead" containers without any processes are not tracked.

Furthermore, this package understands how containers optionally are organized into composer projects Docker compose.

As the focus of this module is on containers that are either in running or paused states, the envisioned use cases are tools that solely interact with processes, Linux-kernel namespaces, et cetera of these containers (often via various elements of the proc filesystem).

In order to cause only as low system load as possible this module monitors the container engine's container lifecycle-related events instead of stupid polling. In particular, this module decouples an application's access to the current state from tracking this container state.

Optionally, applications can subscribe to an events channel that passes on the lifecycle events whalewatcher receives.

Watcher

A github.com/thediveo/whalewatcher/watcher.Watcher monitors ("watches") the containers of a single container engine instance when running its Watch method in a separate go routine. Cancel its passed context to stop watching and then Close the watcher in order to release any allocated resources.

Watchers return information about alive containers (and optionally their organization into projects) via a Portfolio. Please do not keep the Portfolio reference for long periods of time, as might change in case the watcher needs to reconnect to a container engine after losing API contact.

Please refer to example/main.go as an example:

package main

import (
    "context"
    "fmt"
    "sort"

    "github.com/thediveo/whalewatcher/watcher/moby"
)

func main() {
    whalewatcher, err := moby.NewWatcher("unix:///var/run/docker.sock")
    if err != nil {
        panic(err)
    }
    ctx, cancel := context.WithCancel(context.Background())
    fmt.Printf("watching engine ID: %s\n", whalewatcher.ID(ctx))

    // run the watch in a separate go routine.
    go whalewatcher.Watch(ctx)

    // depending on application you don't need to wait for the first results to
    // become ready; in this example we want to wait for results.
    <-whalewatcher.Ready()

    // get list of projects; we add the unnamed "" project which automatically
    // contains all non-project (standalone) containers.
    projectnames := append(whalewatcher.Portfolio().Names(), "")
    sort.Strings(projectnames)
    for _, projectname := range projectnames {
        containers := whalewatcher.Portfolio().Project(projectname)
        if containers == nil {
            continue // doh ... gone!
        }
        fmt.Printf("project %q:\n", projectname)
        for _, container := range containers.Containers() {
            fmt.Printf("  container %q with PID %d\n", container.Name, container.PID)
        }
        fmt.Println()
    }

    // finally stop the watch
    cancel()
    whalewatcher.Close()
}

Note: if an application needs to watch both Docker and "pure" containerd containers, then it needs to create two separate watchers, one for the Docker engine and another one for the containerd instance. The containerd watcher doesn't watch any Docker-managed containers (it cannot as Docker does not attach all information at the containerd level, especially not the container name).

Information Model

The high-level view on whalewatcher's information model is as follows:

• A Watcher has a Portfolio, which is accessible using github.com/thediveo/whalewatcher/watcher.Watcher.Portfolio.
• A Portfolio consists of ComposerProject objects, including the "unnamed" project.
• A ComposerProject consists of Container descriptions; these descriptions represent only a purposely limited set of information, but can be augmented using "Rucksacks".

Portfolio

The container information model starts with the Portfolio: a Portfolio consists of one or more projects in form of ComposerProject, including the "unnamed" ComposerProject (that contains all non-project containers).

ComposerProject

Composer projects are either explicitly named, or the "zero" project that has no name (that is, the empty name). A ComposerProject consists of Container objects.

Container

Containers store limited aspects about individual containers, such as their names, IDs, and PIDs.

Important: Container objects are immutable. For this reason, fetch the most recent updated state (where necessary) by retrieving the ComposerProject from the Watcher, and then querying for your Container using ComposerProject.Container.

Index

• type ComposerProject
  • func (p *ComposerProject) Container(nameorid string) *Container
  • func (p *ComposerProject) ContainerNames() []string
  • func (p *ComposerProject) Containers() []*Container
  • func (p *ComposerProject) SetPaused(nameorid string, paused bool) *Container
  • func (p *ComposerProject) String() string
• type Container
  • func (c Container) ProjectName() string
  • func (c Container) String() string
• type Portfolio
  • func NewPortfolio() *Portfolio
  • func (pf *Portfolio) Add(cntr *Container) bool
  • func (pf *Portfolio) Container(nameorid string) *Container
  • func (pf *Portfolio) ContainerTotal() (total int)
  • func (pf *Portfolio) Names() []string
  • func (pf *Portfolio) Project(name string) *ComposerProject
  • func (pf *Portfolio) Remove(nameorid string, project string) (cntr *Container)

Types

type ComposerProject

type ComposerProject struct {
	Name string // composer project name, guaranteed to be constant.
	// contains filtered or unexported fields
}

ComposerProject represents a set of either running or paused (but always "somehow" alive) [Container]s belonging to a specific Docker Compose/Composer project.

As composer projects are artefacts above the first-level elements of the Docker container engine we can only reconstruct them in an extremely limited fashion from the live container information available to us. Yet that's fine in our context, as we just want to understand the concrete relationships between projects and their containers.

func (*ComposerProject) Container

func (p *ComposerProject) Container(nameorid string) *Container

Container returns container information about the Container with the specified name or ID. If the name or ID wasn't found in this project, then nil is returned instead.

func (*ComposerProject) ContainerNames

func (p *ComposerProject) ContainerNames() []string

ContainerNames returns the current list of container names belonging to this composer project.

func (*ComposerProject) Containers

func (p *ComposerProject) Containers() []*Container

Containers returns the current list of containers belonging to this composer project.

func (*ComposerProject) SetPaused

func (p *ComposerProject) SetPaused(nameorid string, paused bool) *Container

SetPaused changes a Container's Paused state, obeying the design restriction that Container objects are immutable. It returns the container in its new state.

func (*ComposerProject) String

func (p *ComposerProject) String() string

String returns a textual representation of a composer project with its containers (rendering names, but not IDs).

type Container

type Container struct {
	ID       string            // unique identifier of this container.
	Name     string            // user-friendly name of this container.
	Labels   map[string]string // labels assigned to this container.
	PID      int               // PID of container's initial ("ealdorman") process.
	Project  string            // optional composer project name, or zero.
	Paused   bool              // true if container is paused, false if running.
	Rucksack interface{}       // optional additional application-specific container information.
}

Container is a deliberately limited view on containers, dealing with only those few bits of data we're interested in for watching alive containers and how they optionally are organized into composer projects.

Please note that Container objects are considered immutable, so there's no locking them for updating.

Containers to considered to be alive from the perspective of the whalewatcher module when they have an initial process (which might be frozen) and thus a PID corresponding with that initial process. In contrast, we don't care about containers which are either dead without any container process(es) or are just yet created and thus still without any container process(es).

The Rucksack supports storing additional application-specific container (and container engine-specific) information; see also github.com/thediveo/whalewatcher/engineclient.RucksackPacker.

func (Container) ProjectName

func (c Container) ProjectName() string

ProjectName returns the name of the composer project for this container, if any; otherwise, it returns "" if a container isn't associated with a composer project.

func (Container) String

func (c Container) String() string

String renders a textual representation of the information kept about a specific container, such as its name, ID, and PID.

type Portfolio

type Portfolio struct {
	// contains filtered or unexported fields
}

Portfolio represents all known composer projects, including the "zero" (unnamed) project. The "zero" project has the zero name and contains all containers that are not part of any named composer project. The Portfolio manages projects implicitly when adding and removing containers belonging to projects. Thus, there is no need to explicitly add or delete composer projects.

func NewPortfolio

func NewPortfolio() *Portfolio

NewPortfolio returns a new Portfolio.

func (*Portfolio) Add

func (pf *Portfolio) Add(cntr *Container) bool

Add a container to the portfolio, creating also its composer project if that is not yet known. Returns true if the container was newly added, false if it already exists.

func (*Portfolio) Container

func (pf *Portfolio) Container(nameorid string) *Container

Container returns the Container with the specified name, regardless of which project it is in. It returns nil, if no container with the specified name could be found.

func (*Portfolio) ContainerTotal

func (pf *Portfolio) ContainerTotal() (total int)

ContainerTotal returns the total number of containers over all projects, including non-project "standalone" containers.

func (*Portfolio) Names

func (pf *Portfolio) Names() []string

Names returns the names of all composer projects sans the "zero" project.

func (*Portfolio) Project

func (pf *Portfolio) Project(name string) *ComposerProject

Project returns the project with the specified name (including the zero project name), or nil if no project with the specified name currently exists.

func (*Portfolio) Remove

func (pf *Portfolio) Remove(nameorid string, project string) (cntr *Container)

Remove a container identified by its ID or name as well as its composer project name from the portfolio, removing its composer project if it was the only container left in the project.

The information about the removed container is returned, otherwise if no such container exists, nil is returned instead.