whalewatcher

README

Whalewatcher

whalewatcher is a simple Golang module that watches Docker and plain containerd containers becoming "alive" with processes and later die, keeping track of only the "alive" containers. On purpose, this module focuses solely on running and paused containers, so those only that have at least an initial container process running (and thus a PID).

Envisioned use cases are container-aware tools – such as lxkns – that seemingly randomly need the current state of affairs for all running containers. That is, tools that yet do not want to do the ugly lifting of container engine event tracking, engine state resynchronization after reconnects, et cetera. Here, the whalewatcher module reduces system load especially when state is requested in bursts, as it offers a load-optimized kind of "cache". Yet this cache is always closely synchronized to the container engine state.

🛈 If your application requires immediate action upon container lifecycle events then our whalewatcher isn't the right module for it: our module is design for those use cases where the application needing information about containers is completely decoupled from container lifecycle events.

Features

• tracks container information with respect to a container's ID/name, PID, labels, (un)pausing state, and optional (composer) project. See the whalewatcher.Container type for details.
• supports different container engines:
  • Docker/Moby
  • plain containerd
• composer project-aware:
  • docker-compose
  • nerdctl
• optional configurable automatic retries using backoffs (with different strategies as supported by the external backoff module).
• documentation ... please see:

Example Usage

From example/main.go:

package main

import (
    "context"
    "fmt"
    "sort"

    "github.com/thediveo/whalewatcher/watcher/moby"
)

func main() {
    // connect to the Docker engine; configure no backoff.
    whalewatcher, err := moby.New("unix:///var/run/docker.sock", nil)
    if err != nil {
        panic(err)
    }
    ctx, cancel := context.WithCancel(context.Background())
    fmt.Printf("watching engine ID: %s\n", whalewatcher.ID(ctx))

    // run the watch in a separate go routine.
    done := make(chan struct{})
    go func() {
        if err := whalewatcher.Watch(ctx); ctx.Err() != context.Canceled {
            panic(err)
        }
        close(done)
    }()

    // depending on application you don't need to wait for the first results to
    // become ready; in this example we want to wait for results.
    <-whalewatcher.Ready()

    // get list of projects; we add the unnamed "" project which automatically
    // contains all non-project (standalone) containers.
    projectnames := append(whalewatcher.Portfolio().Names(), "")
    sort.Strings(projectnames)
    for _, projectname := range projectnames {
        containers := whalewatcher.Portfolio().Project(projectname)
        if containers == nil {
            continue // doh ... gone!
        }
        fmt.Printf("project %q:\n", projectname)
        for _, container := range containers.Containers() {
            fmt.Printf("  container %q with PID %d\n", container.Name, container.PID)
        }
        fmt.Println()
    }

    // finally stop the watch
    cancel()
    <-done
    whalewatcher.Close()
}

Hacking It

This project comes with comprehensive unit tests, including (limited) mocking of Docker clients to the small extend required for whale watching.

Copyright and License

Copyright 2021 Harald Albrecht, licensed under the Apache License, Version 2.0.

Documentation

Overview

Package whalewatcher watches Docker and containerd containers as they come and go from the perspective of containers that are "alive", that is, only those containers with actual processes. In contrast, freshly created or "dead" containers without any processes are not tracked.

Furthermore, this package understands how containers optionally are organized into composer projects (Docker composer, https://github.com/docker/compose). Please note that full nerdctl project-awareness currently is blocked by issue #241 (https://github.com/containerd/nerdctl/issues/241).

As the focus of this module is on containers that are either in running or paused states, the envisioned use cases are tools that solely interact with processes, Linux-kernel namespaces, et cetera of these containers (often via various elements of the proc filesystem).

In order to cause only as low system load as possible this module monitors the container engine's container lifecycle-related events instead of stupid polling. However, if your application wants to immediately react on such events, then this package is not suitable. Instead, it decouples an application's access to the current state from tracking this container state.

Watcher

A Watcher watches the containers of a single container engine instance when running its Watch method in a separate go routine. Cancel its passed context to stop watching.

Watchers return information about alive containers (and optionally their organization into projects) via a Portfolio. Please do not keep the Portfolio reference for long periods of time, as might change in case the watcher needs to reconnect to a container engine after losing API contact.

Please refer to example/main.go as an example:

package main

import (
	"context"
	"fmt"
	"sort"

	"github.com/thediveo/whalewatcher/watcher/moby"
)

whalewatcher, err := moby.NewWatcher("unix:///var/run/docker.sock")
if err != nil {
	panic(err)
}
ctx, cancel := context.WithCancel(context.Background())
fmt.Printf("watching engine ID: %s\n", whalewatcher.ID(ctx))

// run the watch in a separate go routine.
go whalewatcher.Watch(ctx)

// depending on application you don't need to wait for the first results to
// become ready; in this example we want to wait for results.
<-whalewatcher.Ready()

// get list of projects; we add the unnamed "" project which automatically
// contains all non-project (standalone) containers.
projectnames := append(whalewatcher.Portfolio().Names(), "")
sort.Strings(projectnames)
for _, projectname := range projectnames {
	containers := whalewatcher.Portfolio().Project(projectname)
	if containers == nil {
		continue // doh ... gone!
	}
	fmt.Printf("project %q:\n", projectname)
	for _, container := range containers.Containers() {
		fmt.Printf("  container %q with PID %d\n", container.Name, container.PID)
	}
	fmt.Println()
}

// finally stop the watch
cancel()
whalewatcher.Close()

Note: if an application needs to watch both Docker and "pure" containerd containers, then it needs to create two watchers, one for the Docker engine and another one for the containerd instance. The containerd watcher doesn't watch any Docker-managed containers (it cannot as Docker does not attach all information at the containerd level, especially not the container name).

Portfolio

The container information model starts with the Portfolio: the Portfolio knows about the currently available projects (ComposerProjects).

ComposerProject

Composer projects are either explicitly named or the "zero" project that has no name (empty name). Projects then know the Containers associated to them.

Container

Containers store limited aspects about individual containers, such as their names, IDs, and PIDs.

Index

• type ComposerProject
  • func (p *ComposerProject) Container(nameorid string) *Container
  • func (p *ComposerProject) ContainerNames() []string
  • func (p *ComposerProject) Containers() []*Container
  • func (p *ComposerProject) SetPaused(nameorid string, paused bool)
  • func (p *ComposerProject) String() string
• type Container
  • func (c Container) ProjectName() string
  • func (c Container) String() string
• type Portfolio
  • func NewPortfolio() *Portfolio
  • func (pf *Portfolio) Add(cntr *Container)
  • func (pf *Portfolio) Container(nameorid string) *Container
  • func (pf *Portfolio) ContainerTotal() (total int)
  • func (pf *Portfolio) Names() []string
  • func (pf *Portfolio) Project(name string) *ComposerProject
  • func (pf *Portfolio) Remove(nameorid string, project string)

Types

type ComposerProject

type ComposerProject struct {
	Name string // composer project name, guaranteed to be constant.
	// contains filtered or unexported fields
}

ComposerProject represents a set of (running or paused, yet somehow alive) containers belonging to a specific Docker Compose/Composer project.

As composer projects are artefacts above the first-level elements of the Docker container engine we can only reconstruct them in an extremely limited fashion from the live container information available to us. Yet that's fine in our context, as we just want to understand the concrete relationships between projects and their containers.

func (*ComposerProject) Container

func (p *ComposerProject) Container(nameorid string) *Container

Container returns container information about the container with the specified name or ID. If the name or ID wasn't found in this project, then nil is returned instead.

func (*ComposerProject) ContainerNames

func (p *ComposerProject) ContainerNames() []string

ContainerNames returns the current list of container names belonging to this composer project.

func (*ComposerProject) Containers

func (p *ComposerProject) Containers() []*Container

Containers returns the current list of containers in this composer project.

func (*ComposerProject) SetPaused

func (p *ComposerProject) SetPaused(nameorid string, paused bool)

SetPaused changes a container's Paused state, obeying the design restriction that Container objects are immutable.

func (*ComposerProject) String

func (p *ComposerProject) String() string

String returns a textual representation of a composer project with its containers (rendering names, but not IDs).

type Container

type Container struct {
	ID      string            // unique identifier of this container.
	Name    string            // user-friendly name of this container.
	Labels  map[string]string // labels assigned to this container.
	PID     int               // PID of container's initial ("ealdorman") process.
	Project string            // optional composer project name, or zero.
	Paused  bool              // true if container is paused, false if running.
}

Container is a deliberately limited view on containers, dealing with only those few bits of data we're interested in for watching alive containers and how they optionally are organized into composer projects.

Please note that Container objects are considered immutable, so there's no locking them for updating.

We consider containers to be alive when they have an initial process (which might be frozen) and thus a PID corresponding with that initial process. In contrast, we don't care about containers which are either dead without any container process(es) or are just yet created and thus still without any container process(es).

func (Container) ProjectName

func (c Container) ProjectName() string

ProjectName returns the name of the composer project for this container, if any; otherwise, it returns "" if a container isn't associated with a composer project.

func (Container) String

func (c Container) String() string

String renders a textual representation of the information kept about a specific container, such as its name, ID, and PID.

type Portfolio

type Portfolio struct {
	// contains filtered or unexported fields
}

Portfolio represents all known composer projects, including the "zero" (unnamed) project. The "zero" project has the zero name and contains all containers that are not part of any named composer project. The Portfolio manages projects implicitly when adding and removing containers belonging to projects. Thus, there is no need to explicitly add or delete composer projects.

func NewPortfolio

func NewPortfolio() *Portfolio

NewPortfolio returns a new Portfolio.

func (*Portfolio) Add

func (pf *Portfolio) Add(cntr *Container)

Add a container to the portfolio, creating also its composer project if that is not yet known.

func (*Portfolio) Container

func (pf *Portfolio) Container(nameorid string) *Container

Container returns the container with the specified name, regardless of which project it is in. It returns nil, if no container with the specified name could be found.

func (*Portfolio) ContainerTotal

func (pf *Portfolio) ContainerTotal() (total int)

ContainerTotal returns the total number of containers over all projects, including non-project "standalone" containers.

func (*Portfolio) Names

func (pf *Portfolio) Names() []string

Names returns the names of all composer projects sans the "zero" project.

func (*Portfolio) Project

func (pf *Portfolio) Project(name string) *ComposerProject

Project returns the project with the specified name (including the zero project name), or nil if no project with the specified name currently exists.

func (*Portfolio) Remove

func (pf *Portfolio) Remove(nameorid string, project string)

Remove a container identified by its ID or name as well as its composer project name from the portfolio, removing its composer project if it was the only container left in the project.

It's fine (no error) trying to to Remove a non-existing container.