watchdog

Documentation

Overview

Package watchdog is responsible for monitoring the sentry for tasks that may potentially be stuck or looping inderterminally causing hard to debug hungs in the untrusted app.

It works by periodically querying all tasks to check whether they are in user mode (RunUser), kernel mode (RunSys), or blocked in the kernel (OffCPU). Tasks that have been running in kernel mode for a long time in the same syscall without blocking are considered stuck and are reported.

When a stuck task is detected, the watchdog can take one of the following actions:

1. LogWarning: Logs a warning message followed by a stack dump of all goroutines. If a tasks continues to be stuck, the message will repeat every minute, unless a new stuck task is detected
2. Panic: same as above, followed by panic()

Index

• Constants
• type Action
  • func (a Action) String() string
• type Watchdog
  • func New(k *kernel.Kernel, taskTimeout time.Duration, a Action) *Watchdog
  • func (w *Watchdog) Start()
  • func (w *Watchdog) Stop()

Constants

const DefaultTimeout = 3 * time.Minute

DefaultTimeout is a resonable timeout value for most applications.

Types

type Action

type Action int

Action defines what action to take when a stuck task is detected.

const (
	// LogWarning logs warning message followed by stack trace.
	LogWarning Action = iota
	// Panic will do the same logging as LogWarning and panic().
	Panic
)

func (Action) String

func (a Action) String() string

String returns Action's string representation.

type Watchdog

type Watchdog struct {
	// contains filtered or unexported fields
}

Watchdog is the main watchdog class. It controls a goroutine that periodically analyses all tasks and reports if any of them appear to be stuck.

func New

func New(k *kernel.Kernel, taskTimeout time.Duration, a Action) *Watchdog

New creates a new watchdog.

func (*Watchdog) Start

func (w *Watchdog) Start()

Start starts the watchdog.

func (*Watchdog) Stop

func (w *Watchdog) Stop()

Stop requests the watchdog to stop and wait for it.