scan

Documentation

Overview

Package scan implements functionality for scanning Swarming datastore.

It is used by `monitor` binary.

Index

• func ActiveTasks(ctx context.Context, visitors []TaskVisitor) error
• func Bots(ctx context.Context, visitors []BotVisitor) error
• type ActiveJobsReporter
  • func (r *ActiveJobsReporter) Finalize(ctx context.Context, scanErr error) error
  • func (r *ActiveJobsReporter) Prepare(ctx context.Context)
  • func (r *ActiveJobsReporter) Visit(ctx context.Context, trs *model.TaskResultSummary)
• type BotVisitor
• type BotsDimensionsAggregator
  • func (a *BotsDimensionsAggregator) Finalize(ctx context.Context, scanErr error) error
  • func (*BotsDimensionsAggregator) Frequency() time.Duration
  • func (*BotsDimensionsAggregator) ID() string
  • func (a *BotsDimensionsAggregator) Prepare(ctx context.Context, shards int, lastRun time.Time)
  • func (a *BotsDimensionsAggregator) Visit(ctx context.Context, shard int, bot *model.BotInfo)
• type BotsMetricsReporter
  • func (r *BotsMetricsReporter) Finalize(ctx context.Context, scanErr error) error
  • func (*BotsMetricsReporter) Frequency() time.Duration
  • func (*BotsMetricsReporter) ID() string
  • func (r *BotsMetricsReporter) Prepare(ctx context.Context, shards int, lastRun time.Time)
  • func (r *BotsMetricsReporter) Visit(ctx context.Context, shard int, bot *model.BotInfo)
• type DeadBotDetector
  • func (r *DeadBotDetector) Finalize(ctx context.Context, scanErr error) error
  • func (*DeadBotDetector) Frequency() time.Duration
  • func (*DeadBotDetector) ID() string
  • func (r *DeadBotDetector) Prepare(ctx context.Context, shards int, lastRun time.Time)
  • func (r *DeadBotDetector) Visit(ctx context.Context, shard int, bot *model.BotInfo)
• type NamedCachesAggregator
  • func (a *NamedCachesAggregator) Finalize(ctx context.Context, scanErr error) error
  • func (*NamedCachesAggregator) Frequency() time.Duration
  • func (*NamedCachesAggregator) ID() string
  • func (a *NamedCachesAggregator) Prepare(ctx context.Context, shards int, lastRun time.Time)
  • func (a *NamedCachesAggregator) Visit(ctx context.Context, shard int, bot *model.BotInfo)
• type TaskVisitor

Functions

func ActiveTasks

func ActiveTasks(ctx context.Context, visitors []TaskVisitor) error

ActiveTasks visits all pending and running tasks.

Returns a multi-error with the scan error (if any) and errors from all visitors' finalizers (if any) in arbitrary order.

func Bots

func Bots(ctx context.Context, visitors []BotVisitor) error

Bots visits all bots via multiple parallel queries.

Takes a list of all registered visitors. Will run only ones that are due for execution now per their run frequency.

Returns a multi-error with the scan error (if any) and errors from all visitors' finalizers (if any) in arbitrary order.

Types

type ActiveJobsReporter

type ActiveJobsReporter struct {
	// ServiceName is a service name to put into metrics' target.
	ServiceName string
	// JobName is a job name to put into metrics' target.
	JobName string
	// Monitor to use to flush metrics.
	Monitor monitor.Monitor
	// contains filtered or unexported fields
}

ActiveJobsReporter is TaskVisitor that reports the number of active jobs per combination of dimensions to monitoring.

func (*ActiveJobsReporter) Finalize

func (r *ActiveJobsReporter) Finalize(ctx context.Context, scanErr error) error

Finalize is called once the scan is done.

Part of TaskVisitor interface.

func (*ActiveJobsReporter) Prepare

func (r *ActiveJobsReporter) Prepare(ctx context.Context)

Prepare prepares the visitor state.

Part of TaskVisitor interface.

func (*ActiveJobsReporter) Visit

func (r *ActiveJobsReporter) Visit(ctx context.Context, trs *model.TaskResultSummary)

Visit is called for every visited task.

Part of TaskVisitor interface.

type BotVisitor

type BotVisitor interface {
	// ID returns an unique identifier of this visitor used for storing its state.
	ID() string

	// Frequency returns how frequently this visitor should run.
	//
	// This is approximate. Granularity would be ~= 1 min. Value of 0 means this
	// visitor will approximately run every minute.
	Frequency() time.Duration

	// Prepare prepares the visitor to use `shards` parallel queries.
	//
	// It should initialize per-shard state used by Visit. `lastRun` is the last
	// time this visitor ran or a zero time if this is the first time it is
	// running.
	Prepare(ctx context.Context, shards int, lastRun time.Time)

	// Visit is called for every bot.
	//
	// The overall scan is split into multiple shards. Within a shard the scan is
	// sequential, but shards themselves are processed in parallel.
	Visit(ctx context.Context, shard int, bot *model.BotInfo)

	// Finalize is called once the scan is done.
	//
	// It is passed an error if the scan was incomplete. If the scan was complete,
	// (i.e. Visit visited all bots), it receives nil.
	//
	// The returned error will be reported as an overall scan error.
	Finalize(ctx context.Context, scanErr error) error
}

BotVisitor examines bots in parallel.

Should keep track of errors internally during the scan, reporting them only in the end in Finalize.

type BotsDimensionsAggregator

type BotsDimensionsAggregator struct {
	// contains filtered or unexported fields
}

BotsDimensionsAggregator is BotVisitor that collects the set of all possible bot dimensions and stores it in the datastore.

The result is used by GetBotDimensions RPCs.

func (*BotsDimensionsAggregator) Finalize

func (a *BotsDimensionsAggregator) Finalize(ctx context.Context, scanErr error) error

Finalize is called once the scan is done.

Part of BotVisitor interface.

func (*BotsDimensionsAggregator) Frequency

func (*BotsDimensionsAggregator) Frequency() time.Duration

Frequency returns how frequently this visitor should run.

Part of BotVisitor interface.

func (*BotsDimensionsAggregator) ID

func (*BotsDimensionsAggregator) ID() string

ID returns an unique identifier of this visitor used for storing its state.

Part of BotVisitor interface.

func (*BotsDimensionsAggregator) Prepare

func (a *BotsDimensionsAggregator) Prepare(ctx context.Context, shards int, lastRun time.Time)

Prepare prepares the visitor to use `shards` parallel queries.

Part of BotVisitor interface.

func (*BotsDimensionsAggregator) Visit

func (a *BotsDimensionsAggregator) Visit(ctx context.Context, shard int, bot *model.BotInfo)

Visit is called for every bot.

Part of BotVisitor interface.

type BotsMetricsReporter

type BotsMetricsReporter struct {
	// ServiceName is a service name to put into metrics' target.
	ServiceName string
	// JobName is a job name to put into metrics' target.
	JobName string
	// Monitor to use to flush metrics.
	Monitor monitor.Monitor
	// contains filtered or unexported fields
}

BotsMetricsReporter is BotVisitor that reports stats about bots to the monitoring.

func (*BotsMetricsReporter) Finalize

func (r *BotsMetricsReporter) Finalize(ctx context.Context, scanErr error) error

Finalize is called once the scan is done.

Part of BotVisitor interface.

func (*BotsMetricsReporter) Frequency

func (*BotsMetricsReporter) Frequency() time.Duration

Frequency returns how frequently this visitor should run.

Part of BotVisitor interface.

func (*BotsMetricsReporter) ID

func (*BotsMetricsReporter) ID() string

ID returns an unique identifier of this visitor used for storing its state.

Part of BotVisitor interface.

func (*BotsMetricsReporter) Prepare

func (r *BotsMetricsReporter) Prepare(ctx context.Context, shards int, lastRun time.Time)

Prepare prepares the visitor to use `shards` parallel queries.

Part of BotVisitor interface.

func (*BotsMetricsReporter) Visit

func (r *BotsMetricsReporter) Visit(ctx context.Context, shard int, bot *model.BotInfo)

Visit is called for every bot.

Part of BotVisitor interface.

type DeadBotDetector

type DeadBotDetector struct {
	// BotDeathTimeout is how long a bot must be away before being declared dead.
	BotDeathTimeout time.Duration
	// contains filtered or unexported fields
}

DeadBotDetector is a BotVisitor that recognizes bots that haven't been seen for a while and moves them into "DEAD" state.

func (*DeadBotDetector) Finalize

func (r *DeadBotDetector) Finalize(ctx context.Context, scanErr error) error

Finalize is called once the scan is done.

Part of BotVisitor interface.

func (*DeadBotDetector) Frequency

func (*DeadBotDetector) Frequency() time.Duration

Frequency returns how frequently this visitor should run.

Part of BotVisitor interface.

func (*DeadBotDetector) ID

func (*DeadBotDetector) ID() string

ID returns an unique identifier of this visitor used for storing its state.

Part of BotVisitor interface.

func (*DeadBotDetector) Prepare

func (r *DeadBotDetector) Prepare(ctx context.Context, shards int, lastRun time.Time)

Prepare prepares the visitor to use `shards` parallel queries.

Part of BotVisitor interface.

func (*DeadBotDetector) Visit

func (r *DeadBotDetector) Visit(ctx context.Context, shard int, bot *model.BotInfo)

Visit is called for every bot.

Part of BotVisitor interface.

type NamedCachesAggregator

type NamedCachesAggregator struct {
	// contains filtered or unexported fields
}

NamedCachesAggregator is a BotVisitor that collects all named_caches present on the bot and their respective size.

func (*NamedCachesAggregator) Finalize

func (a *NamedCachesAggregator) Finalize(ctx context.Context, scanErr error) error

Finalize is called once the scan is done.

Part of BotVisitor interface.

func (*NamedCachesAggregator) Frequency

func (*NamedCachesAggregator) Frequency() time.Duration

Frequency returns how frequently this visitor should run.

Part of BotVisitor interface.

func (*NamedCachesAggregator) ID

func (*NamedCachesAggregator) ID() string

ID returns an unique identifier of this visitor used for storing its state.

Part of BotVisitor interface.

func (*NamedCachesAggregator) Prepare

func (a *NamedCachesAggregator) Prepare(ctx context.Context, shards int, lastRun time.Time)

Prepare prepares the visitor to use `shards` parallel queries.

Part of BotVisitor interface.

func (*NamedCachesAggregator) Visit

func (a *NamedCachesAggregator) Visit(ctx context.Context, shard int, bot *model.BotInfo)

Visit is called for every bot.

Part of BotVisitor interface.

type TaskVisitor

type TaskVisitor interface {
	// Prepare prepares the visitor state.
	//
	// It should initialize the state used by Visit.
	Prepare(ctx context.Context)

	// Visit is called for every visited task.
	//
	// Called sequentially from a single goroutine.
	Visit(ctx context.Context, task *model.TaskResultSummary)

	// Finalize is called once the scan is done.
	//
	// It is passed an error if the scan was incomplete. If the scan was complete,
	// (i.e. Visit visited all tasks), it receives nil.
	//
	// The returned error will be reported as an overall scan error.
	Finalize(ctx context.Context, scanErr error) error
}

TaskVisitor examines tasks, **sequentially**.

Should keep track of errors internally during the scan, reporting them only in the end in Finalize.