cache

package
v8.0.1-rc1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 21, 2024 License: Apache-2.0, BSD-2-Clause, BSD-3-Clause, + 1 more Imports: 14 Imported by: 0

Documentation

Overview

Package cache contains definitions for mechanisms used to extract health and statistics data from cache-server-provided data. The most commonly used format is the “stats_over_http” format provided by the plugin of the same name for Apache Traffic Server, followed closely by “astats” which is the legacy format used by older versions of Apache Traffic Control.

Creating A New Stats Type

To create a new Stats Type, for a custom caching proxy with its own stats format:

  1. Create a file for your type in the traffic_monitor/cache directory and package, `github.com/apache/trafficcontrol/v8/traffic_monitor/cache/`
  2. Create Parse and (optionally) Precompute functions in your file, with the signature of `StatisticsParser` and `StatisticsPrecomputer`, respectively
  3. In your file's special `init` func, call `registerDecoder` with your two functions to register the new format. The name of the format MUST be unique!
  4. To apply the new parsing format to a cache server, set its Profile's “health.polling.format“ Parameter's Value to the name of the desired format.

Your Parser should take the raw bytes from the `io.Reader` and populate the raw stats from them. It needs to provide (nearly) all of the data in a Statistics structure. Specifically, the available statistics MUST include:

  • One-minute "loadavg" value for the cache server. The others are optional, as we only use the one-minute value for health checks.
  • At least one network interface (which will be considered the one used for routing, and if multiple "monitored" network interfaces are configured for the cache server in Traffic Ops they MUST all be present) and specifically its name, “speed”, and bytes in and out. Parsers SHOULD return an error if at least one interface cannot be found in the payload data.
  • If your format does not directly indicate if the cache server is available then NotAvailable should just be set to “false”.

All other statistics (e.g. Delivery Service stats) should be returned in the map of statistic names to their values.

Your Precomputer should take the Statistics and other miscellaneous stats that your Parser created, and populate the PrecomputedData. It is essential that all PrecomputedData fields are populated, especially `DeliveryServiceStats`, as they are used for cache and Delivery Service availability and threshold computation. If PrecomputedData is not properly and fully populated, the cache's availability will not be properly computed.

Note the PrecomputedData `Reporting` and `Time` fields are the exception: they do not need to be set, and will be forcibly overridden by the Handler after your Precomputer function returns.

Note these functions will not be called for Health polls, only Stat polls. Your Cache should have two separate stats endpoints: a small light endpoint returning only system stats and used to quickly verify reachability, and a large endpoint with all stats. If your cache does not have two stat endpoints, you may use your large stat endpoint for the Health poll, and configure the Health poll interval to be arbitrarily slow. These are controlled by the “health.polling.url' Parameter in Traffic Ops.

Note your stats functions SHOULD NOT reuse functions from other stats types, even if they are similar, or have identical helper functions. This is a case where "duplicate" code is acceptable, because it's not conceptually duplicate. You don't want your stat parsers to break if the similar stats format you reuse code from changes.

Index

Examples

Constants

View Source 
const DefaultStatsType = "astats"
View Source 
const LOADAVG_SHIFT = 65536

LOADAVG_SHIFT is the amount by which "loadavg" values returned by stats_over_http need to be divided to obtain the values with which ATC operators are more familiar.

The reason for this is that the Linux kernel stores loadavg values as integral types internally, and performs conversions to floating-point numbers on-the-fly when the contents of /proc/loadavg are read. Since astats_over_http used to always just read that file to get the numbers, everyone's used to the floating-point form. But stats_over_http gets the numbers directly from a syscall, so they aren't pre-converted for us.

Dividing by this number is kind of a shortcut, for the actual transformation used by the kernel itself, refer to the source: https://github.com/torvalds/linux/blob/master/fs/proc/loadavg.c

Variables

This section is empty.

Functions

func ComputedStats 

func ComputedStats() map[string]StatComputeFunc

ComputedStats returns a map of cache stats which are computed by Traffic Monitor (rather than returned literally from ATS), mapped to the function to compute them.

Types

type Astats 

type Astats struct {
	Ats    map[string]interface{} `json:"ats"`
	System AstatsSystem           `json:"system"`
}

Astats contains ATS data returned from the Astats ATS plugin. This includes generic stats, as well as fixed system stats.

type AstatsSystem 

type AstatsSystem struct {
	InfName           string `json:"inf.name"`
	InfSpeed          int    `json:"inf.speed"`
	ProcNetDev        string `json:"proc.net.dev"`
	ProcLoadavg       string `json:"proc.loadavg"`
	ConfigLoadRequest int    `json:"configReloadRequests"`
	LastReloadRequest int    `json:"lastReloadRequest"`
	ConfigReloads     int    `json:"configReloads"`
	LastReload        int    `json:"lastReload"`
	AstatsLoad        int    `json:"astatsLoad"`
	NotAvailable      bool   `json:"notAvailable,omitempty"`
}

AstatsSystem represents fixed system stats returned from the 'astats_over_http' ATS plugin.

type AvailableStatus 

type AvailableStatus struct {
	// Available indicates whether a Cache Server is available for various IP
	// protocol versions.
	Available          AvailableTuple
	ProcessedAvailable bool
	LastCheckedIPv4    bool
	// The name of the actual status the cache server has, as configured in
	// Traffic Ops.
	Status string
	// `Why` will contain the reason a cache server has been purposely marked
	// unavailable by a Traffic Ops operator, if indeed that has occurred.
	Why string
	// UnavailableStat is the stat whose threshold made the cache unavailable.
	// If this is the empty string, the cache is unavailable for a
	// non-threshold reason. This exists so a poller (health, stat) won't mark
	// an unavailable cache as available if the stat whose threshold was
	// reached isn't available on that poller.
	UnavailableStat string
	// Poller is the name of the poller which set this availability status.
	Poller string
}

AvailableStatus is the available status of the given cache. It includes a boolean available/unavailable flag, and a descriptive string.

type AvailableStatuses 

type AvailableStatuses map[string]AvailableStatus

CacheAvailableStatuses is the available status of each cache.

func (AvailableStatuses) Copy 

func (a AvailableStatuses) Copy() AvailableStatuses

Copy copies this CacheAvailableStatuses. It does not modify, and thus is safe for multiple reader goroutines.

type AvailableTuple 

type AvailableTuple struct {
	IPv4 bool
	IPv6 bool
}

AvailableTuple contains a boolean value to indicate whether IPv4 is available and a boolean value to indicate whether IPv6 is available.

func (*AvailableTuple) SetAvailability 

func (a *AvailableTuple) SetAvailability(usingIPv4 bool, isAvailable bool)

SetAvailablility sets two booleans to indicate whether IPv4 is available and whether IPv6 is available.

type DSStat 

type DSStat struct {
	// InBytes is the total number of bytes received by the cache server which
	// were for the Delivery Service.
	InBytes uint64
	// OutBytes is the total number of bytes transmitted by the cache server
	// in service of the Delivery Service.
	OutBytes uint64
	// Status2xx is the number of requests for the Delivery Service's content
	// that were served with responses having response codes on the interval
	// [200, 300).
	Status2xx uint64
	// Status3xx is the number of requests for the Delivery Service's content
	// that were served with responses having response codes on the interval
	// [300, 400).
	Status3xx uint64
	// Status4xx is the number of requests for the Delivery Service's content
	// that were served with responses having response codes on the interval
	// [400, 500).
	Status4xx uint64
	// Status2xx is the number of requests for the Delivery Service's content
	// that were served with responses having response codes on the interval
	// [500, 600).
	Status5xx uint64
}

DSStat is a single Delivery Service statistic, which is associated with a particular cache server. DSStats are referenced by name in a map associated with cache server data, and so that name (XMLID) is not reiterated here.

type Filter 

type Filter interface {
	UseCache(tc.CacheName) bool
	UseInterfaceStat(string) bool
	UseStat(string) bool
	WithinStatHistoryMax(uint64) bool
}

Filter filters whether stats and caches should be returned from a data set.

type Handler 

type Handler struct {
	ToData *todata.TODataThreadsafe
	// contains filtered or unexported fields
}

Handler is a cache handler, which fulfills the common/handler `Handler` interface.

func NewHandler 

func NewHandler() Handler

NewHandler returns a new cache handler. Note this handler does NOT precompute stat data before calling ResultChan, and Result.Precomputed will be nil.

func NewPrecomputeHandler 

func NewPrecomputeHandler(toData todata.TODataThreadsafe) Handler

NewPrecomputeHandler constructs a new cache Handler, which precomputes stat data and populates result.Precomputed before passing to ResultChan.

func (Handler) Handle 

func (handler Handler) Handle(id string, rdr io.Reader, format string, reqTime time.Duration, reqEnd time.Time, reqErr error, pollID uint64, usingIPv4 bool, pollCtx interface{}, pollFinished chan<- uint64)

Handle handles results fetched from a cache, parsing the raw Reader data and passing it along to a chan for further processing.

func (Handler) Precompute 

func (handler Handler) Precompute() bool

Precompute returns whether this handler precomputes data before passing the result to the ResultChan

func (Handler) ResultChan 

func (h Handler) ResultChan() <-chan Result

type Interface 

type Interface struct {
	// Speed is the "speed" of the interface, which is of unknown - but vitally
	// important - meaning.
	Speed int64
	// BytesOut is the total number of bytes transmitted by this interface.
	BytesOut uint64
	// BytesIn is the total number of bytes received by this interface.
	BytesIn uint64
}

Interface represents a network interface. The name of the interface is used to access it within a Statistics object, and so is not stored here.

type Kbpses 

type Kbpses map[string]uint64

Kbpses is the kbps values of each interface of a cache server.

func (Kbpses) Copy 

func (a Kbpses) Copy() Kbpses

Copy returns a deep copy of the Kbpses.

type Loadavg 

type Loadavg struct {
	// One is the cache server's "loadavg" in the past minute from the time it was
	// polled.
	One float64
	// Five is the cache server's "loadavg" in the past five minutes from the time
	// it was polled.
	Five float64
	// Fifteen is the cache server's "loadavg" in the past fifteen minutes from the
	// time it was polled.
	Fifteen float64
	// CurrentProcesses is the number of currently executing processes (or threads)
	// on the cache server.
	// Note that stats_over_http doesn't provide this, so in general it can't be
	// relied on to be set properly.
	CurrentProcesses uint64
	// TotalProcesses is the number of total processes (or threads) that exist on
	// the cache server.
	TotalProcesses uint64
	// LatestPID is the process ID of the most recently created process on the
	// cache server at the time of polling.
	// Note that stats_over_http doesn't provide this, so in general it can't be
	// relied on to be set properly - which is fine because what use could that
	// information actually have??
	LatestPID int64
}

Loadavg contains the parsed "loadavg" data for a polled cache server. Specifically, it contains all of the data stored that can be found in /proc/loadavg on a Linux/Unix system.

For more information on what a "loadavg" is, consult the “proc(5)” man page (web-hosted: https://linux.die.net/man/5/proc).

func LoadavgFromRawLine 

func LoadavgFromRawLine(line string) (Loadavg, error)

LoadavgFromRawLine parses a raw line - presumably read from /proc/loadavg - and returns a Loadavg containing all of the same information, as well as any encountered error.

Example 
loadavg, err := LoadavgFromRawLine("0.30 0.12 0.21 1/863 1421")
fmt.Println(err)
fmt.Printf("%.2f %.2f %.2f %d/%d %d", loadavg.One, loadavg.Five, loadavg.Fifteen, loadavg.CurrentProcesses, loadavg.TotalProcesses, loadavg.LatestPID)

Output:

<nil>
0.30 0.12 0.21 1/863 1421

type PrecomputedData 

type PrecomputedData struct {
	DeliveryServiceStats map[string]*DSStat
	// This is the total bytes transmitted by all interfaces on the Cache
	// Server.
	OutBytes uint64
	// MaxKbps is the maximum bandwidth of all interfaces on the Cache Server,
	// each one calculated as the speed of the interface in Kbps.
	MaxKbps   int64
	Errors    []error
	Reporting bool
	Time      time.Time
}

PrecomputedData represents data parsed and pre-computed from the Result.

type Result 

type Result struct {
	// Available indicates whether or not the cache server should be considered
	// "available" based on its status as configured in Traffic Ops, the cache
	// server's own reported availability (if applicable), and the polled
	// vitals and statistics as compared to threshold values.
	Available bool
	// Error holds what error - if any - caused the statistic polling to fail.
	Error error
	// ID is the fully qualified domain name of the cache server being polled.
	// (This is assumed to be unique even though that isn't necessarily true)
	ID string
	// Miscellaneous contains the stats that were not directly gathered into
	// Statistics, but were still found in the stats polling payload. Their
	// contents are NOT guaranteed in ANY way.
	Miscellaneous map[string]interface{}
	// PollFinished is a channel to which data should be sent to indicate that
	// polling has been completed and a Result has been produced.
	PollFinished chan<- uint64
	// PollID is a unique identifier for the specific polling instance that
	// produced this Result.
	PollID          uint64
	PrecomputedData PrecomputedData
	// RequestTime holds the elapsed duration between making a statistics
	// polling request and either receiving a result or giving up.
	RequestTime time.Duration
	// Statistics holds the parsed statistic data returned by the cache server.
	Statistics Statistics
	// Time is the time at which the result has been obtained.
	Time time.Time
	// UsingIPv4 indicates whether IPv4 can/should be/was used by the polling
	// instance that produced this Result. If “false”, it may be assumed that
	// IPv6 was used instead.
	UsingIPv4 bool
	// Vitals holds the parsed health information returned by the cache server.
	Vitals Vitals
	// InterfaceVitals holds the parsed health information returned by the cache server per interface.
	InterfaceVitals map[string]Vitals
}

Result is a result of polling a cache server for statistics.

func (*Result) HasStat 

func (result *Result) HasStat(stat string) bool

HasStat returns whether the given stat is in the Result.

func (*Result) Interfaces 

func (result *Result) Interfaces() map[string]Interface

Interfaces returns the interfaces assigned to this result.

func (*Result) InterfacesNames 

func (result *Result) InterfacesNames() []string

InterfacesNames returns the names of all network interfaces used by the cache server that was monitored to obtain the result.

type ResultHistory 

type ResultHistory map[tc.CacheName][]Result

ResultHistory is a map of cache names, to an array of result history from each cache server.

func (ResultHistory) Copy 

func (a ResultHistory) Copy() ResultHistory

Copy copies returns a deep copy of this ResultHistory.

type ResultInfo 

type ResultInfo struct {
	Available       bool
	Error           error
	ID              string
	PollID          uint64
	RequestTime     time.Duration
	Statistics      Statistics
	Time            time.Time
	UsingIPv4       bool
	Vitals          Vitals
	InterfaceVitals map[string]Vitals
}

ResultInfo contains all the non-stat result info. This includes the cache ID, any errors, the time of the poll, the request time duration, Astats System (Vitals), Poll ID, and Availability. TODO: Determine why this exists, it doesn't seem to differ from Result in any meaningful way.

func ToInfo 

func ToInfo(r Result) ResultInfo

type ResultInfoHistory 

type ResultInfoHistory map[tc.CacheName][]ResultInfo

TODO determine if anything ever needs more than the latest, and if not, change ResultInfo to not be a slice.

func (ResultInfoHistory) Add 

func (a ResultInfoHistory) Add(r Result, limit uint64)

func (ResultInfoHistory) Copy 

func (a ResultInfoHistory) Copy() ResultInfoHistory

type Stat 

type Stat struct {
	Time  int64       `json:"time"`
	Value interface{} `json:"value"`
}

Stat is a generic stat, including the untyped value and the time the stat was taken.

type StatComputeFunc 

type StatComputeFunc func(ResultInfo, tc.TrafficServer, tc.TMProfile, tc.IsAvailable) interface{}

StatComputeFunc functions calculate a specific statistic given a set of polling results, server and profile information, whether or not the server is available, and the name of the specific network interface for which stats will be computed.

type Statistics 

type Statistics struct {
	// Loadavg contains the Unix/Linux "loadavg" values for the cache server.
	Loadavg Loadavg
	// Interfaces is a map of network interface names to statistic data about
	// those interfaces.
	Interfaces map[string]Interface
	// NotAvailable reports whether or not the cache server is unavailable.
	// Sometimes caches can directly report this, but it's not supported by
	// stats_over_http (afaik), so it always just uses “false”
	NotAvailable bool
}

Statistics is a structure containing, most generally, the statistics of a cache server.

func (*Statistics) AddInterfaceFromRawLine 

func (s *Statistics) AddInterfaceFromRawLine(line string) error

AddInterfaceFromRawLine parses the raw line - presumably read from /proc/net/dev - and inserts into the Statistics a new Interface containing the data provided.

This will initialize s.Interfaces if that has not already been done (if the parse is successful).

If the line cannot be parsed, s.Interfaces is unchanged and an error describing the problem is returned.

Note that this does *not* set the interface's Speed.

Example 
var s Statistics
raw := "eth0:47907832129 14601260    0    0    0     0          0   790726 728207677726 10210700052    0    0    0     0       0          0"

if err := s.AddInterfaceFromRawLine(raw); err != nil {
	fmt.Println(err)
	return
}

iface, ok := s.Interfaces["eth0"]
if !ok {
	fmt.Printf("Error, no 'eth0' interface!\n%+v", s.Interfaces)
	return
}
fmt.Printf("eth0: {BytesOut: %d, BytesIn: %d}", iface.BytesOut, iface.BytesIn)

Output:

eth0: {BytesOut: 728207677726, BytesIn: 47907832129}

type StatisticsParser 

type StatisticsParser func(string, io.Reader, interface{}) (Statistics, map[string]interface{}, error)

StatisticsParser is a function that parses raw input data for a given statistics format and returns the meaningful statistics. In addition to the decoded statistics, the decoder should also return whatever miscellaneous data was in the payload but not represented by the properties of a Statistics object, so that it can be used in later calculations if necessary.

type StatisticsPrecomputer 

type StatisticsPrecomputer func(string, todata.TOData, Statistics, map[string]interface{}) PrecomputedData

StatisticsPrecomputer is a function that "pre-computes" some statistics beyond the basic ones covered by a Statistics object. Precomputers aren't called until a statistics poll is done, whereas basic Statistics are calculated even for Health polls.

type StatsDecoder 

type StatsDecoder struct {
	Parse      StatisticsParser
	Precompute StatisticsPrecomputer
}

StatsDecoder is a pair of functions registered for decoding statistics of a particular format, and parsing that data and precomputing related data, respectively.

func GetDecoder 

func GetDecoder(format string) (StatsDecoder, error)

GetDecoder gets a decoder for the given statistic format. Returns an error if no parser for the given format exists.

type Vitals 

type Vitals struct {
	// LoadAvg is the one-minute "loadavg" of the cache server.
	LoadAvg    float64
	BytesOut   uint64
	BytesIn    uint64
	KbpsOut    int64
	MaxKbpsOut int64
}

Vitals is the vitals data returned from a cache.

type Vstats 

type Vstats struct {
	ProcNetDev   string                 `json:"proc.net.dev"`
	ProcLoadAvg  string                 `json:"proc.loadavg"`
	NotAvailable bool                   `json:"not_available"`
	InfSpeed     int64                  `json:"inf_speed"`
	Stats        map[string]interface{} `json:"stats"`
}

Vstats holds Varnish cache statistics

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.