monitor

README

System Monitoring

This functionality should be considered experimental and is subject to change.

System Monitoring means all statistical and diagnostic information made availabe to the user of InfluxDB system, about the system itself. Its purpose is to assist with troubleshooting and performance analysis of the database itself.

Statistics vs. Diagnostics

A distinction is made between statistics and diagnostics for the purposes of monitoring. Generally a statistical quality is something that is being counted, and for which it makes sense to store persistently for historical analysis. Diagnostic information is not necessarily numerical, and may not make sense to store.

An example of statistical information would be the number of points received over UDP, or the number of queries executed. Examples of diagnostic information would be a list of current Graphite TCP connections, the version of InfluxDB, or the uptime of the process.

System Statistics

SHOW STATS [FOR <module>] displays statisics about subsystems within the running influxd process. Statistics include points received, points indexed, bytes written to disk, TCP connections handled etc. These statistics are all zero when the InfluxDB process starts. If module is specified, it must be single-quoted. For example SHOW STATS FOR 'httpd'.

All statistics are written, by default, by each node to a "monitor" database within the InfluxDB system, allowing analysis of aggregated statistical data using the standard InfluxQL language. This allows users to track the performance of their system. Importantly, this allows cluster-level statistics to be viewed, since by querying the monitor database, statistics from all nodes may be queried. This can be a very powerful approach for troubleshooting your InfluxDB system and understanding its behaviour.

System Diagnostics

SHOW DIAGNOSTICS [FOR <module>] displays various diagnostic information about the influxd process. This information is not stored persistently within the InfluxDB system. If module is specified, it must be single-quoted. For example SHOW STATS FOR 'build'.

Standard expvar support

All statistical information is available at HTTP API endpoint /debug/vars, in expvar format, allowing external systems to monitor an InfluxDB node. By default, the full path to this endpoint is http://localhost:8086/debug/vars.

Configuration

The monitor module allows the following configuration:

• Whether to write statistical and diagnostic information to an InfluxDB system. This is enabled by default.
• The name of the database to where this information should be written. Defaults to _internal. The information is written to the default retention policy for the given database.
• The name of the retention policy, along with full configuration control of the retention policy, if the default retention policy is not suitable.
• The rate at which this information should be written. The default rate is once every 10 seconds.

Design and Implementation

A new module named monitor supports all basic statistics and diagnostic functionality. This includes:

• Allowing other modules to register statistics and diagnostics information, allowing it to be accessed on demand by the monitor module.
• Serving the statistics and diagnostic information to the user, in response to commands such as SHOW DIAGNOSTICS.
• Expose standard Go runtime information such as garbage collection statistics.
• Make all collected expvar data via HTTP, for collection by 3rd-party tools.
• Writing the statistical information to the "monitor" database, for query purposes.

Registering statistics and diagnostics

To export statistical information with the monitor system, code simply calls influxdb.NewStatistics() and receives an expvar.Map instance in response. This object can then be used to store statistics.

For example, if you have a component called Service, you can statistics like so:

import (
     "expvar"
     "github.com/influxdb/influxdb"
)
.
.
.
.
type Service struct {
  ....some other fields....
  statMap *expvar.Map         /// Add a map of type *expvar.Map. Check GoDocs for how to use this.
}


func NewService() *Service {
  s = &NewService{}
  s.statMap = NewStatistics(key, name, tags)
}

When calling NewStatistics key should be unique for the Service instance (if a network service, the protocol and binding port are good to include in the key). name will be the name of the Measurement used to store these statistics. Finally, when these statistics are written to the monitor database, all points will be tagged with tags. A value of nil for tags is legal.

To register diagnostic information, monitor.RegisterDiagnosticsClient is called, passing a influxdb.monitor.DiagsClient object to monitor. Implementing the influxdb.monitor.DiagsClient interface requires that your component have function returning diagnostic information in specific form, so that it can be displayed by the monitor system.

expvar

Statistical information is gathered by each package using expvar. Each package registers a map using its package name.

Due to the nature of expvar, statistical information is reset to its initial state when a server is restarted.

Documentation

Index

• Constants
• type Config
  • func NewConfig() Config
• type Diagnostic
  • func DiagnosticFromMap(m map[string]interface{}) *Diagnostic
  • func NewDiagnostic(columns []string) *Diagnostic
  • func (d *Diagnostic) AddRow(r []interface{})
• type DiagsClient
• type DiagsClientFunc
  • func (f DiagsClientFunc) Diagnostics() (*Diagnostic, error)
• type Monitor
  • func New(c Config) *Monitor
  • func (m *Monitor) Close()
  • func (m *Monitor) DeregisterDiagnosticsClient(name string)
  • func (m *Monitor) Diagnostics() (map[string]*Diagnostic, error)
  • func (m *Monitor) Open() error
  • func (m *Monitor) RegisterDiagnosticsClient(name string, client DiagsClient)
  • func (m *Monitor) SetLogger(l *log.Logger)
  • func (m *Monitor) Statistics(tags map[string]string) ([]*Statistic, error)
• type StatementExecutor
  • func (s *StatementExecutor) ExecuteStatement(stmt influxql.Statement) *influxql.Result
• type Statistic

Constants

const (
	// DefaultStoreEnabled is whether the system writes gathered information in
	// an InfluxDB system for historical analysis.
	DefaultStoreEnabled = true

	// DefaultStoreDatabase is the name of the database where gathered information is written
	DefaultStoreDatabase = "_internal"

	// DefaultStoreInterval is the period between storing gathered information.
	DefaultStoreInterval = 10 * time.Second
)

const (
	MonitorRetentionPolicy         = "monitor"
	MonitorRetentionPolicyDuration = 7 * 24 * time.Hour
)

Policy constants.

Types

type Config

type Config struct {
	StoreEnabled  bool          `toml:"store-enabled"`
	StoreDatabase string        `toml:"store-database"`
	StoreInterval toml.Duration `toml:"store-interval"`
}

Config represents the configuration for the monitor service.

func NewConfig

func NewConfig() Config

NewConfig returns an instance of Config with defaults.

type Diagnostic

type Diagnostic struct {
	Columns []string
	Rows    [][]interface{}
}

Diagnostic represents a table of diagnostic information. The first value is the name of the columns, the second is a slice of interface slices containing the values for each column, by row. This information is never written to an InfluxDB system and is display-only. An example showing, say, connections follows:

source_ip    source_port       dest_ip     dest_port
182.1.0.2    2890              127.0.0.1   38901
174.33.1.2   2924              127.0.0.1   38902

func DiagnosticFromMap

func DiagnosticFromMap(m map[string]interface{}) *Diagnostic

DiagnosticFromMap returns a Diagnostic from a map.

func NewDiagnostic

func NewDiagnostic(columns []string) *Diagnostic

NewDiagnostic initialises a new Diagnostic with the specified columns.

func (*Diagnostic) AddRow

func (d *Diagnostic) AddRow(r []interface{})

AddRow appends the provided row to the Diagnostic's rows.

type DiagsClient

type DiagsClient interface {
	Diagnostics() (*Diagnostic, error)
}

DiagsClient is the interface modules implement if they register diags with monitor.

type DiagsClientFunc

type DiagsClientFunc func() (*Diagnostic, error)

The DiagsClientFunc type is an adapter to allow the use of ordinary functions as Diagnostis clients.

func (DiagsClientFunc) Diagnostics

func (f DiagsClientFunc) Diagnostics() (*Diagnostic, error)

Diagnostics calls f().

type Monitor

type Monitor struct {
	// Build information for diagnostics.
	Version   string
	Commit    string
	Branch    string
	BuildTime string

	MetaStore interface {
		ClusterID() (uint64, error)
		NodeID() uint64
		WaitForLeader(d time.Duration) error
		IsLeader() bool
		CreateDatabaseIfNotExists(name string) (*meta.DatabaseInfo, error)
		CreateRetentionPolicyIfNotExists(database string, rpi *meta.RetentionPolicyInfo) (*meta.RetentionPolicyInfo, error)
		SetDefaultRetentionPolicy(database, name string) error
		DropRetentionPolicy(database, name string) error
	}

	PointsWriter interface {
		WritePoints(p *cluster.WritePointsRequest) error
	}

	Logger *log.Logger
	// contains filtered or unexported fields
}

Monitor represents an instance of the monitor system.

func New

func New(c Config) *Monitor

New returns a new instance of the monitor system.

func (*Monitor) Close

func (m *Monitor) Close()

Close closes the monitor system.

func (*Monitor) DeregisterDiagnosticsClient

func (m *Monitor) DeregisterDiagnosticsClient(name string)

DeregisterDiagnosticsClient deregisters a diagnostics client by name.

func (*Monitor) Diagnostics

func (m *Monitor) Diagnostics() (map[string]*Diagnostic, error)

Diagnostics fetches diagnostic information for each registered diagnostic client. It skips any clients that return an error when retrieving their diagnostics.

func (*Monitor) Open

func (m *Monitor) Open() error

Open opens the monitoring system, using the given clusterID, node ID, and hostname for identification purpose.

func (*Monitor) RegisterDiagnosticsClient

func (m *Monitor) RegisterDiagnosticsClient(name string, client DiagsClient)

RegisterDiagnosticsClient registers a diagnostics client with the given name and tags.

func (*Monitor) SetLogger

func (m *Monitor) SetLogger(l *log.Logger)

SetLogger sets the internal logger to the logger passed in.

func (*Monitor) Statistics

func (m *Monitor) Statistics(tags map[string]string) ([]*Statistic, error)

Statistics returns the combined statistics for all expvar data. The given tags are added to each of the returned statistics.

type StatementExecutor

type StatementExecutor struct {
	Monitor interface {
		Statistics(map[string]string) ([]*Statistic, error)
		Diagnostics() (map[string]*Diagnostic, error)
	}
}

StatementExecutor translates InfluxQL queries to Monitor methods.

func (*StatementExecutor) ExecuteStatement

func (s *StatementExecutor) ExecuteStatement(stmt influxql.Statement) *influxql.Result

ExecuteStatement executes monitor-related query statements.

type Statistic

type Statistic struct {
	Name   string                 `json:"name"`
	Tags   map[string]string      `json:"tags"`
	Values map[string]interface{} `json:"values"`
}

Statistic represents the information returned by a single monitor client.