collectd

README

The collectd Input

The collectd input allows InfluxDB to accept data transmitted in collectd native format. This data is transmitted over UDP.

A note on UDP/IP OS Buffer sizes

If you're running Linux or FreeBSD, please adjust your OS UDP buffer size limit, see here for more details.

Configuration

Each collectd input allows the binding address, target database, and target retention policy to be set. If the database does not exist, it will be created automatically when the input is initialized. If the retention policy is not configured, then the default retention policy for the database is used. However if the retention policy is set, the retention policy must be explicitly created. The input will not automatically create it.

Each collectd input also performs internal batching of the points it receives, as batched writes to the database are more efficient. The default batch size is 1000, pending batch factor is 5, with a batch timeout of 1 second. This means the input will write batches of maximum size 1000, but if a batch has not reached 1000 points within 1 second of the first point being added to a batch, it will emit that batch regardless of size. The pending batch factor controls how many batches can be in memory at once, allowing the input to transmit a batch, while still building other batches.

Multi-value plugins can be handled two ways. Setting parse-multivalue-plugin to "split" will parse and store the multi-value plugin data (e.g., df free:5000,used:1000) into separate measurements (e.g., (df_free, value=5000) (df_used, value=1000)), while "join" will parse and store the multi-value plugin as a single multi-value measurement (e.g., (df, free=5000,used=1000)). "split" is the default behavior for backward compatability with previous versions of influxdb.

The path to the collectd types database file may also be set.

Large UDP packets

Please note that UDP packets larger than the standard size of 1452 are dropped at the time of ingestion. Be sure to set MaxPacketSize to 1452 in the collectd configuration.

Config Example

[[collectd]]
  enabled = true
  bind-address = ":25826" # the bind address
  database = "collectd" # Name of the database that will be written to
  retention-policy = ""
  batch-size = 5000 # will flush if this many points get buffered
  batch-pending = 10 # number of batches that may be pending in memory
  batch-timeout = "10s"
  read-buffer = 0 # UDP read buffer size, 0 means to use OS default
  typesdb = "/usr/share/collectd/types.db"
  security-level = "none" # "none", "sign", or "encrypt"
  auth-file = "/etc/collectd/auth_file"
  parse-multivalue-plugin = "split"  # "split" or "join"

Documentation

Overview

Package collectd provides a service for InfluxDB to ingest data via the collectd protocol.

Index

• Constants
• func TypesDBFile(path string) (typesdb *api.TypesDB, err error)
• type Config
  • func NewConfig() Config
  • func (c *Config) Validate() error
  • func (c *Config) WithDefaults() *Config
• type Configs
  • func (c Configs) Diagnostics() (*diagnostics.Diagnostics, error)
  • func (c Configs) Enabled() bool
• type Service
  • func NewService(c Config) *Service
  • func (s *Service) Addr() net.Addr
  • func (s *Service) Close() error
  • func (s *Service) Open() error
  • func (s *Service) SetTypes(types string) (err error)
  • func (s *Service) Statistics(tags map[string]string) []models.Statistic
  • func (s *Service) UnmarshalValueList(vl *api.ValueList) []models.Point
  • func (s *Service) UnmarshalValueListPacked(vl *api.ValueList) []models.Point
  • func (s *Service) WithLogger(log zap.Logger)
• type Statistics

Constants

const (
	// DefaultBindAddress is the default port to bind to.
	DefaultBindAddress = ":25826"

	// DefaultDatabase is the default DB to write to.
	DefaultDatabase = "collectd"

	// DefaultRetentionPolicy is the default retention policy of the writes.
	DefaultRetentionPolicy = ""

	// DefaultBatchSize is the default write batch size.
	DefaultBatchSize = 5000

	// DefaultBatchPending is the default number of pending write batches.
	DefaultBatchPending = 10

	// DefaultBatchDuration is the default batch timeout duration.
	DefaultBatchDuration = toml.Duration(10 * time.Second)

	// DefaultTypesDB is the default location of the collectd types db file.
	DefaultTypesDB = "/usr/share/collectd/types.db"

	// DefaultReadBuffer is the default buffer size for the UDP listener.
	// Sets the size of the operating system's receive buffer associated with
	// the UDP traffic. Keep in mind that the OS must be able
	// to handle the number set here or the UDP listener will error and exit.
	//
	// DefaultReadBuffer = 0 means to use the OS default, which is usually too
	// small for high UDP performance.
	//
	// Increasing OS buffer limits:
	//     Linux:      sudo sysctl -w net.core.rmem_max=<read-buffer>
	//     BSD/Darwin: sudo sysctl -w kern.ipc.maxsockbuf=<read-buffer>
	DefaultReadBuffer = 0

	// DefaultSecurityLevel is the default security level.
	DefaultSecurityLevel = "none"

	// DefaultAuthFile is the default location of the user/password file.
	DefaultAuthFile = "/etc/collectd/auth_file"

	// DefaultParseMultiValuePlugin is "split", defaulting to version <1.2 where plugin values were split into separate rows
	DefaultParseMultiValuePlugin = "split"
)

Functions

func TypesDBFile

func TypesDBFile(path string) (typesdb *api.TypesDB, err error)

TypesDBFile reads a collectd types db from a file.

Types

type Config

type Config struct {
	Enabled               bool          `toml:"enabled"`
	BindAddress           string        `toml:"bind-address"`
	Database              string        `toml:"database"`
	RetentionPolicy       string        `toml:"retention-policy"`
	BatchSize             int           `toml:"batch-size"`
	BatchPending          int           `toml:"batch-pending"`
	BatchDuration         toml.Duration `toml:"batch-timeout"`
	ReadBuffer            int           `toml:"read-buffer"`
	TypesDB               string        `toml:"typesdb"`
	SecurityLevel         string        `toml:"security-level"`
	AuthFile              string        `toml:"auth-file"`
	ParseMultiValuePlugin string        `toml:"parse-multivalue-plugin"`
}

Config represents a configuration for the collectd service.

func NewConfig

func NewConfig() Config

NewConfig returns a new instance of Config with defaults.

func (*Config) Validate

func (c *Config) Validate() error

Validate returns an error if the Config is invalid.

func (*Config) WithDefaults

func (c *Config) WithDefaults() *Config

WithDefaults takes the given config and returns a new config with any required default values set.

type Configs

type Configs []Config

Configs wraps a slice of Config to aggregate diagnostics.

func (Configs) Diagnostics

func (c Configs) Diagnostics() (*diagnostics.Diagnostics, error)

Diagnostics returns one set of diagnostics for all of the Configs.

func (Configs) Enabled

func (c Configs) Enabled() bool

Enabled returns true if any underlying Config is Enabled.

type Service

type Service struct {
	Config       *Config
	MetaClient   metaClient
	PointsWriter pointsWriter
	Logger       zap.Logger
	// contains filtered or unexported fields
}

Service represents a UDP server which receives metrics in collectd's binary protocol and stores them in InfluxDB.

func NewService

func NewService(c Config) *Service

NewService returns a new instance of the collectd service.

func (*Service) Addr

func (s *Service) Addr() net.Addr

Addr returns the listener's address. It returns nil if listener is closed.

func (*Service) Close

func (s *Service) Close() error

Close stops the service.

func (*Service) Open

func (s *Service) Open() error

Open starts the service.

func (*Service) SetTypes

func (s *Service) SetTypes(types string) (err error)

SetTypes sets collectd types db.

func (*Service) Statistics

func (s *Service) Statistics(tags map[string]string) []models.Statistic

Statistics returns statistics for periodic monitoring.

func (*Service) UnmarshalValueList

func (s *Service) UnmarshalValueList(vl *api.ValueList) []models.Point

UnmarshalValueList translates a ValueList into InfluxDB data points.

func (*Service) UnmarshalValueListPacked

func (s *Service) UnmarshalValueListPacked(vl *api.ValueList) []models.Point

UnmarshalValueListPacked is an alternative to the original UnmarshalValueList. The difference is that the original provided measurements like (PLUGIN_DSNAME, ["value",xxx]) while this one will provide measurements like (PLUGIN, {["DSNAME",xxx]}). This effectively joins collectd data that should go together, such as: (df, {["used",1000],["free",2500]}).

func (*Service) WithLogger

func (s *Service) WithLogger(log zap.Logger)

WithLogger sets the service's logger.

type Statistics

type Statistics struct {
	PointsReceived       int64
	BytesReceived        int64
	PointsParseFail      int64
	ReadFail             int64
	BatchesTransmitted   int64
	PointsTransmitted    int64
	BatchesTransmitFail  int64
	InvalidDroppedPoints int64
}

Statistics maintains statistics for the collectd service.