filestorage

README

File Storage

Status
Stability	beta
Distributions	contrib, observiq, splunk, sumo
Issues
Code Owners	@djaglowski

The File Storage extension can persist state to the local file system.

The extension requires read and write access to a directory. A default directory can be used, but it must already exist in order for the extension to operate.

directory is the relative or absolute path to the dedicated data storage directory. The default directory is %ProgramData%\Otelcol\FileStorage on Windows and /var/lib/otelcol/file_storage otherwise.

timeout is the maximum time to wait for a file lock. This value does not need to be modified in most circumstances. The default timeout is 1s.

Compaction

compaction defines how and when files should be compacted. There are two modes of compaction available (both of which can be set concurrently):

• compaction.on_start (default: false), which happens when collector starts
• compaction.on_rebound (default: false), which happens online when certain criteria are met; it's discussed in more detail below

compaction.directory specifies the directory used for compaction (as a midstep).

compaction.max_transaction_size (default: 65536): defines maximum size of the compaction transaction. A value of zero will ignore transaction sizes.

Rebound (online) compaction

For rebound compaction, there are two additional parameters available:

• compaction.rebound_needed_threshold_mib (default: 100) - when allocated data exceeds this amount, the "compaction needed" flag will be enabled
• compaction.rebound_trigger_threshold_mib (default: 10) - if the "compaction needed" flag is set and allocated data drops below this amount, compaction will begin and the "compaction needed" flag will be cleared
• compaction.check_interval (default: 5s) - specifies how frequently the conditions for compaction are being checked

The idea behind rebound compaction is that in certain workloads (e.g. persistent queue) the storage might grow significantly (e.g. when the exporter is unable to send the data due to network problem) after which it is being emptied as the underlying issue is gone (e.g. network connectivity is back). This leaves a significant space that needs to be reclaimed (also, this space is reported in memory usage as mmap() is used underneath). The optimal conditions for this to happen online is after the storage is largely drained, which is being controlled by rebound_trigger_threshold_mib. To make sure this is not too sensitive, there's also rebound_needed_threshold_mib which specifies the total claimed space size that must be met for online compaction to even be considered. Consider following diagram for an example of meeting the rebound (online) compaction conditions.

  ▲
  │
  │             XX.............
m │            XXXX............
e ├───────────XXXXXXX..........────────────  rebound_needed_threshold_mib
m │         XXXXXXXXX..........
o │        XXXXXXXXXXX.........
r │       XXXXXXXXXXXXXXXXX....
y ├─────XXXXXXXXXXXXXXXXXXXXX..────────────  rebound_trigger_threshold_mib
  │   XXXXXXXXXXXXXXXXXXXXXXXXXX.........
  │ XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
  └──────────────── time ─────────────────►
     │           |            |
     issue       draining     compaction happens
     starts      begins       and reclaims space

 X - actually used space
 . - claimed but no longer used space

Example

extensions:
  file_storage:
  file_storage/all_settings:
    directory: /var/lib/otelcol/mydir
    timeout: 1s
    compaction:
      on_start: true
      directory: /tmp/
      max_transaction_size: 65_536

service:
  extensions: [file_storage, file_storage/all_settings]
  pipelines:
    traces:
      receivers: [nop]
      processors: [nop]
      exporters: [nop]

# Data pipeline is required to load the config.
receivers:
  nop:
processors:
  nop:
exporters:
  nop:

Documentation

Index

• func NewFactory() extension.Factory
• type CompactionConfig
• type Config
  • func (cfg *Config) Validate() error

Functions

func NewFactory

func NewFactory() extension.Factory

NewFactory creates a factory for HostObserver extension.

Types

type CompactionConfig

type CompactionConfig struct {
	// OnStart specifies that compaction is attempted each time on start
	OnStart bool `mapstructure:"on_start,omitempty"`
	// OnRebound specifies that compaction is attempted online, when rebound conditions are met.
	// This typically happens when storage usage has increased, which caused increase in space allocation
	// and afterwards it had most items removed. We want to run the compaction online only when there are
	// not too many elements still being stored (which is an indication that "heavy usage" period is over)
	// so compaction should be relatively fast and at the same time there is relatively large volume of space
	// that might be reclaimed.
	OnRebound bool `mapstructure:"on_rebound,omitempty"`
	// Directory specifies where the temporary files for compaction will be stored
	Directory string `mapstructure:"directory,omitempty"`
	// ReboundNeededThresholdMiB specifies the minimum total allocated size (both used and empty)
	// to mark the need for online compaction
	ReboundNeededThresholdMiB int64 `mapstructure:"rebound_needed_threshold_mib"`
	// ReboundTriggerThresholdMiB is used when compaction is marked as needed. When allocated data size drops
	// below the specified value, the compactions starts and the flag marking need for compaction is cleared
	ReboundTriggerThresholdMiB int64 `mapstructure:"rebound_trigger_threshold_mib"`
	// MaxTransactionSize specifies the maximum number of items that might be present in single compaction iteration
	MaxTransactionSize int64 `mapstructure:"max_transaction_size,omitempty"`
	// CheckInterval specifies frequency of compaction check
	CheckInterval time.Duration `mapstructure:"check_interval,omitempty"`
}

CompactionConfig defines configuration for optional file storage compaction.

type Config

type Config struct {
	Directory string        `mapstructure:"directory,omitempty"`
	Timeout   time.Duration `mapstructure:"timeout,omitempty"`

	Compaction *CompactionConfig `mapstructure:"compaction,omitempty"`
}

Config defines configuration for file storage extension.

func (*Config) Validate

func (cfg *Config) Validate() error