logconfig

package
v0.0.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 20, 2022 License: Apache-2.0 Imports: 15 Imported by: 0

Documentation

Overview

Package logconfig manages the configuration of the logging channels and their logging sinks.

General format of the command-line flags:

--log=<yaml>

The YAML configuration format works as follows:

file-defaults: #optional
  dir: <path>           # output directory, defaults to first store dir
  max-file-size: <sz>   # max log file size, default 10MB
  max-group-size: <sz>  # max log file group size, default 100MB
  sync-writes: <bool>   # whether to sync each write, default false
  <common sink parameters>

sinks: #optional
 stderr: #optional
  channels: <chans>        # channel selection for stderr output, default ALL
  <common sink parameters> # if not specified, inherit from file-defaults

 file-groups: #optional
   <filename>:
     channels: <chans>        # channel selection for this file output, mandatory
     max-file-size: <sz>      # defaults to file-defaults.max-file-size
     max-group-size: <sz>     # defaults to file-defaults.max-group-size
     sync-writes: <bool>      # defaults to file-defaults.sync-writes
     <common sink parameters> # if not specified, inherit from file-defaults

   ... repeat ...

capture-stray-errors: #optional
  enable: <bool>       # whether to enable internal fd2 capture
  dir: <optional>      # output directory, defaults to file-defaults.dir
  max-group-size: <sz> # defaults to file-defaults.max-group-size

<common sink parameters>
  filter: <severity>    # min severity level for file output, default INFO
  redact: <bool>        # whether to remove sensitive info, default false
  redactable: <bool>    # whether to strip redaction markers, default false
  format: <fmt>         # format to use for log enries, default
                        # crdb-v1 for files, crdb-v1-tty for stderr
  exit-on-error: <bool> # whether to terminate upon a write error
                        # default true for file+stderr sinks
  auditable: <bool>     # if true, activates sink-specific features
                        # that enhance non-repudiability.
                        # also implies exit-on-error: true.

Index

Constants

View Source
const DefaultFileFormat = `crdb-v2`

DefaultFileFormat is the entry format for file sinks when not specified in a configuration.

View Source
const DefaultFluentFormat = `json-fluent-compact`

DefaultFluentFormat is the entry format for fluent sinks when not specified in a configuration.

View Source
const DefaultHTTPFormat = `json-compact`

DefaultHTTPFormat is the entry format for HTTP sinks when not specified in a configuration.

View Source
const DefaultStderrFormat = `crdb-v2-tty`

DefaultStderrFormat is the entry format for stderr sinks when not specified in a configuration.

Variables

This section is empty.

Functions

func AllChannels

func AllChannels() []logpb.Channel

AllChannels returns a copy of channelValues, for use in the ALL configuration.

Types

type ByteSize

type ByteSize uint64

ByteSize represents a size in bytes.

func (ByteSize) IsZero

func (x ByteSize) IsZero() bool

IsZero implements the yaml.IsZeroer interface.

func (ByteSize) MarshalYAML

func (x ByteSize) MarshalYAML() (interface{}, error)

MarshalYAML implements the yaml.Marshaler interface.

func (ByteSize) String

func (x ByteSize) String() string

String implements the fmt.Stringer interface.

func (*ByteSize) UnmarshalYAML

func (x *ByteSize) UnmarshalYAML(fn func(interface{}) error) error

UnmarshalYAML implements the yaml.Unmarshaler interface.

type CaptureFd2Config

type CaptureFd2Config struct {
	// Enable determine whether the fd2 capture is enabled.
	Enable bool

	// Dir indicates where to generate the cockroach-stderr.log file
	// when the capture is enabled.
	Dir *string `yaml:",omitempty"`

	// MaxGroupSize stores the default approximate maximum combined size
	// of all fd2 capture files.
	// Garbage collection removes files that cause the file set to grow
	// beyond this specified size.
	MaxGroupSize *ByteSize `yaml:"max-group-size,omitempty"`
}

CaptureFd2Config represents the configuration for the fd2 capture sink.

type ChannelFilters

type ChannelFilters struct {
	// Filters represents the input configuration.
	Filters map[logpb.Severity]ChannelList

	// AllChannels lists all the channels listed in the configuration.
	// Populated/overwritten by Update().
	AllChannels ChannelList
	// ChannelFilters inverts the configured filters, for
	// use internally when instantiating the configuration.
	// Populated/overwritten by Update().
	ChannelFilters map[logpb.Channel]logpb.Severity
}

ChannelFilters represents a map of severities to channels.

func SelectChannels

func SelectChannels(chs ...logpb.Channel) ChannelFilters

SelectChannels is a constructor for ChannelFilters that helps in tests.

func (*ChannelFilters) AddChannel

func (c *ChannelFilters) AddChannel(ch logpb.Channel, sev logpb.Severity)

AddChannel adds the given channel at the specified severity.

func (ChannelFilters) MarshalYAML

func (c ChannelFilters) MarshalYAML() (interface{}, error)

MarshalYAML implements the yaml.Marshaler interface.

func (*ChannelFilters) UnmarshalYAML

func (c *ChannelFilters) UnmarshalYAML(fn func(interface{}) error) error

UnmarshalYAML implements the yaml.Unmarshaler interface.

func (*ChannelFilters) Validate

func (c *ChannelFilters) Validate(defSev logpb.Severity) error

Validate changes the expressed filters to substitute the UNKNOWN severity by the proposed default; it also propagates the user configuration into the other accessor fields.

type ChannelList

type ChannelList struct {
	Channels []logpb.Channel
}

ChannelList represents a list of channels.

func (ChannelList) HasChannel

func (c ChannelList) HasChannel(ch logpb.Channel) bool

HasChannel returns true iff the list contains the given channel.

func (ChannelList) MarshalYAML

func (c ChannelList) MarshalYAML() (interface{}, error)

MarshalYAML implements the yaml.Marshaler interface.

func (*ChannelList) Set

func (c *ChannelList) Set(v string) error

Set implements the pflag.Value interface.

func (*ChannelList) Sort

func (c *ChannelList) Sort()

Sort ensures a channel list is sorted, for stability.

func (ChannelList) String

func (c ChannelList) String() string

String implements the fmt.Stringer and pflag.Value interfaces.

func (ChannelList) Type

func (c ChannelList) Type() string

Type implements the pflag.Value interface.

func (*ChannelList) UnmarshalYAML

func (c *ChannelList) UnmarshalYAML(fn func(interface{}) error) error

UnmarshalYAML implements the yaml.Unmarshaler interface.

type CommonBufferSinkConfig

type CommonBufferSinkConfig struct {
	// MaxStaleness is the maximum time a log message will sit in the buffer
	// before a flush is triggered.
	MaxStaleness *time.Duration `yaml:"max-staleness,omitempty"`

	// FlushTriggerSize is the number of bytes that will trigger the buffer
	// to flush.
	FlushTriggerSize *ByteSize `yaml:"flush-trigger-size,omitempty"`

	// MaxInFlight is the maximum number of buffered flushes before messages
	// start being dropped.
	MaxInFlight *int `yaml:"max-in-flight,omitempty"`
}

CommonBufferSinkConfig represents the common buffering configuration for sinks.

User-facing documentation follows. TITLE: Common buffering configuration Buffering may be configured with the following fields. It may also be explicitly set to "NONE" to disable buffering. Example configuration:

file-defaults:
   dir: logs
   buffering:
      max-staleness: 20s
      flush-trigger-size: 25KB
sinks:
   file-groups:
      health:
         channels: HEALTH
         buffering:
            max-staleness: 5s  # Override max-staleness for this sink.
      ops:
         channels: OPS
         buffering: NONE  # Disable buffering for this sink.

type CommonBufferSinkConfigWrapper

type CommonBufferSinkConfigWrapper struct {
	CommonBufferSinkConfig
}

CommonBufferSinkConfigWrapper is a BufferSinkConfig with a special value represented in YAML by the string "NONE", which actively disables buffering (in the sense that it overrides buffering that is enabled by default). This is a separate type so that marshaling and unmarshaling of the inner BufferSinkConfig can be handled by the library without causing infinite recursion.

func (CommonBufferSinkConfigWrapper) IsNone

IsNone before default propagation indicates that the config explicitly disables buffering, such that no propagated defaults can actiate them. After default propagation, buffering is disabled iff IsNone().

func (CommonBufferSinkConfigWrapper) MarshalYAML

func (w CommonBufferSinkConfigWrapper) MarshalYAML() (interface{}, error)

MarshalYAML implements yaml.Marshaler interface.

func (*CommonBufferSinkConfigWrapper) UnmarshalYAML

func (w *CommonBufferSinkConfigWrapper) UnmarshalYAML(fn func(interface{}) error) error

UnmarshalYAML implements the yaml.Unmarshaler interface.

type CommonSinkConfig

type CommonSinkConfig struct {
	// Filter specifies the default minimum severity for log events to
	// be emitted to this sink, when not otherwise specified by the
	// 'channels' sink attribute.
	Filter logpb.Severity `yaml:",omitempty"`

	// Format indicates the entry format to use.
	Format *string `yaml:",omitempty"`

	// Redact indicates whether to strip sensitive information before
	// log events are emitted to this sink.
	Redact *bool `yaml:",omitempty"`

	// Redactable indicates whether to keep redaction markers in the
	// sink's output. The presence of redaction markers makes it possible
	// to strip sensitive data reliably.
	Redactable *bool `yaml:",omitempty"`

	// Criticality indicates whether the logging system should terminate
	// the process if an error is encountered while writing to this
	// sink.
	Criticality *bool `yaml:"exit-on-error,omitempty"`

	// Auditable is translated to tweaks to the other settings
	// for this sink during validation. For example,
	// it enables `exit-on-error` and changes the format of files
	// from `crdb-v1` to `crdb-v1-count`.
	Auditable *bool `yaml:",omitempty"`

	// Buffering configures buffering for this log sink, or NONE to explicitly disable.
	Buffering CommonBufferSinkConfigWrapper `yaml:",omitempty"`
}

CommonSinkConfig represents the common configuration shared across all sinks.

type Config

type Config struct {
	// FileDefaults represents the default configuration for file sinks,
	// inherited when a specific file sink config does not provide a
	// configuration value.
	FileDefaults FileDefaults `yaml:"file-defaults,omitempty"`

	// FluentDefaults represents the default configuration for fluent sinks,
	// inherited when a specific fluent sink config does not provide a
	// configuration value.
	FluentDefaults FluentDefaults `yaml:"fluent-defaults,omitempty"`

	// HTTPDefaults represents the default configuration for HTTP sinks,
	// inherited when a specific HTTP sink config does not provide a
	// configuration value.
	HTTPDefaults HTTPDefaults `yaml:"http-defaults,omitempty"`

	// Sinks represents the sink configurations.
	Sinks SinkConfig `yaml:",omitempty"`

	// CaptureFd2 represents the configuration for the redirection of
	// internal writes to file descriptor 2 (incl that done internally
	// by the go runtime).
	CaptureFd2 CaptureFd2Config `yaml:"capture-stray-errors,omitempty"`
}

Config represents the top-level logging configuration.

func DefaultConfig

func DefaultConfig() (c Config)

DefaultConfig returns a suitable default configuration when logging is meant to primarily go to files.

func DefaultStderrConfig

func DefaultStderrConfig() (c Config)

DefaultStderrConfig returns a suitable default configuration when logging is meant to primarily go to stderr.

func (*Config) Export

func (c *Config) Export(onlyChans ChannelList) (string, string)

Export exports the configuration in the PlantUML synax. onlyChans, if non-empty, restricts the output to only those rules that target the specified channels.

func (*Config) IterateDirectories

func (c *Config) IterateDirectories(fn func(d string) error) error

IterateDirectories calls the provided fn on every directory linked to by the configuration.

func (*Config) String

func (c *Config) String() string

String implements the fmt.Stringer interface.

func (*Config) Validate

func (c *Config) Validate(defaultLogDir *string) (resErr error)

Validate checks the configuration and propagates defaults.

type FileDefaults

type FileDefaults struct {
	// Dir specifies the output directory for files generated by this sink.
	Dir *string `yaml:",omitempty"`

	// MaxFileSize is the approximate maximum size of individual files
	// generated by this sink. If zero, there is no maximum size.
	MaxFileSize *ByteSize `yaml:"max-file-size,omitempty"`

	// MaxGroupSize is the approximate maximum combined size of all files
	// to be preserved for this sink. An asynchronous garbage collection
	// removes files that cause the file set to grow beyond this specified
	// size. If zero, old files are not removed.
	MaxGroupSize *ByteSize `yaml:"max-group-size,omitempty"`

	// FilePermissions is the "chmod-style" permissions the log files are
	// created with as a 3-digit octal number. The executable bit must not
	// be set. Defaults to 644 (readable by all, writable by owner).
	FilePermissions *FilePermissions `yaml:"file-permissions,omitempty"`

	// BufferedWrites specifies whether to buffer log entries.
	// Setting this to false flushes log writes upon every entry.
	BufferedWrites *bool `yaml:"buffered-writes,omitempty"`

	// CommonSinkConfig is the configuration common to all sinks. Note
	// that although the idiom in Go is to place embedded fields at the
	// beginning of a struct, we purposefully deviate from the idiom
	// here to ensure that "general" options appear after the
	// sink-specific options in YAML config dumps.
	CommonSinkConfig `yaml:",inline"`
}

FileDefaults represent configuration defaults for file sinks.

type FilePermissions

type FilePermissions uint

FilePermissions is a 9-bit number corresponding to the fs.FileMode a logfile is created with. It's written and read from the YAML as an octal string, regardless of leading zero or "0o" prefix.

func (FilePermissions) IsZero

func (x FilePermissions) IsZero() bool

IsZero implements the yaml.IsZeroer interface.

func (FilePermissions) MarshalYAML

func (x FilePermissions) MarshalYAML() (r interface{}, err error)

MarshalYAML implements the yaml.Marshaler interface.

func (*FilePermissions) UnmarshalYAML

func (x *FilePermissions) UnmarshalYAML(fn func(interface{}) error) (err error)

UnmarshalYAML implements the yaml.Unmarshaler interface.

type FileSinkConfig

type FileSinkConfig struct {
	// Channels is the list of logging channels that use this sink.
	Channels ChannelFilters `yaml:",omitempty,flow"`

	// FileDefaults contains the defaultable fields of the config.
	FileDefaults `yaml:",inline"`
	// contains filtered or unexported fields
}

FileSinkConfig represents the configuration for one file sink.

User-facing documentation follows. TITLE: Output to files

This sink type causes logging data to be captured into log files in a configurable logging directory.

The configuration key under the `sinks` key in the YAML configuration is `file-groups`. Example configuration:

sinks:
   file-groups:           # file group configurations start here
      health:             # defines one group called "health"
         channels: HEALTH

Each generated log file is prefixed by the name of the process, followed by the name of the group, separated by a hyphen. For example, the group `health` will generate files named `cockroach-health.XXX.log`, assuming the process is named `cockroach`. (A user can influence the prefix by renaming the program executable.)

The files are named so that a lexicographical sort of the directory contents presents the file in creation order.

A symlink (e.g. `cockroach-health.log`) for each group points to the latest generated log file.

Every new file group sink configured automatically inherits the configurations set in the `file-defaults` section.

For example:

file-defaults:
    redactable: false # default: disable redaction markers
    dir: logs
sinks:
  file-groups:
    health:
       channels: HEALTH
       # This sink has redactable set to false,
       # as the setting is inherited from file-defaults
       # unless overridden here.
       #
       # Example override:
       dir: health-logs # override the default 'logs'

{{site.data.alerts.callout_success}} Run `cockroach debug check-log-config` to verify the effect of defaults inheritance. {{site.data.alerts.end}}

type FluentDefaults

type FluentDefaults struct {
	CommonSinkConfig `yaml:",inline"`
}

FluentDefaults represent configuration defaults for fluent sinks.

type FluentSinkConfig

type FluentSinkConfig struct {
	// Channels is the list of logging channels that use this sink.
	Channels ChannelFilters `yaml:",omitempty,flow"`

	// Net is the protocol for the fluent server. Can be "tcp", "udp",
	// "tcp4", etc.
	Net string `yaml:",omitempty"`

	// Address is the network address of the fluent server. The
	// host/address and port parts are separated with a colon. IPv6
	// numeric addresses should be included within square brackets,
	// e.g.: [::1]:1234.
	Address string `yaml:""`

	// FluentDefaults contains the defaultable fields of the config.
	FluentDefaults `yaml:",inline"`
	// contains filtered or unexported fields
}

FluentSinkConfig represents the configuration for one fluentd sink.

User-facing documentation follows. TITLE: Output to Fluentd-compatible log collectors

This sink type causes logging data to be sent over the network to a log collector that can ingest log data in a [Fluentd](https://www.fluentd.org)-compatible protocol.

{{site.data.alerts.callout_danger}} TLS is not supported yet: the connection to the log collector is neither authenticated nor encrypted. Given that logging events may contain sensitive information, care should be taken to keep the log collector and the CockroachDB node close together on a private network, or connect them using a secure VPN. TLS support may be added at a later date. {{site.data.alerts.end}}

At the time of this writing, a Fluent sink buffers at most one log entry and retries sending the event at most one time if a network error is encountered. This is just sufficient to tolerate a restart of the Fluentd collector after a configuration change under light logging activity. If the server is unavailable for too long, or if more than one error is encountered, an error is reported to the process's standard error output with a copy of the logging event and the logging event is dropped.

The configuration key under the `sinks` key in the YAML configuration is `fluent-servers`. Example configuration:

sinks:
   fluent-servers:        # fluent configurations start here
      health:             # defines one sink called "health"
         channels: HEALTH
         address: 127.0.0.1:5170

Every new server sink configured automatically inherits the configurations set in the `fluent-defaults` section.

For example:

fluent-defaults:
    redactable: false # default: disable redaction markers
sinks:
  fluent-servers:
    health:
       channels: HEALTH
       # This sink has redactable set to false,
       # as the setting is inherited from fluent-defaults
       # unless overridden here.

The default output format for Fluent sinks is `json-fluent-compact`. The `fluent` variants of the JSON formats include a `tag` field as required by the Fluentd protocol, which the non-`fluent` JSON [format variants](log-formats.html) do not include.

{{site.data.alerts.callout_info}} Run `cockroach debug check-log-config` to verify the effect of defaults inheritance. {{site.data.alerts.end}}

type HTTPDefaults

type HTTPDefaults struct {
	// Address is the network address of the http server. The
	// host/address and port parts are separated with a colon. IPv6
	// numeric addresses should be included within square brackets,
	// e.g.: [::1]:1234.
	Address *string `yaml:",omitempty"`

	// Method is the HTTP method to be used.  POST and GET are
	// supported; defaults to POST.
	Method *HTTPSinkMethod `yaml:",omitempty"`

	// UnsafeTLS enables certificate authentication to be bypassed.
	// Defaults to false.
	UnsafeTLS *bool `yaml:"unsafe-tls,omitempty"`

	// Timeout is the HTTP timeout.
	// Defaults to 0 for no timeout.
	Timeout *time.Duration `yaml:",omitempty"`

	// DisableKeepAlives causes the logging sink to re-establish a new
	// connection for every outgoing log message. This option is
	// intended for testing only and can cause excessive network
	// overhead in production systems.
	DisableKeepAlives *bool `yaml:"disable-keep-alives,omitempty"`

	CommonSinkConfig `yaml:",inline"`
}

HTTPDefaults refresents the configuration defaults for HTTP sinks.

type HTTPSinkConfig

type HTTPSinkConfig struct {
	// Channels is the list of logging channels that use this sink.
	Channels ChannelFilters `yaml:",omitempty,flow"`

	HTTPDefaults `yaml:",inline"`
	// contains filtered or unexported fields
}

HTTPSinkConfig represents the configuration for one http sink.

User-facing documentation follows. TITLE: Output to HTTP servers.

This sink type causes logging data to be sent over the network as requests to an HTTP server.

The configuration key under the `sinks` key in the YAML configuration is `http-servers`. Example configuration:

sinks:
   http-servers:
      health:
         channels: HEALTH
         address: http://127.0.0.1

Every new server sink configured automatically inherits the configuration set in the `http-defaults` section.

For example:

http-defaults:
    redactable: false # default: disable redaction markers
sinks:
  http-servers:
    health:
       channels: HEALTH
       # This sink has redactable set to false,
       # as the setting is inherited from fluent-defaults
       # unless overridden here.

The default output format for HTTP sinks is `json-compact`. [Other supported formats.](log-formats.html)

{{site.data.alerts.callout_info}} Run `cockroach debug check-log-config` to verify the effect of defaults inheritance. {{site.data.alerts.end}}

type HTTPSinkMethod

type HTTPSinkMethod string

HTTPSinkMethod is a string restricted to "POST" and "GET"

func (*HTTPSinkMethod) Accept

func (hsm *HTTPSinkMethod) Accept(s string)

Accept implements the constrainedString interface.

func (HTTPSinkMethod) AllowedSet

func (HTTPSinkMethod) AllowedSet() []string

AllowedSet implements the constrainedString interface.

func (HTTPSinkMethod) Canonicalize

func (HTTPSinkMethod) Canonicalize(s string) string

Canonicalize implements the constrainedString interface.

func (HTTPSinkMethod) MarshalYAML

func (hsm HTTPSinkMethod) MarshalYAML() (interface{}, error)

MarshalYAML implements yaml.Marshaler interface.

func (*HTTPSinkMethod) UnmarshalYAML

func (hsm *HTTPSinkMethod) UnmarshalYAML(fn func(interface{}) error) error

UnmarshalYAML implements the yaml.Unmarshaler interface.

type Holder

type Holder struct {
	Config Config `yaml:",flow"`
}

Holder is a configuration holder to interface with pflag, which pretty-prints the configuration using the "flow" format. The "flow" format is useful for the output of --help.

func (*Holder) Set

func (h *Holder) Set(value string) error

Set implements the pflag.Value interface.

func (*Holder) String

func (h *Holder) String() string

String implements the pflag.Value interface.

func (*Holder) Type

func (*Holder) Type() string

Type implements the pflag.Value interface.

type SinkConfig

type SinkConfig struct {
	// FileGroups represents the list of configured file sinks.
	FileGroups map[string]*FileSinkConfig `yaml:"file-groups,omitempty"`
	// FluentServer represents the list of configured fluent sinks.
	FluentServers map[string]*FluentSinkConfig `yaml:"fluent-servers,omitempty"`
	// HTTPServers represents the list of configured http sinks.
	HTTPServers map[string]*HTTPSinkConfig `yaml:"http-servers,omitempty"`
	// Stderr represents the configuration for the stderr sink.
	Stderr StderrSinkConfig `yaml:",omitempty"`
}

SinkConfig represents the sink configurations.

type StderrSinkConfig

type StderrSinkConfig struct {
	// Channels is the list of logging channels that use this sink.
	Channels ChannelFilters `yaml:",omitempty,flow"`

	// NoColor forces the omission of VT color codes in the output even
	// when stderr is a terminal.
	NoColor bool `yaml:"no-color,omitempty"`

	// CommonSinkConfig is the configuration common to all sinks. Note
	// that although the idiom in Go is to place embedded fields at the
	// beginning of a struct, we purposefully deviate from the idiom
	// here to ensure that "general" options appear after the
	// sink-specific options in YAML config dumps.
	CommonSinkConfig `yaml:",inline"`
}

StderrSinkConfig represents the configuration for the stderr sink.

User-facing documentation follows. TITLE: Standard error stream

The standard error output stream of the running `cockroach` process.

The configuration key under the `sinks` key in the YAML configuration is `stderr`. Example configuration:

sinks:
   stderr:           # standard error sink configuration starts here
      channels: DEV

{{site.data.alerts.callout_info}} The server start-up messages are still emitted at the start of the standard error stream even when logging to `stderr` is enabled. This makes it generally difficult to automate integration of `stderr` with log analyzers. Generally, we recommend using [file logging](#output-to-files) or [network logging](#output-to-fluentd-compatible-log-collectors) instead of `stderr` when integrating with automated monitoring software. {{site.data.alerts.end}}

It is not possible to enable the `redactable` parameter on the `stderr` sink if `capture-stray-errors` (i.e., capturing stray error information to files) is disabled. This is because when `capture-stray-errors` is disabled, the process's standard error stream can contain an arbitrary interleaving of [logging events](eventlog.html) and stray errors. It is possible for stray error output to interfere with redaction markers and remove the guarantees that information outside of redaction markers does not contain sensitive information.

For a similar reason, no guarantee of parsability of the output format is available when `capture-stray-errors` is disabled, since the standard error stream can then contain an arbitrary interleaving of non-formatted error data.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL