helper

package
v1.1.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 24, 2021 License: Apache-2.0 Imports: 24 Imported by: 24

Documentation

Index

Constants

View Source 
const (
	// HTTP2xx is a special key that is represents a range from 200 to 299. Literal value is "2xx"
	HTTP2xx = "2xx"

	// HTTP3xx is a special key that is represents a range from 300 to 399. Literal value is "3xx"
	HTTP3xx = "3xx"

	// HTTP4xx is a special key that is represents a range from 400 to 499. Literal value is "4xx"
	HTTP4xx = "4xx"

	// HTTP5xx is a special key that is represents a range from 500 to 599. Literal value is "5xx"
	HTTP5xx = "5xx"
)
View Source 
const DropOnError = "drop"

DropOnError specifies an on_error mode for dropping entries after an error.

View Source 
const EpochKey = "epoch"

EpochKey is literally "epoch" and can parse seconds and/or subseconds

View Source 
const GotimeKey = "gotime"

GotimeKey is literally "gotime" and uses Golang's native time.Parse

View Source 
const NativeKey = "native" // provided for operator development

NativeKey is literally "native" and refers to Golang's native time.Time

View Source 
const SendOnError = "send"

SendOnError specifies an on_error mode for sending entries after an error.

View Source 
const StrptimeKey = "strptime"

StrptimeKey is literally "strptime", and is the default layout type

Variables

View Source 
var OffsetsBucket = []byte(`offsets`)

OffsetsBucket is the scope provided to offset persistence

Functions

func GetExprEnv 

func GetExprEnv(e *entry.Entry) map[string]interface{}

GetExprEnv returns a map of key/value pairs that can be be used to evaluate an expression

func NewLineEndSplitFunc added in v1.0.1 

func NewLineEndSplitFunc(re *regexp.Regexp, flushAtEOF bool) bufio.SplitFunc

NewLineEndSplitFunc creates a bufio.SplitFunc that splits an incoming stream into tokens that end with a match to the regex pattern provided

func NewLineStartSplitFunc added in v1.0.1 

func NewLineStartSplitFunc(re *regexp.Regexp, flushAtEOF bool) bufio.SplitFunc

NewLineStartSplitFunc creates a bufio.SplitFunc that splits an incoming stream into tokens that start with a match to the regex pattern provided

func NewNewlineSplitFunc added in v1.0.1 

func NewNewlineSplitFunc(encoding encoding.Encoding, flushAtEOF bool) (bufio.SplitFunc, error)

NewNewlineSplitFunc splits log lines by newline, just as bufio.ScanLines, but never returning an token using EOF as a terminator

func PutExprEnv 

func PutExprEnv(e map[string]interface{})

PutExprEnv adds a key/value pair that will can be used to evaluate an expression

Types

type BasicConfig 

type BasicConfig struct {
	OperatorID   string `json:"id"   yaml:"id"`
	OperatorType string `json:"type" yaml:"type"`
}

BasicConfig provides a basic implemention for an operator config.

func NewBasicConfig 

func NewBasicConfig(operatorID, operatorType string) BasicConfig

NewBasicConfig creates a new basic config

func (BasicConfig) Build 

func (c BasicConfig) Build(context operator.BuildContext) (BasicOperator, error)

Build will build a basic operator.

func (BasicConfig) ID 

func (c BasicConfig) ID() string

ID will return the operator id.

func (BasicConfig) Type 

func (c BasicConfig) Type() string

Type will return the operator type.

type BasicOperator 

type BasicOperator struct {
	OperatorID   string
	OperatorType string
	*zap.SugaredLogger
}

BasicOperator provides a basic implementation of an operator.

func (*BasicOperator) ID 

func (p *BasicOperator) ID() string

ID will return the operator id.

func (*BasicOperator) Logger 

func (p *BasicOperator) Logger() *zap.SugaredLogger

Logger returns the operator's scoped logger.

func (*BasicOperator) Start 

func (p *BasicOperator) Start() error

Start will start the operator.

func (*BasicOperator) Stop 

func (p *BasicOperator) Stop() error

Stop will stop the operator.

func (*BasicOperator) Type 

func (p *BasicOperator) Type() string

Type will return the operator type.

type ByteSize added in v0.13.3 

type ByteSize int64

func (*ByteSize) UnmarshalJSON added in v0.13.3 

func (h *ByteSize) UnmarshalJSON(raw []byte) error

func (*ByteSize) UnmarshalYAML added in v0.13.3 

func (h *ByteSize) UnmarshalYAML(unmarshal func(interface{}) error) error

type Duration added in v0.10.0 

type Duration struct {
	time.Duration
}

Duration is the representation of a length of time

func NewDuration added in v0.10.0 

func NewDuration(t time.Duration) Duration

NewDuration creates a new duration from a time

func (Duration) MarshalJSON added in v0.10.0 

func (d Duration) MarshalJSON() ([]byte, error)

MarshalJSON will marshal the duration as a json string

func (Duration) MarshalYAML added in v0.10.0 

func (d Duration) MarshalYAML() (interface{}, error)

MarshalYAML will marshal the duration as a yaml string

func (*Duration) Raw added in v0.10.0 

func (d *Duration) Raw() time.Duration

Raw will return the raw duration, without modification

func (*Duration) UnmarshalJSON added in v0.10.0 

func (d *Duration) UnmarshalJSON(raw []byte) error

UnmarshalJSON will unmarshal json as a duration

func (*Duration) UnmarshalYAML added in v0.10.0 

func (d *Duration) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML will unmarshal yaml as a duration

type Encoding added in v1.0.1 

type Encoding struct {
	Encoding encoding.Encoding
}

func (*Encoding) Decode added in v1.0.1 

func (e *Encoding) Decode(msgBuf []byte) (string, error)

decode converts the bytes in msgBuf to utf-8 from the configured encoding

type EncodingConfig added in v1.0.1 

type EncodingConfig struct {
	Encoding string `mapstructure:"encoding,omitempty"              json:"encoding,omitempty"             yaml:"encoding,omitempty"`
}

EncodingConfig is the configuration of a Encoding helper

func NewEncodingConfig added in v1.0.1 

func NewEncodingConfig() EncodingConfig

NewBasicConfig creates a new Encoding config

func (EncodingConfig) Build added in v1.0.1 

func (c EncodingConfig) Build(context operator.BuildContext) (Encoding, error)

Build will build an Encoding operator.

type ExprString 

type ExprString struct {
	SubStrings []string
	SubExprs   []*vm.Program
}

An ExprString is made up of a list of string literals interleaved with expressions. len(SubStrings) == len(SubExprs) + 1

func (*ExprString) Render 

func (e *ExprString) Render(env map[string]interface{}) (string, error)

Render will render an ExprString as a string

type ExprStringConfig 

type ExprStringConfig string

ExprStringConfig is a string that represents an expression

func (ExprStringConfig) Build 

func (e ExprStringConfig) Build() (*ExprString, error)

Build creates an ExprStr string from the specified config

type HostIdentifier 

type HostIdentifier struct {
	// contains filtered or unexported fields
}

HostIdentifier is a helper that adds host related metadata to an entry's resource

func (*HostIdentifier) Identify 

func (h *HostIdentifier) Identify(entry *entry.Entry)

Identify will add host related metadata to an entry's resource

type HostIdentifierConfig 

type HostIdentifierConfig struct {
	IncludeHostname bool `json:"include_hostname,omitempty"     yaml:"include_hostname,omitempty"`
	IncludeIP       bool `json:"include_ip,omitempty"     yaml:"include_ip,omitempty"`
	// contains filtered or unexported fields
}

HostIdentifierConfig is the configuration of a host identifier

func NewHostIdentifierConfig 

func NewHostIdentifierConfig() HostIdentifierConfig

NewHostIdentifierConfig returns a HostIdentifierConfig with default values

func (HostIdentifierConfig) Build 

func (c HostIdentifierConfig) Build() (HostIdentifier, error)

Build will build a host labeler from the supplied configuration

type Identifier 

type Identifier struct {
	// contains filtered or unexported fields
}

Identifier is a helper that adds values to the resource of an entry

func (*Identifier) Identify 

func (i *Identifier) Identify(e *entry.Entry) error

Identify will add values to the resource of an entry

type IdentifierConfig 

type IdentifierConfig struct {
	Resource map[string]ExprStringConfig `json:"resource" yaml:"resource"`
}

IdentifierConfig is the configuration of a resource identifier

func NewIdentifierConfig 

func NewIdentifierConfig() IdentifierConfig

NewIdentifierConfig creates a new identifier config with default values

func (IdentifierConfig) Build 

func (c IdentifierConfig) Build() (Identifier, error)

Build will build an identifier from the supplied configuration

type InputConfig 

type InputConfig struct {
	LabelerConfig    `yaml:",inline"`
	IdentifierConfig `yaml:",inline"`
	WriterConfig     `yaml:",inline"`
	WriteTo          entry.Field `json:"write_to" yaml:"write_to"`
}

InputConfig provides a basic implementation of an input operator config.

func NewInputConfig 

func NewInputConfig(operatorID, operatorType string) InputConfig

NewInputConfig creates a new input config with default values.

func (InputConfig) Build 

func (c InputConfig) Build(context operator.BuildContext) (InputOperator, error)

Build will build a base producer.

type InputOperator 

type InputOperator struct {
	Labeler
	Identifier
	WriterOperator
	WriteTo entry.Field
}

InputOperator provides a basic implementation of an input operator.

func (*InputOperator) CanProcess 

func (i *InputOperator) CanProcess() bool

CanProcess will always return false for an input operator.

func (*InputOperator) NewEntry 

func (i *InputOperator) NewEntry(value interface{}) (*entry.Entry, error)

NewEntry will create a new entry using the `write_to`, `labels`, and `resource` configuration.

func (*InputOperator) Process 

func (i *InputOperator) Process(ctx context.Context, entry *entry.Entry) error

Process will always return an error if called.

type Labeler 

type Labeler struct {
	// contains filtered or unexported fields
}

Labeler is a helper that adds labels to an entry

func (*Labeler) Label 

func (l *Labeler) Label(e *entry.Entry) error

Label will add labels to an entry

type LabelerConfig 

type LabelerConfig struct {
	Labels map[string]ExprStringConfig `json:"labels" yaml:"labels"`
}

LabelerConfig is the configuration of a labeler

func NewLabelerConfig 

func NewLabelerConfig() LabelerConfig

NewLabelerConfig creates a new labeler config with default values

func (LabelerConfig) Build 

func (c LabelerConfig) Build() (Labeler, error)

Build will build a labeler from the supplied configuration

type MultilineConfig added in v1.0.1 

type MultilineConfig struct {
	LineStartPattern string `mapstructure:"line_start_pattern"  json:"line_start_pattern" yaml:"line_start_pattern"`
	LineEndPattern   string `mapstructure:"line_end_pattern"    json:"line_end_pattern"   yaml:"line_end_pattern"`
}

MultilineConfig is the configuration of a multiline helper

func NewMultilineConfig added in v1.0.1 

func NewMultilineConfig() MultilineConfig

NewBasicConfig creates a new Multiline config

func (MultilineConfig) Build added in v1.0.1 

func (c MultilineConfig) Build(context operator.BuildContext, encoding encoding.Encoding, flushAtEOF bool) (bufio.SplitFunc, error)

Build will build a Multiline operator.

type OutputConfig 

type OutputConfig struct {
	BasicConfig `mapstructure:",squash" yaml:",inline"`
}

OutputConfig provides a basic implementation of an output operator config.

func NewOutputConfig 

func NewOutputConfig(operatorID, operatorType string) OutputConfig

NewOutputConfig creates a new output config

func (OutputConfig) Build 

func (c OutputConfig) Build(context operator.BuildContext) (OutputOperator, error)

Build will build an output operator.

type OutputIDs 

type OutputIDs []string

OutputIDs is a collection of operator IDs used as outputs.

func NewOutputIDsFromArray added in v0.12.2 

func NewOutputIDsFromArray(array []interface{}) (OutputIDs, error)

NewOutputIDsFromArray creates a new OutputIDs object from an array

func NewOutputIDsFromInterface added in v0.12.2 

func NewOutputIDsFromInterface(value interface{}) (OutputIDs, error)

NewOutputIDsFromInterface creates a new OutputIDs object from an interface

func (*OutputIDs) UnmarshalJSON 

func (o *OutputIDs) UnmarshalJSON(bytes []byte) error

UnmarshalJSON will unmarshal a string or array of strings to OutputIDs.

func (*OutputIDs) UnmarshalYAML 

func (o *OutputIDs) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML will unmarshal a string or array of strings to OutputIDs.

func (OutputIDs) WithNamespace added in v0.12.4 

func (o OutputIDs) WithNamespace(bc operator.BuildContext) OutputIDs

type OutputOperator 

type OutputOperator struct {
	BasicOperator
}

OutputOperator provides a basic implementation of an output operator.

func (*OutputOperator) CanOutput 

func (o *OutputOperator) CanOutput() bool

CanOutput will always return false for an output operator.

func (*OutputOperator) CanProcess 

func (o *OutputOperator) CanProcess() bool

CanProcess will always return true for an output operator.

func (*OutputOperator) Outputs 

func (o *OutputOperator) Outputs() []operator.Operator

Outputs will always return an empty array for an output operator.

func (*OutputOperator) SetOutputs 

func (o *OutputOperator) SetOutputs(operators []operator.Operator) error

SetOutputs will return an error if called.

type ParseFunction 

type ParseFunction = func(interface{}) (interface{}, error)

ParseFunction is function that parses a raw value.

type ParserConfig 

type ParserConfig struct {
	TransformerConfig `yaml:",inline"`

	ParseFrom            entry.Field           `json:"parse_from"          yaml:"parse_from"`
	ParseTo              entry.Field           `json:"parse_to"            yaml:"parse_to"`
	PreserveTo           *entry.Field          `json:"preserve_to"         yaml:"preserve_to"`
	TimeParser           *TimeParser           `json:"timestamp,omitempty" yaml:"timestamp,omitempty"`
	SeverityParserConfig *SeverityParserConfig `json:"severity,omitempty"  yaml:"severity,omitempty"`
}

ParserConfig provides the basic implementation of a parser config.

func NewParserConfig 

func NewParserConfig(operatorID, operatorType string) ParserConfig

NewParserConfig creates a new parser config with default values

func (ParserConfig) Build 

func (c ParserConfig) Build(context operator.BuildContext) (ParserOperator, error)

Build will build a parser operator.

type ParserOperator 

type ParserOperator struct {
	TransformerOperator
	ParseFrom      entry.Field
	ParseTo        entry.Field
	PreserveTo     *entry.Field
	TimeParser     *TimeParser
	SeverityParser *SeverityParser
}

ParserOperator provides a basic implementation of a parser operator.

func (*ParserOperator) ParseWith added in v0.13.3 

func (p *ParserOperator) ParseWith(ctx context.Context, entry *entry.Entry, parse ParseFunction) error

ParseWith will process an entry's field with a parser function.

func (*ParserOperator) ProcessWith 

func (p *ParserOperator) ProcessWith(ctx context.Context, entry *entry.Entry, parse ParseFunction) error

ProcessWith will run ParseWith on the entry, then forward the entry on to the next operators.

func (*ParserOperator) ProcessWithCallback added in v0.14.0 

func (p *ParserOperator) ProcessWithCallback(ctx context.Context, entry *entry.Entry, parse ParseFunction, cb func(*entry.Entry) error) error

type Persister 

type Persister interface {
	Get(string) []byte
	Set(string, []byte)
	Sync() error
	Load() error
}

Persister is a helper used to persist data

type ScopedBBoltPersister 

type ScopedBBoltPersister struct {
	// contains filtered or unexported fields
}

ScopedBBoltPersister is a persister that uses a database for the backend

func NewScopedDBPersister 

func NewScopedDBPersister(db database.Database, scope string) *ScopedBBoltPersister

NewScopedDBPersister returns a new ScopedBBoltPersister

func (*ScopedBBoltPersister) Get 

func (p *ScopedBBoltPersister) Get(key string) []byte

Get retrieves a key from the cache

func (*ScopedBBoltPersister) Load 

func (p *ScopedBBoltPersister) Load() error

Load populates the cache with the values from the database, overwriting anything currently in the cache.

func (*ScopedBBoltPersister) Set 

func (p *ScopedBBoltPersister) Set(key string, val []byte)

Set saves a key in the cache

func (*ScopedBBoltPersister) Sync 

func (p *ScopedBBoltPersister) Sync() error

Sync saves the cache to the backend, ensuring values are safely written to disk before returning

type SeverityParser 

type SeverityParser struct {
	ParseFrom  entry.Field
	PreserveTo *entry.Field
	Mapping    severityMap
}

SeverityParser is a helper that parses severity onto an entry.

func (*SeverityParser) Parse 

func (p *SeverityParser) Parse(ent *entry.Entry) error

Parse will parse severity from a field and attach it to the entry

type SeverityParserConfig 

type SeverityParserConfig struct {
	ParseFrom  *entry.Field                `json:"parse_from,omitempty"  yaml:"parse_from,omitempty"`
	PreserveTo *entry.Field                `json:"preserve_to,omitempty" yaml:"preserve_to,omitempty"`
	Preset     string                      `json:"preset,omitempty"      yaml:"preset,omitempty"`
	Mapping    map[interface{}]interface{} `json:"mapping,omitempty"     yaml:"mapping,omitempty"`
}

SeverityParserConfig allows users to specify how to parse a severity from a field.

func NewSeverityParserConfig 

func NewSeverityParserConfig() SeverityParserConfig

NewSeverityParserConfig creates a new severity parser config

func (*SeverityParserConfig) Build 

func (c *SeverityParserConfig) Build(context operator.BuildContext) (SeverityParser, error)

Build builds a SeverityParser from a SeverityParserConfig

type TimeParser 

type TimeParser struct {
	ParseFrom  *entry.Field `json:"parse_from,omitempty"  yaml:"parse_from,omitempty"`
	Layout     string       `json:"layout,omitempty"      yaml:"layout,omitempty"`
	LayoutType string       `json:"layout_type,omitempty" yaml:"layout_type,omitempty"`
	PreserveTo *entry.Field `json:"preserve_to,omitempty" yaml:"preserve_to,omitempty"`
	Location   string       `json:"location,omitempty"   yaml:"location,omitempty"`
	// contains filtered or unexported fields
}

TimeParser is a helper that parses time onto an entry.

func NewTimeParser 

func NewTimeParser() TimeParser

NewTimeParser creates a new time parser with default values

func (*TimeParser) IsZero 

func (t *TimeParser) IsZero() bool

IsZero returns true if the TimeParser is not a valid config

func (*TimeParser) Parse 

func (t *TimeParser) Parse(entry *entry.Entry) error

Parse will parse time from a field and attach it to the entry

func (*TimeParser) Validate 

func (t *TimeParser) Validate(context operator.BuildContext) error

Validate validates a TimeParser, and reconfigures it if necessary

type TransformFunction 

type TransformFunction = func(*entry.Entry) error

TransformFunction is function that transforms an entry.

type TransformerConfig 

type TransformerConfig struct {
	WriterConfig `yaml:",inline"`
	OnError      string `json:"on_error" yaml:"on_error"`
	IfExpr       string `json:"if"                  yaml:"if"`
}

TransformerConfig provides a basic implementation of a transformer config.

func NewTransformerConfig 

func NewTransformerConfig(operatorID, operatorType string) TransformerConfig

NewTransformerConfig creates a new transformer config with default values

func (TransformerConfig) Build 

func (c TransformerConfig) Build(context operator.BuildContext) (TransformerOperator, error)

Build will build a transformer operator.

type TransformerOperator 

type TransformerOperator struct {
	WriterOperator
	OnError string
	IfExpr  *vm.Program
}

TransformerOperator provides a basic implementation of a transformer operator.

func (*TransformerOperator) CanProcess 

func (t *TransformerOperator) CanProcess() bool

CanProcess will always return true for a transformer operator.

func (*TransformerOperator) HandleEntryError 

func (t *TransformerOperator) HandleEntryError(ctx context.Context, entry *entry.Entry, err error) error

HandleEntryError will handle an entry error using the on_error strategy.

func (*TransformerOperator) ProcessWith 

func (t *TransformerOperator) ProcessWith(ctx context.Context, entry *entry.Entry, transform TransformFunction) error

ProcessWith will process an entry with a transform function.

func (*TransformerOperator) Skip added in v0.13.2 

func (t *TransformerOperator) Skip(ctx context.Context, entry *entry.Entry) (bool, error)

type WriterConfig 

type WriterConfig struct {
	BasicConfig `yaml:",inline"`
	OutputIDs   OutputIDs `json:"output" yaml:"output"`
}

WriterConfig is the configuration of a writer operator.

func NewWriterConfig 

func NewWriterConfig(operatorID, operatorType string) WriterConfig

NewWriterConfig creates a new writer config

func (WriterConfig) Build 

func (c WriterConfig) Build(bc operator.BuildContext) (WriterOperator, error)

Build will build a writer operator from the config.

type WriterOperator 

type WriterOperator struct {
	BasicOperator
	OutputIDs       OutputIDs
	OutputOperators []operator.Operator
}

WriterOperator is an operator that can write to other operators.

func (*WriterOperator) CanOutput 

func (w *WriterOperator) CanOutput() bool

CanOutput always returns true for a writer operator.

func (*WriterOperator) Outputs 

func (w *WriterOperator) Outputs() []operator.Operator

Outputs returns the outputs of the writer operator.

func (*WriterOperator) SetOutputs 

func (w *WriterOperator) SetOutputs(operators []operator.Operator) error

SetOutputs will set the outputs of the operator.

func (*WriterOperator) Write 

func (w *WriterOperator) Write(ctx context.Context, e *entry.Entry)

Write will write an entry to the outputs of the operator.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.