pantherlog

package
v1.16.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 10, 2021 License: AGPL-3.0 Imports: 24 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	// TagNameIndicator is used for defining a field as an indicator field
	TagNameIndicator = "panther"

	// TagEventTime is used for defining a field as an event time
	//
	// Mark a struct field of type time.Time with a `event_time:"true"` tag to set the result timestamp.
	// If multiple timestamps are present in a struct the first one in the order of definition in the struct
	// will set the event timestamp.
	// This does not affect events that implement EventTimer and have already set their timestamp.
	TagNameEventTime = "event_time"
)
View Source 
const (
	// FieldPrefixJSON is the prefix for field names injected by panther to log events.
	FieldPrefixJSON      = "p_"
	FieldPrefix          = "Panther"
	FieldLogTypeJSON     = FieldPrefixJSON + "log_type"
	FieldRowIDJSON       = FieldPrefixJSON + "row_id"
	FieldEventTimeJSON   = FieldPrefixJSON + "event_time"
	FieldParseTimeJSON   = FieldPrefixJSON + "parse_time"
	FieldSourceIDJSON    = FieldPrefixJSON + "source_id"
	FieldSourceLabelJSON = FieldPrefixJSON + "source_label"
)

Variables

This section is empty.

Functions

func BuildEventSchema 

func BuildEventSchema(event interface{}, indicators ...FieldID) (interface{}, error)

BuildEventSchema builds a struct that extends the fields of `event` with all the fields added by Panther. It automatically detects indicator field ids required for `event` if no `indicators` are passed. It checks for duplicate field names in both JSON and go.

func BuildEventTypeSchema 

func BuildEventTypeSchema(eventType reflect.Type, indicators ...FieldID) (reflect.Type, error)

BuildEventTypeSchema builds a struct that extends the fields of `eventType` with all the fields added by Panther. It automatically detects indicator field ids required for `eventType` if no `indicators` are passed. It checks for duplicate field names in both JSON and go.

func ConfigJSON added in v1.13.0 

func ConfigJSON() jsoniter.API

func ExtractRawMessageIndicators added in v1.15.0 

func ExtractRawMessageIndicators(w ValueWriter, extractor func(ValueWriter, *jsoniter.Iterator, string), messages ...RawMessage)

func FieldNameJSON 

func FieldNameJSON(kind FieldID) string

FieldNameJSON returns the JSON field name of a field id.

func LookupScanner 

func LookupScanner(name string) (scanner ValueScanner, fields []FieldID)

LookupScanner finds a registered scanner and field ids by name.

func MustBuildEventSchema 

func MustBuildEventSchema(event interface{}, indicators ...FieldID) interface{}

MustBuildEventSchema builds a struct that extends the fields of `event` with all the fields added by Panther. It automatically detects indicator field ids required for `event` if no `indicators` are passed. It panics if an error occurred while building the new struct

func MustRegisterIndicator 

func MustRegisterIndicator(id FieldID, field FieldMeta)

MustRegisterIndicator allows modules to define their own indicator fields. It panics if a registration error occurs. WARNING: This function is not concurrent safe and it *must* be used during `init()`

func MustRegisterScanner 

func MustRegisterScanner(name string, scanner ValueScanner, fields ...FieldID)

MustRegisterScanner registers a value scanner to be used on string fields with a `panther` struct tag. It panics in case of a registration error.

func MustRegisterScannerFunc added in v1.8.0 

func MustRegisterScannerFunc(name string, scanner ValueScannerFunc, fields ...FieldID)

MustRegisterScannerFunc registers a value scanner to be used on string fields with a `panther` struct tag. It panics in case of a registration error.

func NewExtension 

func NewExtension() jsoniter.Extension

func RegisterIndicator 

func RegisterIndicator(id FieldID, field FieldMeta) error

RegisterIndicator allows modules to define their own indicator fields. WARNING: This function is not concurrent safe and it *must* be used during `init()` These fields are always added as `[]string` and values can be collected can by scanners using `RegisterScanner`.

func RegisterScanner 

func RegisterScanner(name string, scanner ValueScanner, fields ...FieldID) error

RegisterScanner tries to register a value scanner to be used on string fields with a `panther` struct tag. Scanner names should be unique and field ids should already be registered with `RegisterField`. Argument `name` defines the name to use for this scanner (ie "foo" will be used for tags with `panther:"foo"). Argument `scanner` is the actual scanner being registered. Argument `fields` defines all the possible field ids this scanner can produce values for.

func RegisteredFieldNamesJSON 

func RegisteredFieldNamesJSON() (names []string)

RegisteredFieldNamesJSON returns the JSON field names for registered indicator fields

func ScanARN added in v1.13.0 

func ScanARN(w ValueWriter, input string)

func ScanAWSAccountID added in v1.13.0 

func ScanAWSAccountID(w ValueWriter, input string)

func ScanAWSInstanceID added in v1.13.0 

func ScanAWSInstanceID(w ValueWriter, input string)

func ScanAWSTag added in v1.13.0 

func ScanAWSTag(w ValueWriter, input string)

func ScanDomainName added in v1.15.0 

func ScanDomainName(w ValueWriter, input string)

ScanDomainName scans `input` domain name value, if internationalized then decodes

func ScanEmail added in v1.14.0 

func ScanEmail(w ValueWriter, input string)

func ScanHostname 

func ScanHostname(w ValueWriter, input string)

ScanHostname scans `input` for either an ip address or a domain name value.

func ScanIPAddress 

func ScanIPAddress(w ValueWriter, input string)

ScanIPAddress scans `input` for an ip address value.

func ScanMD5Hash added in v1.15.0 

func ScanMD5Hash(w ValueWriter, input string)

func ScanNetworkAddress 

func ScanNetworkAddress(w ValueWriter, input string)

Tries to split host:port address or falls back to Hostname scanning if `:` is not present in input

func ScanSHA1Hash added in v1.15.0 

func ScanSHA1Hash(w ValueWriter, input string)

func ScanSHA256Hash added in v1.15.0 

func ScanSHA256Hash(w ValueWriter, input string)

func ScanURL 

func ScanURL(dest ValueWriter, input string)

ScanURL scans a URL string for domain or ip address

func StaticNow 

func StaticNow(now time.Time) func() time.Time

StaticNow returns a function to be used as ResultBuilder.Now to always set the ParseTime to a specific time

func StaticRowID 

func StaticRowID(id string) func() string

StaticRowID returns a function to be used as ResultBuilder.NextRowID to always set the RowID to a specific value

func ValidateStruct added in v1.13.0 

func ValidateStruct(x interface{}) error

Types

type Bool added in v1.8.0 

type Bool = null.Bool

type CoreFields 

type CoreFields struct {
	PantherEventTime   time.Time `json:"p_event_time" validate:"required" description:"Panther added standardized event time (UTC)"`
	PantherParseTime   time.Time `json:"p_parse_time" validate:"required" description:"Panther added standardized log parse time (UTC)"`
	PantherLogType     string    `json:"p_log_type" validate:"required" description:"Panther added field with type of log"`
	PantherRowID       string    `json:"p_row_id" validate:"required" description:"Panther added field with unique id (within table)"`
	PantherSourceID    string    `json:"p_source_id,omitempty" description:"Panther added field with the source id"`
	PantherSourceLabel string    `json:"p_source_label,omitempty" description:"Panther added field with the source label"`
}

CoreFields are the 'core' fields Panther adds to each log. External modules cannot add core fields.

type EventTimer 

type EventTimer interface {
	PantherEventTime() time.Time
}

EventTimer returns the event timestamp. ResultBuilder checks for events that implement this interface and uses the appropriate timestamp as the event time. Events that require custom logic to decide their timestamp should implement this interface.

type FactoryFunc added in v1.15.0 

type FactoryFunc func(params interface{}) (LogParser, error)

func (FactoryFunc) NewParser added in v1.15.0 

func (f FactoryFunc) NewParser(params interface{}) (LogParser, error)

type FieldID 

type FieldID int

FieldID is the id of a field added by Panther. This includes both core fields that are common to all events and indicator fields that are added on a per-logtype basis. 

const (
	FieldNone FieldID = 0 - iota
	CoreFieldEventTime
	CoreFieldParseTime
	CoreFieldLogType
	CoreFieldRowID
	CoreFieldSourceID
	CoreFieldSourceLabel
)

Core field ids (<=0) All core fields ids are negative integers to distinguish them. 

const (
	FieldIPAddress FieldID = 1 + iota
	FieldDomainName
	FieldMD5Hash
	FieldSHA1Hash
	FieldSHA256Hash
	FieldTraceID
	FieldAWSAccountID
	FieldAWSInstanceID
	FieldAWSARN
	FieldAWSTag
	FieldEmail
	FieldUsername
)

Indicator fields (>0) These fields collect string values from the log event. Each log type can choose the indicator fields it requires. Modules can register new indicator fields at init() using RegisterIndicator

func (FieldID) IsCore 

func (id FieldID) IsCore() bool

IsCore checks if a field id is core

func (FieldID) ScanValues 

func (id FieldID) ScanValues(w ValueWriter, input string)

ScanValues implements ValueScanner interface

type FieldMeta 

type FieldMeta struct {
	Name        string
	NameJSON    string
	Description string
}

FieldMeta describes a panther field.

func (*FieldMeta) StructField 

func (m *FieldMeta) StructField() reflect.StructField

type FieldSet 

type FieldSet []FieldID

FieldSet is a set of field ids. It provides helper methods to sort, filter and extend a set of uniquee fields ids.

func DefaultIndicators 

func DefaultIndicators() FieldSet

DefaultIndicators returns the default panther indicator fields. It creates a new copy so that outside packages cannot affect the defaults.

func FieldSetFromJSON 

func FieldSetFromJSON(input []byte) (fields FieldSet)

FieldSetFromJSON checks top-level field names in a JSON object and produces the field set of all panther fields.

func FieldSetFromTag 

func FieldSetFromTag(tag string) FieldSet

FieldSetFromTag produces the minimum required field set to support scanners defined in a struct tag.

func FieldSetFromType 

func FieldSetFromType(typ reflect.Type) (fields FieldSet)

FieldSetFromType produces the minimum required field set to support scanners and core fields defined in a struct.

func NewFieldSet 

func NewFieldSet(ids ...FieldID) (fields FieldSet)

NewFieldSet creates a new set of distinct field ids

func (FieldSet) Add 

func (fields FieldSet) Add(id FieldID) FieldSet

Add appends a field id to the set if it is not already there.

func (FieldSet) Extend 

func (fields FieldSet) Extend(ids ...FieldID) FieldSet

Extend extends the set to include ids.

func (FieldSet) Indicators 

func (fields FieldSet) Indicators() (indicators FieldSet)

Indicators returns a copy of the set containing only indicator field ids

func (FieldSet) Len 

func (fields FieldSet) Len() int

Len implements sort.Interface

func (FieldSet) Less 

func (fields FieldSet) Less(i, j int) bool

Less implements sort.Interface

func (FieldSet) Swap 

func (fields FieldSet) Swap(i, j int)

Swap implements sort.Interface

type Float32 added in v1.8.0 

type Float32 = null.Float32

type Float64 added in v1.8.0 

type Float64 = null.Float64

type Int16 added in v1.8.0 

type Int16 = null.Int16

type Int32 added in v1.8.0 

type Int32 = null.Int32

type Int64 added in v1.8.0 

type Int64 = null.Int64

type Int8 added in v1.8.0 

type Int8 = null.Int8

type JSONParserFactory added in v1.15.0 

type JSONParserFactory struct {
	LogType        string
	NewEvent       func() interface{}
	JSON           jsoniter.API
	Validate       func(event interface{}) error
	ReadBufferSize int
	NextRowID      func() string
	Now            func() time.Time
}

func (*JSONParserFactory) NewParser added in v1.15.0 

func (f *JSONParserFactory) NewParser(_ interface{}) (LogParser, error)

type LogParser added in v1.15.0 

type LogParser interface {
	ParseLog(log string) ([]*Result, error)
}

type LogParserFactory added in v1.15.0 

type LogParserFactory interface {
	NewParser(_ interface{}) (LogParser, error)
}

type ParserResolver added in v1.16.0 

type ParserResolver interface {
	ResolveParser(ctx context.Context, name string) (LogParser, error)
}

type RawMessage added in v1.8.0 

type RawMessage = jsoniter.RawMessage

type Result 

type Result struct {
	// Result extends all core panther fields
	CoreFields
	// The underlying event
	Event interface{}
	// Used for log events that embed parsers.PantherLog. This is a low-overhead, temporary work-around
	// to avoid duplicate panther fields in resulting JSON.
	// FIXME: Remove this field once all parsers are ported to the new method.
	EventIncludesPantherFields bool
	// contains filtered or unexported fields
}

Result is the result of parsing a log event.

func (*Result) MarshalJSON 

func (r *Result) MarshalJSON() ([]byte, error)

func (*Result) UnmarshalJSON 

func (r *Result) UnmarshalJSON(_ []byte) error

func (*Result) WriteValues 

func (r *Result) WriteValues(kind FieldID, values ...string)

WriteValues implements ValueWriter interface

type ResultBuilder 

type ResultBuilder struct {
	// Override this to have static row ids for tests
	NextRowID func() string
	// Override this to have static parse time for tests
	Now func() time.Time
}

ResultBuilder builds new results filling out result fields.

func (*ResultBuilder) BuildResult 

func (b *ResultBuilder) BuildResult(logType string, event interface{}) (*Result, error)

BuildResult builds a new result for an event. Log type is passed as an argument so that a single result builder can be reused for producing results of different log types.

type String added in v1.8.0 

type String = null.String

Re-export field types from the pantherlog package so event types only need to import a single package. This makes explaining the process of adding support for a new log type much easier. It also allows us to change implementations of a field type in the future without modifying parser code

type Time added in v1.8.0 

type Time = time.Time

type Uint16 added in v1.8.0 

type Uint16 = null.Uint16

type Uint32 added in v1.8.0 

type Uint32 = null.Uint32

type Uint64 added in v1.8.0 

type Uint64 = null.Uint64

type Uint8 added in v1.8.0 

type Uint8 = null.Uint8

type ValueBuffer 

type ValueBuffer struct {
	// contains filtered or unexported fields
}

ValueBuffer is a reusable buffer of field values. It provides helper methods to collect fields from log entries. A ValueBuffer can be reset and used in a pool.

func BlankValueBuffer 

func BlankValueBuffer() *ValueBuffer

func (*ValueBuffer) Contains 

func (b *ValueBuffer) Contains(id FieldID, value string) bool

Contains checks if a field buffer contains a specific field.

func (*ValueBuffer) Fields 

func (b *ValueBuffer) Fields() []FieldID

Fields returns the field ids that contain values in this buffer.

func (*ValueBuffer) Get 

func (b *ValueBuffer) Get(id FieldID) []string

Get returns the values stored for a field id (sorted)

func (*ValueBuffer) Inspect 

func (b *ValueBuffer) Inspect() map[FieldID][]string

Inspect returns a sorted copy snapshot of the value index This is mainly useful for tests.

func (*ValueBuffer) IsEmpty 

func (b *ValueBuffer) IsEmpty() bool

func (*ValueBuffer) Recycle 

func (b *ValueBuffer) Recycle()

func (*ValueBuffer) Reset 

func (b *ValueBuffer) Reset()

Reset clears all fields from a buffer retaining allocated memory.

func (*ValueBuffer) WriteValues 

func (b *ValueBuffer) WriteValues(id FieldID, values ...string)

WriteValues adds values to the buffer.

func (*ValueBuffer) WriteValuesTo 

func (b *ValueBuffer) WriteValuesTo(w ValueWriter)

type ValueScanner 

type ValueScanner interface {
	// ScanValues scans `input` and writes values to `w`
	ScanValues(w ValueWriter, input string)
}

ValueScanner parses values from a string and writes them to a ValueWriter. Implementations should parse `input` and write valid values to `w`. If errors occur while parsing `input` no values should be written to `w`.

func MultiScanner added in v1.11.0 

func MultiScanner(scanners ...ValueScanner) ValueScanner

MultiScanner scans a value with multiple scanners

type ValueScannerFunc 

type ValueScannerFunc func(dest ValueWriter, value string)

ValueScannerFunc is a function implementing ValueScanner interface

func (ValueScannerFunc) ScanValues 

func (f ValueScannerFunc) ScanValues(dest ValueWriter, value string)

ScanValues implements ValueScanner interface

type ValueWriter 

type ValueWriter interface {
	WriteValues(field FieldID, values ...string)
}

ValueWriter provides the interface to write field values

type ValueWriterTo 

type ValueWriterTo interface {
	WriteValuesTo(w ValueWriter)
}

ValueWriterTo can write field values to a ValueWriter

Source Files

View all Source files

Directories

Path Synopsis
nolint: dupl nolint: dupl nolint: dupl nolint: dupl nolint: dupl nolint: dupl nolint: dupl Package null provides performant nullable values for JSON serialization/deserialization nolint: dupl nolint: dupl nolint: dupl nolint: dupl
 nolint: dupl nolint: dupl nolint: dupl nolint: dupl nolint: dupl nolint: dupl nolint: dupl Package null provides performant nullable values for JSON serialization/deserialization nolint: dupl nolint: dupl nolint: dupl nolint: dupl

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.