parsers

package
v1.5.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 29, 2020 License: AGPL-3.0 Imports: 16 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	PantherFieldPrefix = "p_"
)

Variables

View Source 
var JSON = func() jsoniter.API {
	config := jsoniter.Config{
		EscapeHTML: true,

		ValidateJsonRawMessage: true,

		SortMapKeys: false,
	}
	api := config.Froze()
	rewriteFields := jsonutil.NewEncoderNamingStrategy(RewriteFieldName)
	api.RegisterExtension(rewriteFields)
	return api
}()

JSON is a custom jsoniter config to properly remap field names for compatibility with Athena views

View Source 
var Validator = validator.New()

Validator can be used to validate schemas of log fields

Functions

func AppendAnyString added in v0.2.0 

func AppendAnyString(any *PantherAnyString, values ...string)

func CsvStringToArray added in v0.2.0 

func CsvStringToArray(value string) []string

func CsvStringToFloat64Pointer added in v0.2.0 

func CsvStringToFloat64Pointer(value string) *float64

func CsvStringToInt16Pointer added in v0.2.0 

func CsvStringToInt16Pointer(value string) *int16

func CsvStringToIntPointer added in v0.2.0 

func CsvStringToIntPointer(value string) *int

func CsvStringToPointer added in v0.2.0 

func CsvStringToPointer(value string) *string

func LooksLikeCSV added in v1.5.0 

func LooksLikeCSV(log string) bool

Returns true if log looks like a CSV log. It can be used to fail fast for logs that are not CSV

func RewriteFieldName added in v1.3.0 

func RewriteFieldName(name string) string

Types

type Factory added in v1.5.1 

type Factory func(params interface{}) (Interface, error)

Factory creates new parser instances. The params argument defines parameters for a parser.

func AdapterFactory added in v1.5.1 

func AdapterFactory(parser LogParser) Factory

AdapterFactory returns a pantherlog.LogParser factory from a parsers.Parser This is used to ease transition to the new pantherlog.EventTypeEntry registry.

type Interface added in v1.5.1 

type Interface interface {
	ParseLog(log string) ([]*Result, error)
}

Interface is the interface to be used for log parsers.

func NewAdapter added in v1.5.1 

func NewAdapter(parser LogParser) Interface

NewAdapter creates a pantherlog.LogParser from a parsers.Parser

type LogParser 

type LogParser interface {
	// LogType returns the log type supported by this parser
	LogType() string

	// Parse attempts to parse the provided log line
	// If the provided log is not of the supported type the method returns nil and an error
	Parse(log string) ([]*PantherLog, error)

	// New returns a new instance of the log parser, used like a factory method for stateful parsers
	New() LogParser
}

LogParser represents a parser for a supported log type NOTE: We will be transitioning parsers to the `pantherlog.LogParser` interface. Until all parsers are converted to the new interface the `AdapterFactory()` helper should be used when registering a `logtypes.Entry` that uses this interface.

type PantherAnyString added in v0.2.0 

type PantherAnyString struct {
	// contains filtered or unexported fields
}

func NewPantherAnyString added in v0.2.0 

func NewPantherAnyString() *PantherAnyString

func (*PantherAnyString) MarshalJSON added in v0.2.0 

func (any *PantherAnyString) MarshalJSON() ([]byte, error)

func (*PantherAnyString) UnmarshalJSON added in v0.2.0 

func (any *PantherAnyString) UnmarshalJSON(jsonBytes []byte) error

type PantherLog added in v0.2.0 

type PantherLog struct {

	//  required
	PantherLogType   *string            `json:"p_log_type,omitempty" validate:"required" description:"Panther added field with type of log"`
	PantherRowID     *string            `json:"p_row_id,omitempty" validate:"required" description:"Panther added field with unique id (within table)"`
	PantherEventTime *timestamp.RFC3339 `json:"p_event_time,omitempty" validate:"required" description:"Panther added standardize event time (UTC)"`
	PantherParseTime *timestamp.RFC3339 `json:"p_parse_time,omitempty" validate:"required" description:"Panther added standardize log parse time (UTC)"`

	// optional (any)
	PantherAnyIPAddresses  *PantherAnyString `json:"p_any_ip_addresses,omitempty" description:"Panther added field with collection of ip addresses associated with the row"`
	PantherAnyDomainNames  *PantherAnyString `json:"p_any_domain_names,omitempty" description:"Panther added field with collection of domain names associated with the row"`
	PantherAnySHA1Hashes   *PantherAnyString `json:"p_any_sha1_hashes,omitempty" description:"Panther added field with collection of SHA1 hashes associated with the row"`
	PantherAnyMD5Hashes    *PantherAnyString `json:"p_any_md5_hashes,omitempty" description:"Panther added field with collection of MD5 hashes associated with the row"`
	PantherAnySHA256Hashes *PantherAnyString `` /* 144-byte string literal not displayed */
	// contains filtered or unexported fields
}

All log parsers should extend from this to get standardized fields (all prefixed with 'p_' as JSON for uniqueness) NOTE: It is VERY important that fields are added to END of the structure to avoid needed to re-build existing Glue partitions. 

See https://github.com/awsdocs/amazon-athena-user-guide/blob/master/doc_source/updates-and-partitions.md

nolint(lll)

func (*PantherLog) AppendAnyDomainNamePtrs added in v0.2.0 

func (pl *PantherLog) AppendAnyDomainNamePtrs(values ...*string)

func (*PantherLog) AppendAnyDomainNames added in v0.2.0 

func (pl *PantherLog) AppendAnyDomainNames(values ...string)

func (*PantherLog) AppendAnyIPAddress added in v1.0.1 

func (pl *PantherLog) AppendAnyIPAddress(value string) bool

func (*PantherLog) AppendAnyIPAddressInField added in v1.1.0 

func (pl *PantherLog) AppendAnyIPAddressInField(value string) bool

AppendAnyIPAddressInField extracts all IPs from the value using a regexp

func (*PantherLog) AppendAnyIPAddressInFieldPtr added in v1.1.0 

func (pl *PantherLog) AppendAnyIPAddressInFieldPtr(value *string) bool

AppendAnyIPAddressInFieldPtr makes sure the value passed is not nil before passing into AppendAnyIPAddressInField

func (*PantherLog) AppendAnyIPAddressPtr added in v1.0.1 

func (pl *PantherLog) AppendAnyIPAddressPtr(value *string) bool

AppendAnyIPAddressPtr returns true if the IP address was successfully appended, otherwise false if the value was not an IP

func (*PantherLog) AppendAnyMD5HashPtrs added in v0.3.0 

func (pl *PantherLog) AppendAnyMD5HashPtrs(values ...*string)

func (*PantherLog) AppendAnyMD5Hashes added in v0.3.0 

func (pl *PantherLog) AppendAnyMD5Hashes(values ...string)

func (*PantherLog) AppendAnySHA1HashPtrs added in v0.3.0 

func (pl *PantherLog) AppendAnySHA1HashPtrs(values ...*string)

func (*PantherLog) AppendAnySHA1Hashes added in v0.3.0 

func (pl *PantherLog) AppendAnySHA1Hashes(values ...string)

func (*PantherLog) AppendAnySHA256Hashes added in v1.2.0 

func (pl *PantherLog) AppendAnySHA256Hashes(values ...string)

func (*PantherLog) AppendAnySHA256HashesPtr added in v1.2.0 

func (pl *PantherLog) AppendAnySHA256HashesPtr(values ...*string)

func (*PantherLog) Event added in v1.0.0 

func (pl *PantherLog) Event() interface{}

Event returns event data, used when composed

func (*PantherLog) Log added in v1.0.0 

func (pl *PantherLog) Log() *PantherLog

Log returns pointer to self, used when composed

func (*PantherLog) Logs added in v1.0.0 

func (pl *PantherLog) Logs() []*PantherLog

Logs returns a slice with pointer to self, used when composed

func (*PantherLog) Result added in v1.5.1 

func (pl *PantherLog) Result() (*Result, error)

Result converts a PantherLog to Result NOTE: Currently in this file to help with review

func (*PantherLog) Results added in v1.5.1 

func (pl *PantherLog) Results() ([]*Result, error)

Results converts a PantherLog to a slice of results NOTE: Currently in this file to help with review

func (*PantherLog) SetCoreFields added in v0.2.0 

func (pl *PantherLog) SetCoreFields(logType string, eventTime *timestamp.RFC3339, event interface{})

func (*PantherLog) SetEvent added in v1.0.0 

func (pl *PantherLog) SetEvent(event interface{})

SetEvent set event data, used for testing

type Result added in v1.5.1 

type Result struct {
	LogType   string
	EventTime time.Time
	JSON      []byte
}

Result is the result of parsing a log event. It contains the JSON form of the pantherlog to be stored for queries.

func ToResults added in v1.5.1 

func ToResults(logs []*PantherLog, err error) ([]*Result, error)

func (*Result) Results added in v1.5.1 

func (r *Result) Results() []*Result

Results wraps a single Result in a slice.

type RowID added in v0.2.0 

type RowID uint64

func (*RowID) NewRowID added in v0.2.0 

func (rid *RowID) NewRowID() string

NewRowID returns a unique row id as a hex string, name spaced as nodeID + timeOffset + rowCounter

Source Files

View all Source files

Directories

Path Synopsis
Package apachelogs contains parsers for logs of the Apache HTTP Server
 Package apachelogs contains parsers for logs of the Apache HTTP Server
Package awslogs defines parsers and log types for AWS logs.
 Package awslogs defines parsers and log types for AWS logs.
Package gcplogs has log parsers for Google Cloud Platform
 Package gcplogs has log parsers for Google Cloud Platform
Package gitlablogs parses GitLab JSON logs.
 Package gitlablogs parses GitLab JSON logs.
Package juniperlogs provides parsers for Juniper logs
 Package juniperlogs provides parsers for Juniper logs
Package nginxlogs provides parsers for NGINX server logs
 Package nginxlogs provides parsers for NGINX server logs
Package sysloglogs provides parsers for syslog messages.
 Package sysloglogs provides parsers for syslog messages.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.