parsers

package
v1.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 24, 2020 License: AGPL-3.0 Imports: 16 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	PantherFieldPrefix = "p_"
)

Variables

View Source 
var JSON = func() jsoniter.API {
	config := jsoniter.Config{
		EscapeHTML: true,

		ValidateJsonRawMessage: true,

		SortMapKeys: false,
	}
	api := config.Froze()
	rewriteFields := jsonutil.NewEncoderNamingStrategy(RewriteFieldName)
	api.RegisterExtension(rewriteFields)
	return api
}()

JSON is a custom jsoniter config to properly remap field names for compatibility with Athena views

View Source 
var Validator = validator.New()

Validator can be used to validate schemas of log fields

Functions

func AppendAnyString added in v0.2.0 

func AppendAnyString(any *PantherAnyString, values ...string)

func CsvStringToArray added in v0.2.0 

func CsvStringToArray(value string) []string

func CsvStringToFloat64Pointer added in v0.2.0 

func CsvStringToFloat64Pointer(value string) *float64

func CsvStringToInt16Pointer added in v0.2.0 

func CsvStringToInt16Pointer(value string) *int16

func CsvStringToIntPointer added in v0.2.0 

func CsvStringToIntPointer(value string) *int

func CsvStringToPointer added in v0.2.0 

func CsvStringToPointer(value string) *string

func LooksLikeCSV added in v1.5.0 

func LooksLikeCSV(log string) bool

Returns true if log looks like a CSV log. It can be used to fail fast for logs that are not CSV

func RewriteFieldName added in v1.3.0 

func RewriteFieldName(name string) string

Types

type LogParser 

type LogParser interface {
	// LogType returns the log type supported by this parser
	LogType() string

	// Parse attempts to parse the provided log line
	// If the provided log is not of the supported type the method returns nil and an error
	Parse(log string) ([]*PantherLog, error)

	// New returns a new instance of the log parser, used like a factory method for stateful parsers
	New() LogParser
}

LogParser represents a parser for a supported log type

type PantherAnyString added in v0.2.0 

type PantherAnyString struct {
	// contains filtered or unexported fields
}

func NewPantherAnyString added in v0.2.0 

func NewPantherAnyString() *PantherAnyString

func (*PantherAnyString) MarshalJSON added in v0.2.0 

func (any *PantherAnyString) MarshalJSON() ([]byte, error)

func (*PantherAnyString) UnmarshalJSON added in v0.2.0 

func (any *PantherAnyString) UnmarshalJSON(jsonBytes []byte) error

type PantherLog added in v0.2.0 

type PantherLog struct {

	//  required
	PantherLogType   *string            `json:"p_log_type,omitempty" validate:"required" description:"Panther added field with type of log"`
	PantherRowID     *string            `json:"p_row_id,omitempty" validate:"required" description:"Panther added field with unique id (within table)"`
	PantherEventTime *timestamp.RFC3339 `json:"p_event_time,omitempty" validate:"required" description:"Panther added standardize event time (UTC)"`
	PantherParseTime *timestamp.RFC3339 `json:"p_parse_time,omitempty" validate:"required" description:"Panther added standardize log parse time (UTC)"`

	// optional (any)
	PantherAnyIPAddresses  *PantherAnyString `json:"p_any_ip_addresses,omitempty" description:"Panther added field with collection of ip addresses associated with the row"`
	PantherAnyDomainNames  *PantherAnyString `json:"p_any_domain_names,omitempty" description:"Panther added field with collection of domain names associated with the row"`
	PantherAnySHA1Hashes   *PantherAnyString `json:"p_any_sha1_hashes,omitempty" description:"Panther added field with collection of SHA1 hashes associated with the row"`
	PantherAnyMD5Hashes    *PantherAnyString `json:"p_any_md5_hashes,omitempty" description:"Panther added field with collection of MD5 hashes associated with the row"`
	PantherAnySHA256Hashes *PantherAnyString `` /* 144-byte string literal not displayed */
	// contains filtered or unexported fields
}

All log parsers should extend from this to get standardized fields (all prefixed with 'p_' as JSON for uniqueness) NOTE: It is VERY important that fields are added to END of the structure to avoid needed to re-build existing Glue partitions. 

See https://github.com/awsdocs/amazon-athena-user-guide/blob/master/doc_source/updates-and-partitions.md

nolint(lll)

func (*PantherLog) AppendAnyDomainNamePtrs added in v0.2.0 

func (pl *PantherLog) AppendAnyDomainNamePtrs(values ...*string)

func (*PantherLog) AppendAnyDomainNames added in v0.2.0 

func (pl *PantherLog) AppendAnyDomainNames(values ...string)

func (*PantherLog) AppendAnyIPAddress added in v1.0.1 

func (pl *PantherLog) AppendAnyIPAddress(value string) bool

func (*PantherLog) AppendAnyIPAddressInField added in v1.1.0 

func (pl *PantherLog) AppendAnyIPAddressInField(value string) bool

AppendAnyIPAddressInField extracts all IPs from the value using a regexp

func (*PantherLog) AppendAnyIPAddressInFieldPtr added in v1.1.0 

func (pl *PantherLog) AppendAnyIPAddressInFieldPtr(value *string) bool

AppendAnyIPAddressInFieldPtr makes sure the value passed is not nil before passing into AppendAnyIPAddressInField

func (*PantherLog) AppendAnyIPAddressPtr added in v1.0.1 

func (pl *PantherLog) AppendAnyIPAddressPtr(value *string) bool

AppendAnyIPAddressPtr returns true if the IP address was successfully appended, otherwise false if the value was not an IP

func (*PantherLog) AppendAnyMD5HashPtrs added in v0.3.0 

func (pl *PantherLog) AppendAnyMD5HashPtrs(values ...*string)

func (*PantherLog) AppendAnyMD5Hashes added in v0.3.0 

func (pl *PantherLog) AppendAnyMD5Hashes(values ...string)

func (*PantherLog) AppendAnySHA1HashPtrs added in v0.3.0 

func (pl *PantherLog) AppendAnySHA1HashPtrs(values ...*string)

func (*PantherLog) AppendAnySHA1Hashes added in v0.3.0 

func (pl *PantherLog) AppendAnySHA1Hashes(values ...string)

func (*PantherLog) AppendAnySHA256Hashes added in v1.2.0 

func (pl *PantherLog) AppendAnySHA256Hashes(values ...string)

func (*PantherLog) AppendAnySHA256HashesPtr added in v1.2.0 

func (pl *PantherLog) AppendAnySHA256HashesPtr(values ...*string)

func (*PantherLog) Event added in v1.0.0 

func (pl *PantherLog) Event() interface{}

Event returns event data, used when composed

func (*PantherLog) Log added in v1.0.0 

func (pl *PantherLog) Log() *PantherLog

Log returns pointer to self, used when composed

func (*PantherLog) Logs added in v1.0.0 

func (pl *PantherLog) Logs() []*PantherLog

Logs returns a slice with pointer to self, used when composed

func (*PantherLog) SetCoreFields added in v0.2.0 

func (pl *PantherLog) SetCoreFields(logType string, eventTime *timestamp.RFC3339, event interface{})

func (*PantherLog) SetEvent added in v1.0.0 

func (pl *PantherLog) SetEvent(event interface{})

SetEvent set event data, used for testing

type RowID added in v0.2.0 

type RowID uint64

func (*RowID) NewRowID added in v0.2.0 

func (rid *RowID) NewRowID() string

NewRowID returns a unique row id as a hex string, name spaced as nodeID + timeOffset + rowCounter

Source Files

View all Source files

Directories

Path Synopsis
Package apachelogs contains parsers for logs of the Apache HTTP Server
 Package apachelogs contains parsers for logs of the Apache HTTP Server
Package juniperlogs provides parsers for Juniper logs
 Package juniperlogs provides parsers for Juniper logs

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.