xdrgateway

package module
v0.1.7 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 18, 2021 License: Apache-2.0 Imports: 9 Imported by: 0

README

godoc

Basic PAN-OS to XDR HTTP log forwarding GW

A compact application offering a HTTP server that can listen to alerts generated by the HTTP log forwarding feature available on all PAN-OS devices and forward them to Cortex XDR using its alert ingestion API

Features

  • configurable buffered pipeline to accomodate alert burts (XDR alert ingestion API defaults to 600 external alerts per minute)
  • support for XDR Advanced API Keys (no support for Standard API Keys)
  • engine statistics

Build the docker image

docker build -t xdrgw https://github.com/xhoms/xdrgateway.git#main

Running the application

The application requires some mandatory environmental variables and accepts some optional ones.

The following are the required variables (the application will refuse to start without them)

  • API_KEY - XDR API Key (Advanced)
  • API_KEY_ID - The XDR API Key identifier (its sequence number)
  • FQDN - Full Qualified Domain Name of the corresponding XDR Instance (i.e. myxdr.xdr.us.paloaltonetworks.com)

The following are optional variables

  • PSK - the server will check the value in the Authorization header to accept the request (default to no authentication)
  • DEBUG - if it exists then the engine will be more verbose (defaults to false)
  • PORT - TCP port to bind the http server to (defaults to 8080)
  • OFFSET - PAN-OS timestamp does not include time zone. By default they will be considerd in UTC (defauls to +0 hours)
  • QUOTA_SIZE - XDR ingestion alert quota (defaults to 600)
  • QUOTA_SECONDS - XDR ingestion alert quota refresh period (defaults to 60 seconds)
  • UPDATE_SIZE - XDR ingestion alert max number of alerts per update (defaults to 60)
  • BUFFER_SIZE - size of the pipe buffer (defaults to 6000 = 10 minutes)
  • T1 - how often the pipe buffer polled for new alerts (defaulst to 2 seconds)

Example shell session running the application

$ docker run --rm -p 8080:8080 \
-e API_KEY="O4Bw...wEX" \
-e API_KEY_ID="36" \
-e FQDN="myxdr.xdr.us.paloaltonetworks.com" \
-e PSK="hello" \
xdrgw
PAN-OS to Cortex XDR alert ingestion Gateway
--------------------------------------------
  - Send PAN_OS alerts to /in using HTTP POST
  - The endpoint /stats provides runtime statistics
  - Use the following payload in the HTTP Log Forwarding feature
{
        "src": "$src",
        "sport": $sport,
        "dst": "$dst",
        "dport": $dport,
        "time_generated": "$time_generated",
        "rule": "$rule",
        "serial": "$serial",
        "sender_sw_version": "$sender_sw_version",
        "subtype": "$subtype",
        "severity": "$severity",
        "threat_name": "$threat_name", 
        "action": "$action"
}
---annex---
$misc

2021/02/09 11:51:32 nonce set to EEH4PO4BQY42YSFEY2X2F4KYDKFZKJPCB7NGRET7FMX7QNXXGV4NWD5FJQU7P7MS
2021/02/09 11:51:32 ednpoint set to https://api-myxdr.xdr.us.paloaltonetworks.com/public_api/v1/alerts/insert_parsed_alerts/
2021/02/09 11:51:32 starting http service on port 8080
2021/02/09 11:51:32 starting sender goroutine
2021/02/09 11:51:32 starting ticker goroutine

Servicing on TLS

You're encouraged to run this container image behind a forward proxy service providing the TLS frontend (i.e. GCP Cloud Run or a NGINX server)

Configuring the PAN-OS device

Check PAN-OS documentation on how to configure a HTTP Server and use it in a Log Forwarding Profile. Only Medium/High/Critical threat alerts should be forwarded to avoid exceeding the ingestion quota. The payload seen bellow leverages the attribute $threat_name that was introduced in PAN-OS 10.1. For earlier versions use $threatid instead.

The application provides the endpoint /dump that returns the payload that should be used in the PAN-OS HTTP Server.

Alerts must be sent to the /in endpoint in the application using method POST

Example bash session retrieving the payload to be configured in the PAN-OS device.

$ curl 127.0.0.1:8080/dump -H "Authorization: hello" 
{
    "src": "$src",
    "sport": $sport,
    "dst": "$dst",
    "dport": $dport,
    "time_generated": "$time_generated",
    "rule": "$rule",
    "serial": "$serial",
    "sender_sw_version": "$sender_sw_version",
    "subtype": "$subtype",
    "severity": "$severity",
    "threat_name": "$threat_name",
    "action": "$action"
}
---annex---
$misc

Runtime Statistics

The application provides, as well, the /stats endpoint.

Example session retrieving the statistics

% curl 127.0.0.1:8080/stats -H "Authorization: hello"
{
  "ParseErrors": 0,
  "EventsReceived": 0,
  "PSKErrors": 0,
  "POSTSend": 0,
  "POSTFailures": 0,
  "AlertsSend": 0,
  "SendFailures": 0,
  "UpdatesSend": 0,
  "Discards": 0
}
  • ParseErrors - events received by the application in the /in endpoint that could not be parsed into alerts (payload error?)
  • EventsReceived - number of times the /in endpoint has been reached
  • PSKErrors - authentication errors
  • POSTSend - successful updates to the XDR insert alert API (status = 200 OK)
  • POSTFailures - unsuccessful updates to the XDR insert alert API (status != 200 OK)
  • AlertsSend - Amount of alerts successfully moved across the buffered pipe
  • SendFailures - Internal errors rendering the XDR API update payload
  • UpdatesSend - Successful XDR API update payloads rendered
  • Discards - alerts dropped in the buffered pipe (too many?)

Documentation

Overview

Package xdrgateway provides the tools needed to create an alert pipeline ingestion into Palo Alto Network Cortex XDR

The API type provides methods to implement a HTTP API to ingest third party alerts into XDR. Not only provides HTTP handlers for receiving alerts sent using POST but also implements a synchonous ingestion pipeline that will enforce Cortex XDR ingestion quotas.

The API requires a XDRClient and a Parser instances. The Parser interface defines the methods to convert the []byte data received by the API in its ingestion endpoint into a valid *Alert 

type Parser interface {
	Parse(data []byte) (*Alert, error)
	DumpPayloadLayout() []byte
}

Ingestion(w http.ResponseWriter, r *http.Request) is the most important method provided by API. It is a ready-to-consume http handler to process POST request containing third party alerts.

Look at the provided examplex to see an implementation parsing alerts generated by the HTTP Log Forwarding PAN-OS feature.

Ready-to-consume PAN-OS to Cortex XDR implementation

This repository contains a standalone application example (/cmd/server.go) that can be used to cover the use case of PAN-OS threat alerts being ingested into Cortex XDR for small or highly distributed environments that do not qualify for Cortex Data Lake The example application can be run as a compact container application (FROM distroless/static). It binds the HTTP server into the port provided in the PORT environmental variable (defaults to 8080) which means it can run in almost any container managed service. 

docker build -t xdrgw https://github.com/xhoms/xdrgateway.git#main

docker run --rm -p 8080:8080 \
-e API_KEY="O4Bw...wEX" \
-e API_KEY_ID="36" \
-e FQDN="myxdr.xdr.us.paloaltonetworks.com" \
-e PSK="hello" \
xdrgw

2021/02/18 12:30:11 nonce set to EEH4PO4BQY42YSFEY2X2F4KYDKFZKJPCB7NGRET7FMX7QNXXGV4NWD5FJQU7P7MS
2021/02/18 12:30:11 endpoint set to https://api-illicium-industrial.xdr.us.paloaltonetworks.com/public_api/v1/alerts/insert_parsed_alerts/
2021/02/18 12:30:11 starting http service on port 8081
PAN-OS to Cortex XDR alert ingestion Gateway
--------------------------------------------
version: v0.1.4 2021-02-18T12:28+00:00
- Send PAN_OS alerts to /in using HTTP POST
- The endpoint /stats provides runtime statistics
- Use the following payload in the HTTP Log Forwarding feature
{
	"src": "$src",
	"sport": $sport,
	"dst": "$dst",
	"dport": $dport,
	"time_generated": "$time_generated",
	"rule": "$rule",
	"serial": "$serial",
	"sender_sw_version": "$sender_sw_version",
	"subtype": "$subtype",
	"threat_name": "$threat_name",
	"severity": "$severity",
	"action": "$action"
}
---annex---
$misc

2021/02/18 12:30:11 starting ticker goroutine
2021/02/18 12:30:11 starting sender goroutine

Index

Examples

Constants

View Source 
const Version = "v0.1.6"

Version of the package

Variables

This section is empty.

Functions

This section is empty.

Types

type API 

type API struct {
	// contains filtered or unexported fields
}

API provides HTTP methods to implement the PAN-OS facing ingestion API

Example

Example that creates a HTTP server that can be used to ingest PAN-OS alerts into Cortex XDR leveraging the NGFW's HTTP Log Forwarding feature 

port := "8080"
client := xdrclient.NewClientFromEnv()
parser := NewBasicParser(0, false)
api := NewAPI(parser, client, "", false, nil)
http.HandleFunc("/in", api.HandlerIngestion)
log.Println("payload template to be used in the PAN-OS device")
fmt.Println(string(parser.DumpPayloadLayout()))
log.Println("starting http service on port", port)
log.Fatal(http.ListenAndServe(":"+port, nil))

Output:

func NewAPI 

func NewAPI(parser Parser, xdrClient *xdrclient.Client, psk string, debug bool, pipe *AlertPipeOps) (api *API)

NewAPI creates and initializes a xdrgateway instance from values

func (*API) Close added in v0.1.5 

func (a *API) Close()

Close attempts to gracefully shutdown the pipeline goroutines

func (*API) HandlerHint added in v0.1.5 

func (a *API) HandlerHint(w http.ResponseWriter, r *http.Request)

HandlerHint http.HandleFunc compatible handler that dumps the parser layout hint

func (*API) HandlerIngestion added in v0.1.5 

func (a *API) HandlerIngestion(w http.ResponseWriter, r *http.Request)

HandlerIngestion http.HandleFunc compatible handler for PAN-OS alert ingestion only POST method supported

func (*API) HandlerStats added in v0.1.5 

func (a *API) HandlerStats(w http.ResponseWriter, r *http.Request)

HandlerStats http.HandleFunc compatible handler that dumps runtime statistics

func (*API) Ingest added in v0.1.5 

func (a *API) Ingest(payload []byte) (err error)

Ingest attempts to parse the provide payload and, if successful, ingests the resulting alert into the pipe

type APIStats 

type APIStats struct {
	// ParseErrors is the number of received events that have failed to be parsed
	ParseErrors int64
	// EventsReceived is the total number of API invocations in the ingestion endpoint
	EventsReceived int64
	// PSKErrors is increased each time an event is rejected due to PSK mismatch
	PSKErrors int64
}

APIStats provides counters for the PAN-OS facing API part

type AlertPipeOps 

type AlertPipeOps struct {
	// XDRUpdateSize max amount of alerts in a single XDR API update
	XDRUpdateSize int
	// XDRMQuotaSize max amount of alerts that can be ingested into XDR API per each period
	XDRMQuotaSize int
	// XDRQuotaSeconds XDR API ingestion quota refresh period (seconds)
	XDRQuotaSeconds int
	// AlertBufferSize how many alerts can he held in the buffered pipe
	AlertBufferSize int
	// T1 controls how fast the pipe is polled to drain alerts (seconds)
	T1 int
	// Debug to increase the verbosity of the pipe
	Debug bool
}

AlertPipeOps options to fine-tune the pipe behavior

func NewPipeOpsFromEnv 

func NewPipeOpsFromEnv() (ops *AlertPipeOps)

NewPipeOpsFromEnv creates pipe options by reading environmental variables

Optional environmental variables

- DEBUG if it exists then the engine will be more verbose (defaults to false)

- QUOTA_SIZE XDR ingestion alert quota (defaults to 600)

- QUOTA_SECONDS XDR ingestion alert quota refresh period (defaults to 60 seconds)

- UPDATE_SIZE XDR ingestion alert max number of alerts per update (defaults to 60)

- BUFFER_SIZE size of the pipe buffer (defaults to 6000 = 10 minutes)

- T1 how often the pipe buffer is polled for new alerts (defaulst to 2 seconds)

type AppStats 

type AppStats struct {
	APIStats
	xdrclient.Stats
	PipeStats
}

AppStats hold runtime statistics for the application

type BasicParser 

type BasicParser struct {
	// contains filtered or unexported fields
}

BasicParser implements xdrgateway.Parser interface

func NewBasicParser 

func NewBasicParser(offset int, debug bool) (b *BasicParser)

NewBasicParser returns a parser with TimeZone set to `offset`-hours (negative values supported)

func (*BasicParser) DumpPayloadLayout 

func (b *BasicParser) DumpPayloadLayout() []byte

DumpPayloadLayout provides human-readable format of the supported PAN-OS payload for this parser

func (*BasicParser) Parse 

func (b *BasicParser) Parse(data []byte) (alert *xdrclient.Alert, err error)

Parse converts data into a XDR Alert. Return error if parsing fails

type Parser 

type Parser interface {
	// Parse attempts to fill a XDR alert with the payload pushed by the PAN-OS device
	Parse(data []byte) (*xdrclient.Alert, error)
	// DumpPayloadLayout returns a human-readable helper to assist the PAN-OS administrator preparing the payload for this parser
	DumpPayloadLayout() []byte
}

Parser provides methods to parse PAN-OS alerts into XDR Alerts

type PipeStats added in v0.1.5 

type PipeStats struct {
	// PipeIn is the amount of alerts that have traversed the pipe
	PipeIn uint64
	// PipeInErr accumulates the number of PAN-OS alerts that have been discarded due to pipe buffer overflow
	PipeInErr uint64
	// PipeOutErr is the number of times the pipe failed to render a XDR API payload
	PipeOutErr uint64
	// PipeOut is the number of valid XDR API payloads generated by the pipe
	PipeOut uint64
}

PipeStats provides counters for the buffered Alert pipe

Source Files

View all Source files

Directories

Path Synopsis
Package xdrclient provides code to implement a client to the ingestion parser alerts API in XDR.
 Package xdrclient provides code to implement a client to the ingestion parser alerts API in XDR.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.