anomaly

README

Anomaly Detector

The anomaly service type implements anomaly detection for payloads. The anomaly detection algorithm is based on a statistical model for compression. The anomaly detection algorithm computes the relative complexity, K(payload | previous payloads), of a payload and then updates the statistical model. A running mean and standard deviation of the complexity is then computed using this algorithm. If the complexity of a payload is some number of deviations from the mean then it is an anomaly. An anomaly is a payload that is statistically significant relative to previous payloads. The anomaly detection algorithm uses real time learning, so what is considered an anomaly can change over time.

The available service settings are as follows:

Name	Type	Description
depth	number	The size of the statistical model. Defaults to 2

The available inputs for the request are as follows:

Name	Type	Description
payload	JSON object	A payload to do anomaly detection on

The available response outputs are as follows:

Name	Type	Description
complexity	number	How unusual the payload is in terms of standard deviations from the mean
count	number	The number of payloads that have been processed

A sample service definition is:

{
  "name": "Anomaly",
  "description": "Look for anomalies",
  "ref": "github.com/AiRISTAFlowInc/fs-microgateway/activity/anomaly",
  "settings": {
    "depth": 3
  }
}

An example step that invokes the above Anomaly service using payload is:

{
  "service": "Anomaly",
  "input": {
    "payload": "=$.payload.content"
  }
}

Utilizing the response values can be seen in a response handler:

{
  "if": "($.Anomaly.outputs.count < 100) || ($Anomaly.outputs.complexity < 3)",
  "error": false,
  "output": {
    "code": 200,
    "data": "=$.Update.outputs.data"
  }
}

Documentation

Index

• Constants
• func New(ctx activity.InitContext) (activity.Activity, error)
• type Activity
  • func (a *Activity) Eval(ctx activity.Context) (done bool, err error)
  • func (a *Activity) Metadata() *activity.Metadata
• type CDF16
  • func NewCDF16() *CDF16
  • func (c *CDF16) Model(ctxt *Context16) []uint16
  • func (c *CDF16) Update(s uint16, ctxt *Context16)
• type Complexity
  • func NewComplexity(depth int) *Complexity
  • func (c *Complexity) Complexity(input []byte) (float32, int)
• type Context16
  • func NewContext16(depth int) *Context16
  • func (c *Context16) AddContext(s uint16)
  • func (c *Context16) ResetContext()
• type Input
  • func (r *Input) FromMap(values map[string]interface{}) error
  • func (r *Input) ToMap() map[string]interface{}
• type Node16
  • func NewNode16() *Node16
• type Output
  • func (o *Output) FromMap(values map[string]interface{}) error
  • func (o *Output) ToMap() map[string]interface{}
• type Settings

Constants

const (
	// CDF16Fixed is the shift for 16 bit coders
	CDF16Fixed = 16 - 3
	// CDF16Scale is the scale for 16 bit coder
	CDF16Scale = 1 << CDF16Fixed
	// CDF16Rate is the damping factor for 16 bit coder
	CDF16Rate = 5
	// CDF16Size is the size of the cdf
	CDF16Size = 256
	// CDF16Depth is the depth of the context tree
	CDF16Depth = 2
)

Functions

func New

func New(ctx activity.InitContext) (activity.Activity, error)

Types

type Activity

type Activity struct {
	// contains filtered or unexported fields
}

Activity is an anomaly detector

func (*Activity) Eval

func (a *Activity) Eval(ctx activity.Context) (done bool, err error)

Eval executes the activity

func (*Activity) Metadata

func (a *Activity) Metadata() *activity.Metadata

Metadata return the metadata for the activity

type CDF16

type CDF16 struct {
	Root  *Node16
	Mixin [][]uint16
}

CDF16 is a context based cumulative distributive function model https://fgiesen.wordpress.com/2015/05/26/models-for-adaptive-arithmetic-coding/

func NewCDF16

func NewCDF16() *CDF16

NewCDF16 creates a new CDF16 with a given context depth

func (*CDF16) Model

func (c *CDF16) Model(ctxt *Context16) []uint16

Model gets the model for the current context

func (*CDF16) Update

func (c *CDF16) Update(s uint16, ctxt *Context16)

Update updates the model

type Complexity

type Complexity struct {
	*CDF16

	sync.RWMutex
	// contains filtered or unexported fields
}

Complexity is an entorpy based anomaly detector

func NewComplexity

func NewComplexity(depth int) *Complexity

NewComplexity creates a new entorpy based model

func (*Complexity) Complexity

func (c *Complexity) Complexity(input []byte) (float32, int)

Complexity outputs the complexity

type Context16

type Context16 struct {
	Context []uint16
	First   int
}

Context16 is a 16 bit context

func NewContext16

func NewContext16(depth int) *Context16

NewContext16 creates a new context

func (*Context16) AddContext

func (c *Context16) AddContext(s uint16)

AddContext adds a symbol to the context

func (*Context16) ResetContext

func (c *Context16) ResetContext()

ResetContext resets the context

type Input

type Input struct {
	Payload interface{} `md:"payload"`
}

func (*Input) FromMap

func (r *Input) FromMap(values map[string]interface{}) error

func (*Input) ToMap

func (r *Input) ToMap() map[string]interface{}

type Node16

type Node16 struct {
	Model    []uint16
	Children map[uint16]*Node16
}

Node16 is a context node

func NewNode16

func NewNode16() *Node16

NewNode16 creates a new context node

type Output

type Output struct {
	Complexity float32 `md:"complexity"`
	Count      int     `"md:"count`
}

func (*Output) FromMap

func (o *Output) FromMap(values map[string]interface{}) error

func (*Output) ToMap

func (o *Output) ToMap() map[string]interface{}

type Settings

type Settings struct {
	Depth int `md:"depth"`
}