ipfix

README

ipfix

Package ipfix implements an IPFIX (RFC 5101) parser and interpreter.

Input data from an io.Reader or a []byte is parsed and chunked into messages. Template management and the standard IPFIX types are implemented so a fully parsed data set can be produced. Vendor fields can be added at runtime.

Example

To read an IPFIX stream, create a Session and then use ParseBuffer to parse data coming from a single UDP packet or similar.

var conn net.PacketConn // from somewhere
buf := make([]byte, 65507) // maximum UDP payload length
s := ipfix.NewSession()
for {
    n, _, err := conn.ReadFrom(buf)
    // handle err
    msg, err := s.ParseBuffer(buf[:n])
    // handle msg and err
}

To interpret records for correct data types and field names, use an interpreter:

i := ipfix.NewInterpreter(s)
var fieldList []ipfix.InterpretedField
for _, rec := range msg.DataRecords {
    fieldList = i.InterpretInto(rec, fieldList[:cap(fieldList)])
    // handle the field list
}

To add a vendor field to the dictionary so that it will be resolved by Interpret, create a DictionaryEntry and call AddDictionaryEntry.

e := ipfix.DictionaryEntry{
    Name: "someVendorField",
    FieldId: 42,
    EnterpriseId: 123456,
    Type: ipfix.Int32
}
i.AddDictionaryEntry(e)

License

The MIT license.

Usage

See the documentation.

Documentation

Overview

Package ipfix implements an IPFIX (RFC 5101) parser and interpreter.

Index

• Variables
• type DataRecord
• type DictionaryEntry
• type FieldType
  • func (f *FieldType) UnmarshalText(bs []byte) error
• type InterpretedField
• type InterpretedTemplateFieldSpecifier
• type Interpreter
  • func NewInterpreter(s *Session) *Interpreter
  • func (i *Interpreter) AddDictionaryEntry(e DictionaryEntry)
  • func (i *Interpreter) Interpret(rec DataRecord) []InterpretedField
  • func (i *Interpreter) InterpretInto(rec DataRecord, fieldList []InterpretedField) []InterpretedField
  • func (i *Interpreter) InterpretTemplate(rec TemplateRecord) []InterpretedTemplateFieldSpecifier
• type Message
• type MessageHeader
  • func Read(r io.Reader, bs []byte) ([]byte, MessageHeader, error)
• type Option
  • func WithIDAliasing(v bool) Option
• type Session
  • func NewSession(opts ...Option) *Session
  • func (s *Session) ExportTemplateRecords() []TemplateRecord
  • func (s *Session) LoadTemplateRecords(trecs []TemplateRecord)
  • func (s *Session) ParseBuffer(bs []byte) (Message, error)
  • func (s *Session) ParseBufferAll(bs []byte) ([]Message, error)
  • func (s *Session) ParseReader(r io.Reader) (Message, error)
• type TemplateFieldSpecifier
• type TemplateRecord

Examples

• Interpreter
• Interpreter.AddDictionaryEntry
• Session

Variables

var ErrProtocol = errors.New("protocol error")

ErrProtocol is returned when impossible values that constitute a protocol error are encountered.

var ErrRead = errors.New("short read - malformed packet?")

ErrRead is returned when a packet is not long enough for the field it is supposed to contain. This is a sign of an earlier read error or a corrupted packet.

var ErrVersion = errors.New("incorrect version field in message header - out of sync?")

The version field in IPFIX messages should always have the value 10. If it does not, you get this error. It's probably a sign of a bug in the parser or the exporter and that we have lost synchronization with the data stream. Reestablishing the session is the only way forward at this point.

var FieldTypes = map[string]FieldType{
	"unsigned8":            Uint8,
	"unsigned16":           Uint16,
	"unsigned32":           Uint32,
	"unsigned64":           Uint64,
	"signed8":              Int8,
	"signed16":             Int16,
	"signed32":             Int32,
	"signed64":             Int64,
	"float32":              Float32,
	"float64":              Float64,
	"boolean":              Boolean,
	"macAddress":           MacAddress,
	"octetArray":           OctetArray,
	"string":               String,
	"dateTimeSeconds":      DateTimeSeconds,
	"dateTimeMilliseconds": DateTimeMilliseconds,
	"dateTimeMicroseconds": DateTimeMicroseconds,
	"dateTimeNanoseconds":  DateTimeNanoseconds,
	"ipv4Address":          Ipv4Address,
	"ipv6Address":          Ipv6Address,
}

FieldTypes maps string representations of field types into their corresponding FieldType value.

Types

type DataRecord

type DataRecord struct {
	TemplateID uint16
	Fields     [][]byte
}

The DataRecord represents a single exported flow. The Fields each describe different aspects of the flow (source and destination address, counters, service, etc.).

type DictionaryEntry

type DictionaryEntry struct {
	Name         string
	FieldID      uint16
	EnterpriseID uint32
	Type         FieldType
}

DictionaryEntry provides a mapping between an (Enterprise, Field) pair and a Name and Type.

type FieldType

type FieldType int

FieldType is the IPFIX type of an Information Element ("Field").

const (
	Unknown FieldType = iota
	Uint8
	Uint16
	Uint32
	Uint64
	Int8
	Int16
	Int32
	Int64
	Float32
	Float64
	Boolean
	MacAddress
	OctetArray
	String
	DateTimeSeconds
	DateTimeMilliseconds
	DateTimeMicroseconds
	DateTimeNanoseconds
	Ipv4Address
	Ipv6Address
)

The available field types as defined by RFC 5102.

func (*FieldType) UnmarshalText

func (f *FieldType) UnmarshalText(bs []byte) error

type InterpretedField

type InterpretedField struct {
	Name         string
	EnterpriseID uint32
	FieldID      uint16
	Value        interface{}
	RawValue     []byte
}

An InterpretedField is a field with the field name filled in and the value converted to the appropriate type. If this is not possible (because the name and type of the field is unknown at the time of interpretation), Name will be the empty string, Value will be a nil interface and RawValue will contain the original bytes.

type InterpretedTemplateFieldSpecifier

type InterpretedTemplateFieldSpecifier struct {
	Name string
	TemplateFieldSpecifier
}

An InterpretedTemplateFieldSpecifier is a template specifier with the field name filled in, if found in the dictionary.

type Interpreter

type Interpreter struct {
	// contains filtered or unexported fields
}

Interpreter provides translation between the raw bytes of a DataRecord and the actual values as specified by the corresponding template.

Example

package main

import (
	"fmt"
	"os"

	"github.com/calmh/ipfix"
)

func main() {
	s := ipfix.NewSession()
	i := ipfix.NewInterpreter(s)

	for {
		// ParseReader will block until a full message is available.
		msg, err := s.ParseReader(os.Stdin)
		if err != nil {
			panic(err)
		}

		var fieldList []ipfix.InterpretedField
		for _, record := range msg.DataRecords {
			fieldList = i.InterpretInto(record, fieldList)
			fmt.Println(fieldList)
		}
	}
}

func NewInterpreter

func NewInterpreter(s *Session) *Interpreter

NewInterpreter craets a new Interpreter based on the specified Session.

func (*Interpreter) AddDictionaryEntry

func (i *Interpreter) AddDictionaryEntry(e DictionaryEntry)

AddDictionaryEntry adds a DictionaryEntry (containing a vendor field) to the dictionary used by Interpret.

Example

package main

import (
	"github.com/calmh/ipfix"
)

func main() {
	s := ipfix.NewSession()
	i := ipfix.NewInterpreter(s)

	entry := ipfix.DictionaryEntry{
		Name:         "someVendorField",
		FieldID:      42,
		EnterpriseID: 123456,
		Type:         ipfix.Int32,
	}

	i.AddDictionaryEntry(entry)

	// Now use i.Interpret() etc as usual.
}

func (*Interpreter) Interpret

func (i *Interpreter) Interpret(rec DataRecord) []InterpretedField

Interpret a raw DataRecord into a list of InterpretedFields.

func (*Interpreter) InterpretInto

func (i *Interpreter) InterpretInto(rec DataRecord, fieldList []InterpretedField) []InterpretedField

InterpretInto interprets a raw DataRecord into an existing slice of InterpretedFields. If the slice is not long enough it will be reallocated.

func (*Interpreter) InterpretTemplate

func (i *Interpreter) InterpretTemplate(rec TemplateRecord) []InterpretedTemplateFieldSpecifier

InterpretTemplate interprets a template record and adds a name to the interpreted fields, if a given {EnterpriseID,FieldID} can be find in the dictionary.

type Message

type Message struct {
	Header          MessageHeader
	DataRecords     []DataRecord
	TemplateRecords []TemplateRecord
}

A Message is the top level construct representing an IPFIX message. A well formed message contains one or more sets of data or template information.

type MessageHeader

type MessageHeader struct {
	Version        uint16 // Always 0x0a
	Length         uint16
	ExportTime     uint32 // Epoch seconds
	SequenceNumber uint32
	DomainID       uint32
}

The MessageHeader provides metadata for the entire Message. The sequence number and domain ID can be used to gain knowledge of messages lost on an unreliable transport such as UDP.

func Read

func Read(r io.Reader, bs []byte) ([]byte, MessageHeader, error)

Read reads and returns an IPFIX message, the parsed message header and an error or nil. The given byte slice is used and returned (sliced to the message length) if it is large enough to contain the message; otherwise a new slice is allocated. The returned message slice contains the message header.

type Option

type Option func(*Session)

An option can be passed to New()

func WithIDAliasing

func WithIDAliasing(v bool) Option

WithIDAliasing enables or disables template id aliasing. The default is disabled.

type Session

type Session struct {
	// contains filtered or unexported fields
}

The Session is the context for IPFIX messages.

Example

package main

import (
	"fmt"
	"os"

	"github.com/calmh/ipfix"
)

func main() {
	s := ipfix.NewSession()

	for {
		// ParseReader will block until a full message is available.
		msg, err := s.ParseReader(os.Stdin)
		if err != nil {
			panic(err)
		}

		for _, record := range msg.DataRecords {
			// record contains raw enterpriseId, fieldId => []byte information
			fmt.Println(record)
		}
	}
}

func NewSession

func NewSession(opts ...Option) *Session

NewSession initializes a new Session based on the provided io.Reader.

func (*Session) ExportTemplateRecords

func (s *Session) ExportTemplateRecords() []TemplateRecord

func (*Session) LoadTemplateRecords

func (s *Session) LoadTemplateRecords(trecs []TemplateRecord)

func (*Session) ParseBuffer

func (s *Session) ParseBuffer(bs []byte) (Message, error)

ParseBuffer extracts one message from the given buffer and returns it. Err is nil if the buffer could be parsed correctly. ParseBuffer is goroutine safe.

func (*Session) ParseBufferAll

func (s *Session) ParseBufferAll(bs []byte) ([]Message, error)

ParseBufferAll extracts all message from the given buffer and returns them. Err is nil if the buffer could be parsed correctly. ParseBufferAll is goroutine safe.

func (*Session) ParseReader

func (s *Session) ParseReader(r io.Reader) (Message, error)

ParseReader extracts and returns one message from the IPFIX stream. As long as err is nil, further messages can be read from the stream. Errors are not recoverable -- once an error has been returned, ParseReader should not be called again on the same session.

Deprecated: use ParseBuffer instead.

type TemplateFieldSpecifier

type TemplateFieldSpecifier struct {
	EnterpriseID uint32
	FieldID      uint16
	Length       uint16
}

The TemplateFieldSpecifier describes the ID and size of the corresponding Fields in a DataRecord.

type TemplateRecord

type TemplateRecord struct {
	TemplateID      uint16
	FieldSpecifiers []TemplateFieldSpecifier
}

The TemplateRecord describes a data template, as used by DataRecords.