netcap

README

The Netcap (NETwork CAPture) framework efficiently converts a stream of network packets into platform neutral type-safe structured audit records that represent specific protocols or custom abstractions. These audit records can be stored on disk or exchanged over the network, and are well suited as a data source for machine learning algorithms. Since parsing of untrusted input can be dangerous and network data is potentially malicious, implementation was performed in a programming language that provides a garbage collected memory safe runtime.

It was developed for a series of experiments in my bachelor thesis: Implementation and evaluation of secure and scalable anomaly-based network intrusion detection. Slides from my presentation at the Leibniz Supercomputing Centre of the Bavarian Academy of Sciences and Humanities are available on researchgate.

The project won the 2nd Place at Kaspersky Labs SecurIT Cup 2018 in Budapest.

Netcap uses Google's Protocol Buffers to encode its output, which allows accessing it across a wide range of programming languages. Alternatively, output can be emitted as comma separated values, which is a common input format for data analysis tools and systems. The tool is extensible and provides multiple ways of adding support for new protocols, while implementing the parsing logic in a memory safe way. It provides high dimensional data about observed traffic and allows the researcher to focus on experimenting with novel approaches for detecting malicious behavior in network environments, instead of fiddling with data collection mechanisms and post processing steps. It has a concurrent design that makes use of multi-core architectures. The name Netcap was chosen to be simple and descriptive. The command-line tool was designed with usability and readability in mind, and displays progress when processing packets. The latest version offers 58 audit record types of which 53 are protocol specific and 5 are flow models.

For more details please refer to the Documentation, visit the Homepage and read the thesis.

A simple demonstration of generating audit records from a PCAP dump file, querying and displaying the collected information in various ways:

And live operation decoding traffic from my wireless network interface, while I am surfing the web:

Design Goals

• memory safety when parsing untrusted input
• ease of extension
• output format interoperable with many different programming languages
• concurrent design
• output with small storage footprint on disk
• maximum data availability
• allow implementation of custom abstractions
• rich platform and architecture support

Use Cases

• monitoring honeypots
• monitoring medical / industrial devices
• research on anomaly-based detection mechanisms
• Forensic data analysis

License

Netcap is licensed under the GNU General Public License v3, which is a very permissive open source license, that allows others to do almost anything they want with the project, except to distribute closed source versions. This license type was chosen with Netcaps research purpose in mind, and in the hope that it leads to further improvements and new capabilities contributed by other researchers on the long term.

Documentation

Overview

Netcap is a framework for network packet processing at the protocol level. Learn more at https://netcap.io.

Index

• Constants
• Variables
• func CloseFile(outDir string, file *os.File, typ string) (name string, size int64)
• func CloseGzipWriters(writers ...*gzip.Writer)
• func Count(filename string) (count int64)
• func CreateFile(name, ext string) *os.File
• func Dump(c DumpConfig)
• func FlushWriters(writers ...flushableWriter)
• func InitRecord(typ types.Type) (record proto.Message)
• func NewHeader(t types.Type, source, version string, includesPayloads bool) *types.Header
• func PrintBuildInfo()
• func PrintLogo()
• func RemoveAuditRecordFileIfEmpty(name string) (size int64)
• type DumpConfig
• type Reader
  • func Open(file string, memBufSize int) (*Reader, error)
  • func (r *Reader) Close() error
  • func (r *Reader) Next(msg proto.Message) error
  • func (r *Reader) ReadHeader() *types.Header
• type Writer
  • func NewWriter(name string, buffer, compress, csv bool, out string, writeChan bool, ...) *Writer
  • func (w *Writer) Close() (name string, size int64)
  • func (w *Writer) GetChan() <-chan []byte
  • func (w *Writer) IsCSV() bool
  • func (w *Writer) Write(msg proto.Message) error
  • func (w *Writer) WriteCSV(msg proto.Message) (int, error)
  • func (w *Writer) WriteCSVHeader(msg proto.Message) (int, error)
  • func (w *Writer) WriteHeader(t types.Type, source string, version string, includesPayloads bool) error
  • func (w *Writer) WriteProto(msg proto.Message) error

Constants

const DefaultBufferSize = 1024 * 1024 * 10 // 10MB

Variables

var Commit = "6759e53197c4df3fc56d6e769863cbe7fbd35db2"

var Version = "v0.4.7"

Netcap version.

Functions

func CloseFile

func CloseFile(outDir string, file *os.File, typ string) (name string, size int64)

CloseFile closes the netcap file handle and removes files that do only contain a header but no audit records

func CloseGzipWriters

func CloseGzipWriters(writers ...*gzip.Writer)

func Count

func Count(filename string) (count int64)

Count returns the total number of records found in an audit record file

func CreateFile

func CreateFile(name, ext string) *os.File

CreateFile is a wrapper to create new audit record file

func Dump

func Dump(c DumpConfig)

Dump reads the specified netcap file and dumps the output according to the configuration to stdout

func FlushWriters

func FlushWriters(writers ...flushableWriter)

func InitRecord

func InitRecord(typ types.Type) (record proto.Message)

InitRecord initializes a new record of the given type that conforms to the proto.Message interface if netcap is extended with new audit records they need to be added here as well

func NewHeader

func NewHeader(t types.Type, source, version string, includesPayloads bool) *types.Header

NewHeader creates and returns a new netcap audit file header

func PrintBuildInfo

func PrintBuildInfo()

PrintBuildInfo displays build information related to netcap

func PrintLogo

func PrintLogo()

PrintLogo prints the netcap logo

func RemoveAuditRecordFileIfEmpty

func RemoveAuditRecordFileIfEmpty(name string) (size int64)

RemoveAuditRecordFileIfEmpty removes the audit record file if it does not contain audit records

Types

type DumpConfig

type DumpConfig struct {
	Path          string
	Separator     string
	TabSeparated  bool
	Structured    bool
	Table         bool
	Selection     string
	UTC           bool
	Fields        bool
	JSON          bool
	MemBufferSize int
}

DumpConfig contains all possible settings for dumping an audit records

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

Reader implements reading netcap files

func Open

func Open(file string, memBufSize int) (*Reader, error)

Open a file

func (*Reader) Close

func (r *Reader) Close() error

Close the file

func (*Reader) Next

func (r *Reader) Next(msg proto.Message) error

Next Message

func (*Reader) ReadHeader

func (r *Reader) ReadHeader() *types.Header

ReadHeader reads the file header

type Writer

type Writer struct {

	// Name of the associated audit record type
	Name string

	IsChanWriter bool
	// contains filtered or unexported fields
}

Writer is a structure that supports writing audit records to disk

func NewWriter

func NewWriter(name string, buffer, compress, csv bool, out string, writeChan bool, memBufferSize int) *Writer

NewWriter initializes and configures a new Writer

func (*Writer) Close

func (w *Writer) Close() (name string, size int64)

func (*Writer) GetChan

func (w *Writer) GetChan() <-chan []byte

GetChan returns a channel for receiving bytes

func (*Writer) IsCSV

func (w *Writer) IsCSV() bool

func (*Writer) Write

func (w *Writer) Write(msg proto.Message) error

func (*Writer) WriteCSV

func (w *Writer) WriteCSV(msg proto.Message) (int, error)

WriteCSV writes a csv record

func (*Writer) WriteCSVHeader

func (w *Writer) WriteCSVHeader(msg proto.Message) (int, error)

WriteCSVHeader writes a CSV record

func (*Writer) WriteHeader

func (w *Writer) WriteHeader(t types.Type, source string, version string, includesPayloads bool) error

func (*Writer) WriteProto

func (w *Writer) WriteProto(msg proto.Message) error

WriteProto writes a protobuf message