gopcap

README

GOPCAP

This package provides support for reading Packet Capture (PCAP) files efficiently in pure Go and provides benchmarks against other packages offering the same functionality.

Update: I just discovered the go implementation from gopacket https://github.com/google/gopacket/tree/master/pcapgo, which is comparatively fast and has support for both reading and writing PCAP + PCAP-NG and live capture in pure Go (awesome!). I've added the gopacket/pcapgo implementation to the benchmarks as well. I recommend to use the pcapgo library instead of this one, it offers far more functionality.

However, in my benchmarks this implementation still is slightly faster. So, if performance is crucial and you only need to read PCAP, this lib might still be interesting.

The problem:

Existing implementations often use binary.Read for parsing the binary data into the expected structure. However, the function makes use of reflection, which hurts performance.

This implementation does not use binary.Read to parse the file and packet headers, and therefore is much faster.

The following API is exported by the package:

// Count all packet inside the PCAP file at path
func Count(path string) int64

// Reader for PCAP
type Reader

// Open file at path and return reader and error
func Open(filename string) (*Reader, error)

// Close reader
func (r *Reader) Close() error

// ReadNextPacket reads the next packet and returns header,data,error
func (r *Reader) ReadNextPacket() (PacketHeader, []byte, error)

// ReadNextPacketHeader reads the next packet header and returns header,data,error
func (r *Reader) ReadNextPacketHeader() (PacketHeader, []byte, error)

Data structures for the PCAP file header and packet headers are provided:

type FileHeader struct {
    // magic number
    MagicNumber uint32
    // major version number
    VersionMajor uint16
    // minor version number
    VersionMinor uint16
    // GMT to local correction
    Thiszone int32
    // accuracy of timestamps
    Sigfigs uint32
    // max length of captured packets, in octets
    Snaplen uint32
    // data link type
    Network uint32
}

type PacketHeader struct {
    // timestamp seconds
    TsSec int32
    // timestamp microseconds
    TsUsec int32
    // number of octets of packet saved in file
    CaptureLen int32
    // actual length of packet
    OriginalLen int32
}

Usage

Reading a PCAP file from disk can be done by calling gopcap.Open(path) and looping on r.ReadNextPacket() of the returned reader. Additionally there is a function called gopcap.Count(path) that returns the total count of packets in the file (useful for displaying progress).

// get total packet count
fmt.Println("total:", gopcap.Count(path))

// create reader
r, err := gopcap.Open(path)
if err != nil {
    panic(err)
}
defer r.Close()

// loop over packets
for {
    h, data, err := r.ReadNextPacket()
    if err != nil {
        if err == io.EOF {
            println("EOF")
            break
        }
        panic(err)
    }
    fmt.Println(h, len(data))
}

Benchmarks

There are a few pure go implementations for parsing PCAP files available. The following have not been evaluated in the benchmark, for the given reason(s):

• https://github.com/davecheney/pcap, Latest commit 10760a1 on Aug 19, 2012, fails to compile with various errors
• https://github.com/Lukasa/gopcap, limited in functionality, API only allows to parse a file completely, which is unpractical for big files.

These implementation are included in the benchmarks, in the order they are listed:

• https://github.com/dreadl0ck/gopcap (this package)
• https://github.com/github.com/0intro/pcap
• https://github.com/google/gopacket/pcap
• https://godoc.org/go.universe.tf/netboot
• https://github.com/github.com/miekg/pcap
• https://github.com/google/gopacket/tree/master/pcapgo

The benchmark code fetches a single packet in a loop, and discards all data that is not needed. Make sure the PCAP file for the test is big enough, otherwise the tests wont produce meaningful results.

I didn't include the dump in the repo because it is around 1.0G in size. The used PCAP file (maccdc2012_00000.pcap) is from the National CyberWatch Mid-Atlantic Collegiate Cyber Defense Competition (MACCDC), which can be downloaded here: https://www.netresec.com/?page=MACCDC

Here are the results of a few runs on my development machine (MacBook Pro 2018, 32GB RAM, 2,9 GHz Intel Core i9):

$ go test -bench=.
goos: darwin
goarch: amd64
pkg: github.com/dreadl0ck/gopcap
BenchmarkReadPcap-12                  	10000000	       142 ns/op	     111 B/op	       1 allocs/op
BenchmarkReadPcap0Intro-12            	  300000	      4714 ns/op	     203 B/op	       8 allocs/op
BenchmarkReadPcapGoPacket-12          	 5000000	       258 ns/op	       0 B/op	       0 allocs/op
BenchmarkReadPcapNetboot-12           	 3000000	       510 ns/op	     235 B/op	       8 allocs/op
BenchmarkReadPcapMiekg-12             	 3000000	       582 ns/op	     283 B/op	       6 allocs/op
BenchmarkReadPcapGopacketPcapGo-12    	10000000	       149 ns/op	      98 B/op	       0 allocs/op
PASS
ok  	github.com/dreadl0ck/gopcap	10.555s

$ go test -bench=.
goos: darwin
goarch: amd64
pkg: github.com/dreadl0ck/gopcap
BenchmarkReadPcap-12                  	10000000	       193 ns/op	     111 B/op	       1 allocs/op
BenchmarkReadPcap0Intro-12            	  200000	      6181 ns/op	     224 B/op	       8 allocs/op
BenchmarkReadPcapGoPacket-12          	 5000000	       356 ns/op	       0 B/op	       0 allocs/op
BenchmarkReadPcapNetboot-12           	 2000000	       653 ns/op	     223 B/op	       8 allocs/op
BenchmarkReadPcapMiekg-12             	 2000000	       737 ns/op	     271 B/op	       6 allocs/op
BenchmarkReadPcapGopacketPcapGo-12    	10000000	       200 ns/op	      98 B/op	       0 allocs/op
PASS
ok  	github.com/dreadl0ck/gopcap	11.911s

$ go test -bench=.
goos: darwin
goarch: amd64
pkg: github.com/dreadl0ck/gopcap
BenchmarkReadPcap-12                  	10000000	       145 ns/op	     111 B/op	       1 allocs/op
BenchmarkReadPcap0Intro-12            	  300000	      4800 ns/op	     203 B/op	       8 allocs/op
BenchmarkReadPcapGoPacket-12          	 5000000	       262 ns/op	       0 B/op	       0 allocs/op
BenchmarkReadPcapNetboot-12           	 3000000	       664 ns/op	     235 B/op	       8 allocs/op
BenchmarkReadPcapMiekg-12             	 2000000	       732 ns/op	     271 B/op	       6 allocs/op
BenchmarkReadPcapGopacketPcapGo-12    	10000000	       197 ns/op	      98 B/op	       0 allocs/op
PASS
ok  	github.com/dreadl0ck/gopcap	11.557s

$ go test -bench=.
goos: darwin
goarch: amd64
pkg: github.com/dreadl0ck/gopcap
BenchmarkReadPcap-12                  	10000000	       172 ns/op	     125 B/op	       2 allocs/op
BenchmarkReadPcap0Intro-12            	  300000	      4175 ns/op	     203 B/op	       8 allocs/op
BenchmarkReadPcapGoPacket-12          	 5000000	       275 ns/op	       0 B/op	       0 allocs/op
BenchmarkReadPcapNetboot-12           	 3000000	       659 ns/op	     235 B/op	       8 allocs/op
BenchmarkReadPcapMiekg-12             	 2000000	       560 ns/op	     271 B/op	       6 allocs/op
BenchmarkReadPcapGopacketPcapGo-12    	10000000	       194 ns/op	      98 B/op	       0 allocs/op
PASS
ok  	github.com/dreadl0ck/gopcap	11.370s

$ go test -bench=.
goos: darwin
goarch: amd64
pkg: github.com/dreadl0ck/gopcap
BenchmarkReadPcap-12                  	10000000	       196 ns/op	     111 B/op	       1 allocs/op
BenchmarkReadPcap0Intro-12            	  300000	      4821 ns/op	     203 B/op	       8 allocs/op
BenchmarkReadPcapGoPacket-12          	 3000000	       436 ns/op	       0 B/op	       0 allocs/op
BenchmarkReadPcapNetboot-12           	 2000000	       766 ns/op	     223 B/op	       8 allocs/op
BenchmarkReadPcapMiekg-12             	 2000000	       855 ns/op	     271 B/op	       6 allocs/op
BenchmarkReadPcapGopacketPcapGo-12    	 5000000	       278 ns/op	     104 B/op	       1 allocs/op
PASS
ok  	github.com/dreadl0ck/gopcap	11.904s

Results vary, but this implementation appears to be slightly faster than gopacket/pcapgo. Can anyone explain why the amount of allocations varies in between different runs? How can 0 allocs/op happen?

The gopacket pcap library and fork from miekg both use C bindings. For the benchmark of the gopacket pcap implementation, the ZeroCopyReadPacketData() function was used.

Documentation

Index

• func Count(path string) (int64, error)
• type FileHeader
• type PacketHeader
• type Reader
  • func Open(filename string) (*Reader, error)
  • func (r *Reader) Close() error
  • func (r *Reader) ReadNextPacket() (PacketHeader, []byte, error)
  • func (r *Reader) ReadNextPacketHeader() (PacketHeader, []byte, error)

Functions

func Count

func Count(path string) (int64, error)

Count packets in file TODO: dont actually read anything... just seek to next offset

Types

type FileHeader

type FileHeader struct {
	// magic number
	MagicNumber uint32 // 0
	// major version number
	VersionMajor uint16 // 4
	// minor version number
	VersionMinor uint16 // 6
	// GMT to local correction
	Thiszone int32 // 8
	// accuracy of timestamps
	Sigfigs uint32 // 12
	// max length of captured packets, in octets
	Snaplen uint32 // 16
	// data link type
	Network uint32 // 20

} // 24

FileHeader global PCAP file header for more info: https://wiki.wireshark.org/Development/LibpcapFileFormat

type PacketHeader

type PacketHeader struct {
	// timestamp seconds
	TsSec int32 // 0
	// timestamp microseconds
	TsUsec int32 // 4
	// number of octets of packet saved in file
	CaptureLen int32 // 8
	// actual length of packet
	OriginalLen int32 // 12

} // 16

PacketHeader is a PCAP packet header

type Reader

type Reader struct {
	FileHandle *os.File
	Buffer     *bufio.Reader
	Header     FileHeader
}

Reader struct

func Open

func Open(filename string) (*Reader, error)

Open pcap file

func (*Reader) Close

func (r *Reader) Close() error

Close pcap file

func (*Reader) ReadNextPacket

func (r *Reader) ReadNextPacket() (PacketHeader, []byte, error)

ReadNextPacket reads the next packet. returns header,data,error

func (*Reader) ReadNextPacketHeader

func (r *Reader) ReadNextPacketHeader() (PacketHeader, []byte, error)

ReadNextPacketHeader read next packet header. returns header,data,error @TODO: add a bytes.Reader after the buffered reader, and seek inside that buffer instead of reading all the bytes.