nmsg

package module
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 27, 2023 License: MPL-2.0 Imports: 22 Imported by: 3

README

Pure Golang NMSG Library

go-nmsg is a pure go implementation of the NMSG container and payload format used by the C nmsg toolkit and library.

Synopsis

import "github.com/farsightsec/go-nmsg"
import "github.com/farsightsec/go-nmsg/nmsg_base"

var r io.Reader
var w io.Writer
...
input := nmsg.NewInput(r, mtu)
output := nmsg.BufferedOutput(w)
output.SetMaxSize(nmsg.MaxContainerSize, 0)

for {
	payload, err := input.Recv()
	if err != nil {
		if nmsg.IsDataError(err) {
			continue
		}
		break
	}
	message := payload.Message()

	switch message.(type) {
	case *nmsg_base.Dnstap:
		// process dnstap
		// write copy to output
		output.Send(payload)
	}
}

output.Close()

Requirements

go-nmsg requires the following open source libraries:

"google.golang.org/protobuf/proto"
"github.com/dnstap/golang-dnstap"

Limitations

go-nmsg can pack and unpack the protobuf structure of an NMSG payload, and the protobuf structure of the data contained in the payload. It does not implement the full functionality of the C libnmsg message modules, such as:

  • Advanced field types (e.g., a protobuf []byte as an IP address)
  • Generated fields
  • Formatting of fields for presentation and JSON output

Applications needing such functionality in go should use the cgo-nmsg package included in this distribution under:

"github.com/farsightsec/go-nmsg/cgo-nmsg"

Documentation

Index

Constants

View Source 
const (
	MinContainerSize   = 512
	MaxContainerSize   = 1048576
	EtherContainerSize = 1280
)

Container size limits to avoid silly fragmentation and memory exhaustion.

Variables

View Source 
var File_nmsg_proto protoreflect.FileDescriptor

Functions

func IsDataError 

func IsDataError(err error) bool

IsDataError returns true of the supplied error is an error unpacking or decoding the NMSG data rather than an I/O error with the input.

func MessageTypeByName added in v0.3.0 

func MessageTypeByName(vname string, mname string) (vid uint32, mtype uint32, err error)

MessageTypeByName returns the numeric vendor id and message type for the given vendor name and message type name, for the purposes of creating a new message with NewMessage().

func MessageTypeName added in v0.3.0 

func MessageTypeName(vid uint32, msgtype uint32) (vname string, mname string, err error)

MessageTypeName returns a vendor and message type name for a given numeric vendor id and message type, if any

func Register 

func Register(m Message)

Register records the supplied message's type, indexed by its MessageType and VendorID, for the purposes of decoding protobuf-encoded payloads.

Register should be called from the init() function of the module defining the payload type. It is not safe to call from multiple goroutines, and may not be called if any goroutine is concurrently decoding NMSG payloads.

func RegisterVendor added in v0.3.0 

func RegisterVendor(vname string, vid uint32)

RegisterVendor records an association between the vendor named `vname` and a numeric vendor id `vid`

func VendorByName added in v0.3.0 

func VendorByName(vname string) (uint32, error)

VendorByname returns the numeric vendor id registered for the given name, if any.

func VendorName added in v0.3.0 

func VendorName(vid uint32) (string, error)

VendorName returns the vendor name registered for the given numeric vid, if any.

Types

type Container 

type Container struct {
	Nmsg
	*NmsgFragment
	// contains filtered or unexported fields
}

A Container encapsulates an Nmsg envelope, and maintains metadata for sizing containers as payloads are added.

func NewContainer 

func NewContainer() *Container

NewContainer creates a new empty NMSG container.

func (*Container) AddPayload 

func (c *Container) AddPayload(p *NmsgPayload) (ok, full bool)

AddPayload adds the supplied NmsgPayload to the Container if possible.

The return value 'full' is true if the container is full and needs to be emptied with WriteTo().

The return value 'ok' is true if the payload was successfully added to the container, otherwise, AddPayload() must be called again after WriteTo().

Both ok and full may be true if the payload is larger than the container's MaxSize, or if the container is full after adding the payload.

func (*Container) FromBytes 

func (c *Container) FromBytes(b []byte) error

FromBytes parses the given buffer as an NMSG container and stores the result in the receiver *Container.

func (*Container) ReadFrom 

func (c *Container) ReadFrom(r io.Reader) (n int64, err error)

ReadFrom Reads a Container from the given io.Reader. It returns the number of container bytes read on success.

func (*Container) Reset 

func (c *Container) Reset()

Reset discards payloads and crcs from the Container

func (*Container) SetCompression 

func (c *Container) SetCompression(compress bool)

SetCompression instructs WriteTo to write containers with compressed (if true) or uncompressed (if false) contents.

func (*Container) SetCompressionRatio 

func (c *Container) SetCompressionRatio(ratio float32)

SetCompressionRatio sets an estimated compression ratio for the data. The default value is 2.0

func (*Container) SetMaxSize 

func (c *Container) SetMaxSize(size, writeSize int)

SetMaxSize sets the maximum size (including Marshaling overhead, container header, and anticipated compression ratio) of a container. AddPayload attempts to keep the container within this size.

writeSize specifies the maximum size of containers or fragments. Containers larger than writeSize will be written as fragments instead of single containers.

A writeSize value of 0 is treated as equal to size.

func (*Container) SetSequenced 

func (c *Container) SetSequenced(sequenced bool)

SetSequenced sets or unsets sequencing on the container stream. The sequence number is updated every time WriteTo() is called.

func (*Container) WriteTo 

func (c *Container) WriteTo(w io.Writer) (int64, error)

WriteTo writes the Container to Writer w. If the container requires fragmentation, it will call w.Write() multiple times.

type Input 

type Input interface {
	// Recv() returns the next Nmsg Payload from the input,
	// blocking if none is available.
	Recv() (*NmsgPayload, error)
	// Stats() returns interface statistics
	Stats() *InputStatistics
}

An Input is a source of NMSG Payloads.

func NewInput 

func NewInput(r io.Reader, size int) Input

NewInput constructs an input from the supplied Reader. The size parameter sizes the input buffer, and should be greater than the maximum anticipated container size for datagram inputs.

type InputStatistics 

type InputStatistics struct {
	// Count of total container received, including fragments
	InputContainers uint64
	// Count of total bytes received and processed
	InputBytes uint64
	// Count of containers marked lost by sequence tracking
	LostContainers uint64
	// Count of fragment containers received
	InputFragments uint64
	// Count of fragments expired from cache
	ExpiredFragments uint64
	// Count of containers dropped due to incomplete fragments
	PartialContainers uint64
}

InputStatistics holds useful metrics for input performance.

type Message 

type Message interface {
	proto.Message
	GetVid() uint32
	GetMsgtype() uint32
}

Message encapsulates a protobuf-encoded payload.

The values returned by the GetVid() and GetMsgtype() methods return identify the format of the payload.

func NewMessage 

func NewMessage(vid, msgtype uint32) (Message, error)

NewMessage creates a new Message with an underlying type identified by vid, msgtype.

type Nmsg 

type Nmsg struct {
	Payloads    []*NmsgPayload `protobuf:"bytes,1,rep,name=payloads" json:"payloads,omitempty"`
	PayloadCrcs []uint32       `protobuf:"varint,2,rep,name=payload_crcs,json=payloadCrcs" json:"payload_crcs,omitempty"`
	Sequence    *uint32        `protobuf:"varint,3,opt,name=sequence" json:"sequence,omitempty"`
	SequenceId  *uint64        `protobuf:"varint,4,opt,name=sequence_id,json=sequenceId" json:"sequence_id,omitempty"`
	// contains filtered or unexported fields
}

func (*Nmsg) Descriptor deprecated 

func (*Nmsg) Descriptor() ([]byte, []int)

Deprecated: Use Nmsg.ProtoReflect.Descriptor instead.

func (*Nmsg) GetPayloadCrcs 

func (x *Nmsg) GetPayloadCrcs() []uint32

func (*Nmsg) GetPayloads 

func (x *Nmsg) GetPayloads() []*NmsgPayload

func (*Nmsg) GetSequence 

func (x *Nmsg) GetSequence() uint32

func (*Nmsg) GetSequenceId 

func (x *Nmsg) GetSequenceId() uint64

func (*Nmsg) ProtoMessage 

func (*Nmsg) ProtoMessage()

func (*Nmsg) ProtoReflect 

func (x *Nmsg) ProtoReflect() protoreflect.Message

func (*Nmsg) Reset 

func (x *Nmsg) Reset()

func (*Nmsg) String 

func (x *Nmsg) String() string

type NmsgFragment 

type NmsgFragment struct {
	Id       *uint32 `protobuf:"varint,1,req,name=id" json:"id,omitempty"`
	Current  *uint32 `protobuf:"varint,2,req,name=current" json:"current,omitempty"`
	Last     *uint32 `protobuf:"varint,3,req,name=last" json:"last,omitempty"`
	Fragment []byte  `protobuf:"bytes,4,req,name=fragment" json:"fragment,omitempty"`
	Crc      *uint32 `protobuf:"varint,5,opt,name=crc" json:"crc,omitempty"`
	// contains filtered or unexported fields
}

func (*NmsgFragment) Descriptor deprecated 

func (*NmsgFragment) Descriptor() ([]byte, []int)

Deprecated: Use NmsgFragment.ProtoReflect.Descriptor instead.

func (*NmsgFragment) GetCrc 

func (x *NmsgFragment) GetCrc() uint32

func (*NmsgFragment) GetCurrent 

func (x *NmsgFragment) GetCurrent() uint32

func (*NmsgFragment) GetFragment 

func (x *NmsgFragment) GetFragment() []byte

func (*NmsgFragment) GetId 

func (x *NmsgFragment) GetId() uint32

func (*NmsgFragment) GetLast 

func (x *NmsgFragment) GetLast() uint32

func (*NmsgFragment) ProtoMessage 

func (*NmsgFragment) ProtoMessage()

func (*NmsgFragment) ProtoReflect 

func (x *NmsgFragment) ProtoReflect() protoreflect.Message

func (*NmsgFragment) Reset 

func (x *NmsgFragment) Reset()

func (*NmsgFragment) String 

func (x *NmsgFragment) String() string

type NmsgPayload 

type NmsgPayload struct {
	Vid      *uint32 `protobuf:"varint,1,req,name=vid" json:"vid,omitempty"`
	Msgtype  *uint32 `protobuf:"varint,2,req,name=msgtype" json:"msgtype,omitempty"`
	TimeSec  *int64  `protobuf:"varint,3,req,name=time_sec,json=timeSec" json:"time_sec,omitempty"`
	TimeNsec *uint32 `protobuf:"fixed32,4,req,name=time_nsec,json=timeNsec" json:"time_nsec,omitempty"`
	Payload  []byte  `protobuf:"bytes,5,opt,name=payload" json:"payload,omitempty"`
	Source   *uint32 `protobuf:"varint,7,opt,name=source" json:"source,omitempty"`
	Operator *uint32 `protobuf:"varint,8,opt,name=operator" json:"operator,omitempty"`
	Group    *uint32 `protobuf:"varint,9,opt,name=group" json:"group,omitempty"`
	// contains filtered or unexported fields
}

func Payload 

func Payload(m Message) (*NmsgPayload, error)

Payload encapsulates an nmsg message in a NmsgPayload, suitable for sending to an Output

func (*NmsgPayload) Descriptor deprecated 

func (*NmsgPayload) Descriptor() ([]byte, []int)

Deprecated: Use NmsgPayload.ProtoReflect.Descriptor instead.

func (*NmsgPayload) GetGroup 

func (x *NmsgPayload) GetGroup() uint32

func (*NmsgPayload) GetMsgtype 

func (x *NmsgPayload) GetMsgtype() uint32

func (*NmsgPayload) GetOperator 

func (x *NmsgPayload) GetOperator() uint32

func (*NmsgPayload) GetPayload 

func (x *NmsgPayload) GetPayload() []byte

func (*NmsgPayload) GetSource 

func (x *NmsgPayload) GetSource() uint32

func (*NmsgPayload) GetTimeNsec 

func (x *NmsgPayload) GetTimeNsec() uint32

func (*NmsgPayload) GetTimeSec 

func (x *NmsgPayload) GetTimeSec() int64

func (*NmsgPayload) GetVid 

func (x *NmsgPayload) GetVid() uint32

func (*NmsgPayload) Message 

func (p *NmsgPayload) Message() (Message, error)

Message returns the message encapsulated in the NmsgPayload, Unmarshaled

func (*NmsgPayload) ProtoMessage 

func (*NmsgPayload) ProtoMessage()

func (*NmsgPayload) ProtoReflect 

func (x *NmsgPayload) ProtoReflect() protoreflect.Message

func (*NmsgPayload) Reset 

func (x *NmsgPayload) Reset()

func (*NmsgPayload) SetGroup 

func (p *NmsgPayload) SetGroup(g uint32)

SetGroup sets the NmsgPayload group identifier.

func (*NmsgPayload) SetOperator 

func (p *NmsgPayload) SetOperator(o uint32)

SetOperator sets the NmsgPayload operator identifier.

func (*NmsgPayload) SetSource 

func (p *NmsgPayload) SetSource(s uint32)

SetSource sets the NmsgPayload source identifier.

func (*NmsgPayload) String 

func (x *NmsgPayload) String() string

type Output 

type Output interface {
	// Send sends an Nmsg Payload along the output. Implementations
	// may queue the payload for later sending, so the caller should
	// not modify the payload after calling Send.
	//
	// Send may be safely called from multiple goroutines.
	Send(*NmsgPayload) error
	// SetSequenced controls whether the Nmsg containers generated by
	// the Output have sequence numbers.
	SetSequenced(bool)
	// SetCompression controls whether the Output generates compressed
	// containers or uncompressed.
	SetCompression(bool)
	// SetCompressionRatio sets the anticipated compression ratio for
	// compressed containers.
	SetCompressionRatio(float32)
	// SetMaxSize sets the maximum size of a container the Output will
	// buffer, and the maximum size of a container or fragment the Output
	// will write. For Ethernet, consider using nmsg.EtherContainerSize.
	SetMaxSize(size int, writeSize int)
	// Close shuts down the output, flushing any queued payloads.
	// It will not close the underlying io.Writer.
	Close() error
}

An Output encapsulates NmsgPayloads in Nmsg containers and writes them to an io.Writer.

func BufferedOutput 

func BufferedOutput(w io.Writer) Output

BufferedOutput creates an Output which collects NmsgPayloads and sends them in containers as close as possible to the size set by SetMaxSize()

func TimedBufferedOutput 

func TimedBufferedOutput(w io.Writer, d time.Duration) Output

TimedBufferedOutput creates an Output which collects NmsgPayloads and sends them in containers as close as possible to the size provided to SetMaxSize or after the given Duration, whichever comes first.

func UnbufferedOutput 

func UnbufferedOutput(w io.Writer) Output

UnbufferedOutput returns an Output which writes an Nmsg container for each payload.

type Sockspec 

type Sockspec struct {
	Addr   *net.UDPAddr
	Hiport int
}

A Sockspec is an address of a single socket (addr/port) or a series of sockets with contiguous port numbers (addr/loport..hiport)

func ParseSockspec 

func ParseSockspec(v string) (*Sockspec, error)

ParseSockspec creates a Sockspec from its text representaion v.

func (*Sockspec) Addrs 

func (s *Sockspec) Addrs() []*net.UDPAddr

Addrs returns the list of UDP socket addresses of the Sockspec, or nil if the Sockspec is uninitialized.

func (*Sockspec) Set 

func (s *Sockspec) Set(v string) error

Set initializes a Sockspec from its text representation v. Set satisfies flag.Value allowing a sockspec to be conveniently specified as a command line parameter.

func (*Sockspec) String 

func (s *Sockspec) String() string

String returns the string representation of the Sockspec. If the Sockspec is uninitialized, String returns the empty string.

func (*Sockspec) UnmarshalJSON 

func (s *Sockspec) UnmarshalJSON(b []byte) error

UnmarshalJSON satisifies json.Unmarshaler allowing Sockspecs to be parsed from JSON configurations.

func (*Sockspec) UnmarshalYAML 

func (s *Sockspec) UnmarshalYAML(u func(interface{}) error) error

UnmarshalYAML satisifies yaml.Unmarshaler allowing Sockspecs to be parsed from YAML configurations.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.