goDB

package
v2.1.1+incompatible Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 22, 2018 License: GPL-2.0 Imports: 18 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	EPOCH_DAY         int64 = 86400 // one day in seconds
	DB_WRITE_INTERVAL int64 = 300   // write out interval of capture probe
)
View Source 
const (
	BUF_SIZE = 4096         // 512 * 64bit
	N_ELEM   = BUF_SIZE / 8 // 512
)
View Source 
const (
	// First the attribute columns...
	SIP_COLIDX, _ columnIndex = iota, iota
	DIP_COLIDX, _
	PROTO_COLIDX, _
	DPORT_COLIDX, _
	// ... and then the columns we aggregate
	BYTESRCVD_COLIDX, COLIDX_ATTRIBUTE_COUNT
	BYTESSENT_COLIDX, _
	PKTSRCVD_COLIDX, _
	PKTSSENT_COLIDX, _
	COLIDX_COUNT, _
)

Indizes for all column types

View Source 
const (
	SIP_SIZEOF       int = 16
	DIP_SIZEOF       int = 16
	PROTO_SIZEOF     int = 1
	DPORT_SIZEOF     int = 2
	BYTESRCVD_SIZEOF int = 8
	BYTESSENT_SIZEOF int = 8
	PKTSRCVD_SIZEOF  int = 8
	PKTSSENT_SIZEOF  int = 8
)

Sizeof (entry) for all column types

View Source 
const (
	SUMMARY_FILE_NAME      = "summary.json"
	SUMMARY_LOCK_FILE_NAME = "summary.lock"
)
View Source 
const (
	// Used for compression applied by GPFile.
	COMPRESSION_LEVEL = 512

	METADATA_FILE_NAME = "meta.json"
)
View Source 
const SLOG_ADDR = "127.0.0.1"
View Source 
const SLOG_PORT = "514"
View Source 
const (
	SOCKET_PATH = "/var/run/goprobe.sock"
)

Variables

View Source 
var SysLog *syslog.Writer
View Source 
var TimeFormats []string = []string{
	time.ANSIC,
	time.RubyDate,
	time.RFC822Z,
	time.RFC1123Z,
	time.RFC3339,

	"2006-01-02 15:04:05 -0700",
	"2006-01-02 15:04:05",
	"2006-01-02 15:04 -0700",
	"2006-01-02 15:04",
	"02.01.2006 15:04",
	"02.01.2006 15:04 -0700",
	"02.01.06 15:04",
	"02.01.06 15:04 -0700",
	"2.1.06 15:04:05",
	"2.1.06 15:04:05 -0700",
	"2.1.06 15:04",
	"2.1.06 15:04 -0700",
	"2.1.2006 15:04:05",
	"2.1.2006 15:04:05 -0700",
	"2.1.2006 15:04",
	"2.1.2006 15:04 -0700",
	"02.1.2006 15:04:05",
	"02.1.2006 15:04:05 -0700",
	"02.1.2006 15:04",
	"02.1.2006 15:04 -0700",
	"2.01.2006 15:04:05",
	"2.01.2006 15:04:05 -0700",
	"2.01.2006 15:04",
	"2.01.2006 15:04 -0700",
	"02.1.06 15:04:05",
	"02.1.06 15:04:05 -0700",
	"02.1.06 15:04",
	"02.1.06 15:04 -0700",
	"2.01.06 15:04:05",
	"2.01.06 15:04:05 -0700",
	"2.01.06 15:04",
	"2.01.06 15:04 -0700"}

Utility variables and functions for time parsing -----------------------------

Functions

func DayTimestamp 

func DayTimestamp(timestamp int64) int64

DayTimestamp returns timestamp rounded down to the nearest day

func HasDNSAttributes 

func HasDNSAttributes(attributes []Attribute) bool

Find out if any of the attributes are usable for a reverse DNS lookup (e.g. check for IP attributes)

func IPStringToBytes 

func IPStringToBytes(ip string) ([]byte, error)

Condition conversion utility functions ------------------------------------------------

func InitDBLog 

func InitDBLog() error

func LockDBSummary 

func LockDBSummary(dbpath string) (acquired bool, err error)

LockDBSummary tries to acquire a lockfile for the database summary. Its return values indicate whether it successfully acquired the lock and whether a file system error occurred.

func ModifyDBSummary 

func ModifyDBSummary(dbpath string, timeout time.Duration, modify func(*DBSummary) (*DBSummary, error)) (modErr error)

Safely modifies the database summary when there are multiple processes accessing it.

If no lock can be acquired after (roughly) timeout time, returns an error.

modify is expected to obey the following contract:

  • The input summary is nil if no summary file is present.
  • modify returns the summary to be written (must be non-nil) and an error.
  • Since the summary is locked while modify is running, modify shouldn't take longer than roughly half a second.

func ParseTimeArgument 

func ParseTimeArgument(timeString string) (int64, error)

Entry point for external calls -------------------------------------------------

func SanitizeUserInput 

func SanitizeUserInput(conditional string) (string, error)

SanitizeUserInput sanitizes a conditional string provided by the user. Its main purpose is to convert other forms of precedence and logical operators to the condition grammar used. For example, some people may prefer a more verbose forms such as "dport=443 or dport=8080" or exotic forms such as "{dport=443 || dport=8080}". These should be caught and converted to the grammar-conforming expression "(dport=443|dport=8080)"

Input: 

conditional: string containing the conditional specified in "user grammar"

Output: 

string:  conditional string in the condition grammar. Note that this may still
         include syntactical errors or malspecified conditions. These will be caught
         at a latter stage
error:   any error from golang's regex module

NOTE: the current implementation of GPDPIProtocols.go has to make sure that the map keys 

of "proto" to numbers are all lower case

func TokenizeConditional 

func TokenizeConditional(condExpression string) ([]string, error)

TokenizeConditional tokenizes the given conditional. Note that the tokenization is "loose": All valid conditionals will be correctly tokenized, but there are invalid conditionals that will also be tokenized. Its the parser's job to catch those. Whitespace in conditionals is only useful for tokenization and not needed afterwards. TokenizeConditional doesn't emit any whitespace tokens.

Limitations: Only ASCII is supported. May give strange results on fancy Unicode strings.

func UnlockDBSummary 

func UnlockDBSummary(dbpath string) (err error)

LockDBSummary removes the lockfile for the database summary. Its return values indicates whether a file system error occurred.

func WriteDBSummary 

func WriteDBSummary(dbpath string, summ *DBSummary) error

Writes a new summary for the given database. If multiple processes might be operating on the summary simultaneously, you should lock it first.

func WriteMetadata 

func WriteMetadata(path string, meta *Metadata) error

Types

type AggFlowMap 

type AggFlowMap map[Key]*Val

type Attribute 

type Attribute interface {
	Name() string
	// Some attributes use more than a single output column.
	ExtraColumns() []string
	// ExtractStrings() extracts a list of records representing the
	// attribute from a given key.
	// You may assume that the length of the returned list is always
	// the length of ExtraColumns() + 1.
	ExtractStrings(key *ExtraKey) []string
	// contains filtered or unexported methods
}

Interface for attributes This interface is not meant to be implemented by structs outside this package

func NewAttribute 

func NewAttribute(name string) (Attribute, error)

Returns an Attribute for the given name. If no such attribute exists, an error is returned.

func ParseQueryType 

func ParseQueryType(queryType string) (attributes []Attribute, hasAttrTime, hasAttrIface bool, err error)

Parses the given query type into a list of attributes. The returned list is guaranteed to have no duplicates. A valid query type can either be a comma-separated list of attribute names (e.g. "sip,dip,dport") or something like "talk_conv". The return variable hasAttrTime indicates whether the special time attribute is present. (time is never a part of the returned attribute list.) The time attribute is present for the query type 'raw', or if it is explicitly mentioned in a list of attribute names.

type BlockMetadata 

type BlockMetadata struct {
	Timestamp            int64 `json:"timestamp"`
	PcapPacketsReceived  int   `json:"pcap_packets_received"`
	PcapPacketsDropped   int   `json:"pcap_packets_dropped"`
	PcapPacketsIfDropped int   `json:"pcap_packets_if_dropped"`
	PacketsLogged        int   `json:"packets_logged"`

	// As in Summary
	FlowCount uint64 `json:"flowcount"`
	Traffic   uint64 `json:"traffic"`
}

Represents metadata for one database block.

type BytesRecStringParser 

type BytesRecStringParser struct{}

value parsers

func (*BytesRecStringParser) ParseVal 

func (b *BytesRecStringParser) ParseVal(element string, val *Val) error

type BytesSentStringParser 

type BytesSentStringParser struct{}

func (*BytesSentStringParser) ParseVal 

func (b *BytesSentStringParser) ParseVal(element string, val *Val) error

type DBData 

type DBData struct {
	// counters
	Bytes_rcvd []byte
	Bytes_sent []byte
	Pkts_rcvd  []byte
	Pkts_sent  []byte

	// attributes
	Dip   []byte
	Sip   []byte
	Dport []byte
	Proto []byte

	// metadata (important for folder naming)
	Tstamp int64
	Iface  string
}

func NewDBData 

func NewDBData(br []byte, bs []byte, pr []byte, ps []byte, dip []byte, sip []byte, dport []byte, proto []byte, tstamp int64, iface string) DBData

constructor for the DBData struct in case it needs to be set from an external go program that included goProbe

type DBLog 

type DBLog struct {
	Log *syslog.Writer
}

type DBSummary 

type DBSummary struct {
	Interfaces map[string]InterfaceSummary `json:"interfaces"`
}

Summary for an entire database

func NewDBSummary 

func NewDBSummary() *DBSummary

func ReadDBSummary 

func ReadDBSummary(dbpath string) (*DBSummary, error)

Reads the summary of the given database. If multiple processes might be operating on the summary simultaneously, you should lock it first.

func (*DBSummary) Update 

func (s *DBSummary) Update(u InterfaceSummaryUpdate)

type DBWorkManager 

type DBWorkManager struct {
	// contains filtered or unexported fields
}

func NewDBWorkManager 

func NewDBWorkManager(dbpath string, iface string, numProcessingUnits int) (*DBWorkManager, error)

func (*DBWorkManager) CreateWorkerJobs 

func (w *DBWorkManager) CreateWorkerJobs(tfirst int64, tlast int64, query *Query) (nonempty bool, err error)

func (*DBWorkManager) ExecuteWorkerReadJobs 

func (w *DBWorkManager) ExecuteWorkerReadJobs(mapChan chan map[ExtraKey]Val)

Spawning of processing units and pushing of workload onto factory channel -----------

func (*DBWorkManager) GetCoveredTimeInterval 

func (w *DBWorkManager) GetCoveredTimeInterval() (time.Time, time.Time)

used to determine the time span actually covered by the query

func (*DBWorkManager) GetNumWorkers 

func (w *DBWorkManager) GetNumWorkers() int

make number of workloads available to the outside world for loop bounds etc.

type DBWorkload 

type DBWorkload struct {
	// contains filtered or unexported fields
}

type DBWriter 

type DBWriter struct {
	// contains filtered or unexported fields
}

func NewDBWriter 

func NewDBWriter(dbpath string, iface string) (w *DBWriter)

func (*DBWriter) Write 

func (w *DBWriter) Write(flowmap AggFlowMap, meta BlockMetadata, timestamp int64) (InterfaceSummaryUpdate, error)

type DipAttribute 

type DipAttribute struct{}

func (DipAttribute) ExtraColumns 

func (_ DipAttribute) ExtraColumns() []string

func (DipAttribute) ExtractStrings 

func (_ DipAttribute) ExtractStrings(key *ExtraKey) []string

func (DipAttribute) Name 

func (_ DipAttribute) Name() string

type DipStringParser 

type DipStringParser struct{}

func (*DipStringParser) ParseKey 

func (d *DipStringParser) ParseKey(element string, key *ExtraKey) error

type DportAttribute 

type DportAttribute struct{}

func (DportAttribute) ExtraColumns 

func (_ DportAttribute) ExtraColumns() []string

func (DportAttribute) ExtractStrings 

func (_ DportAttribute) ExtractStrings(key *ExtraKey) []string

func (DportAttribute) Name 

func (_ DportAttribute) Name() string

type DportStringParser 

type DportStringParser struct{}

func (*DportStringParser) ParseKey 

func (d *DportStringParser) ParseKey(element string, key *ExtraKey) error

type ExtraKey 

type ExtraKey struct {
	Time  int64
	Iface string
	Key
}

ExtraKey is a key with extra information

type GPFile 

type GPFile struct {
	// contains filtered or unexported fields
}

func NewGPFile 

func NewGPFile(p string) (*GPFile, error)

func (*GPFile) BlocksUsed 

func (f *GPFile) BlocksUsed() (int, error)

func (*GPFile) Close 

func (f *GPFile) Close() error

func (*GPFile) GetBlocks 

func (f *GPFile) GetBlocks() []int64

func (*GPFile) GetTimestamps 

func (f *GPFile) GetTimestamps() []int64

func (*GPFile) ReadBlock 

func (f *GPFile) ReadBlock(block int) ([]byte, error)

func (*GPFile) ReadTimedBlock 

func (f *GPFile) ReadTimedBlock(timestamp int64) ([]byte, error)

func (*GPFile) WriteTimedBlock 

func (f *GPFile) WriteTimedBlock(timestamp int64, data []byte, comp int) error

type GPFiler 

type GPFiler interface {
	BlocksUsed() (int, error)
	WriteTimedBlock(timestamp int64, data []byte) error
	ReadTimedBlock(timestamp int64) ([]byte, error)
	ReadBlock(block int) ([]byte, error)
	GetBlocks() ([]int64, error)
	GetTimestamps() ([]int64, error)
	Close() error
}

type IfaceStringParser 

type IfaceStringParser struct{}

func (*IfaceStringParser) ParseKey 

func (i *IfaceStringParser) ParseKey(element string, key *ExtraKey) error

type InterfaceSummary 

type InterfaceSummary struct {
	// Number of flows
	FlowCount uint64 `json:"flowcount"`
	// Total traffic volume in byte
	Traffic uint64 `json:"traffic"`
	Begin   int64  `json:"begin"`
	End     int64  `json:"end"`
}

Summary for a single interface

type InterfaceSummaryUpdate 

type InterfaceSummaryUpdate struct {
	// Name of the interface. For example, "eth0".
	Interface string
	// Number of flows
	FlowCount uint64
	// Traffic volume in bytes
	Traffic   uint64
	Timestamp time.Time
}

type Key 

type Key struct {
	Sip      [16]byte
	Dip      [16]byte
	Dport    [2]byte
	Protocol byte
}

func (Key) String 

func (k Key) String() string

print the key as a comma separated attribute list

type Metadata 

type Metadata struct {
	Blocks []BlockMetadata `json:"blocks"`
}

Metadata for a collection of database blocks. By convention all blocks belong the same day.

func NewMetadata 

func NewMetadata() *Metadata

func ReadMetadata 

func ReadMetadata(path string) (*Metadata, error)

Reads the given metadata file.

func TryReadMetadata 

func TryReadMetadata(path string) *Metadata

Tries to read the given metadata file. If an error occurs, a fresh Metadata struct is returned.

type NOPStringParser 

type NOPStringParser struct{}

func (*NOPStringParser) ParseKey 

func (n *NOPStringParser) ParseKey(element string, key *ExtraKey) error

The NOP parser doesn't do anything and just lets everything through which is not understandable by the other attribute parsers (e.g. the % field or any other field not mentioned above)

func (*NOPStringParser) ParseVal 

func (n *NOPStringParser) ParseVal(element string, val *Val) error

type Node 

type Node interface {
	fmt.Stringer
	// contains filtered or unexported methods
}

An AST node for the conditional grammar This interface is not meant to be implemented by structs outside of this package.

func ParseAndInstrumentConditional 

func ParseAndInstrumentConditional(conditional string, dnsTimeout time.Duration) (Node, error)

Parses and instruments the given conditional string for evaluation. This is the main external function related to conditionals.

type PacketsRecStringParser 

type PacketsRecStringParser struct{}

func (*PacketsRecStringParser) ParseVal 

func (p *PacketsRecStringParser) ParseVal(element string, val *Val) error

type PacketsSentStringParser 

type PacketsSentStringParser struct{}

func (*PacketsSentStringParser) ParseVal 

func (p *PacketsSentStringParser) ParseVal(element string, val *Val) error

type ProtoAttribute 

type ProtoAttribute struct{}

func (ProtoAttribute) ExtraColumns 

func (_ ProtoAttribute) ExtraColumns() []string

func (ProtoAttribute) ExtractStrings 

func (_ ProtoAttribute) ExtractStrings(key *ExtraKey) []string

func (ProtoAttribute) Name 

func (_ ProtoAttribute) Name() string

type ProtoStringParser 

type ProtoStringParser struct{}

func (*ProtoStringParser) ParseKey 

func (p *ProtoStringParser) ParseKey(element string, key *ExtraKey) error

type Query 

type Query struct {
	// list of attributes that will be compared, e.g. "dip" "sip"
	// in a "talk_conv" query
	Attributes  []Attribute
	Conditional Node
	// contains filtered or unexported fields
}

func NewQuery 

func NewQuery(attributes []Attribute, conditional Node, hasAttrTime, hasAttrIface bool) *Query

type SipAttribute 

type SipAttribute struct{}

func (SipAttribute) ExtraColumns 

func (_ SipAttribute) ExtraColumns() []string

func (SipAttribute) ExtractStrings 

func (_ SipAttribute) ExtractStrings(key *ExtraKey) []string

func (SipAttribute) Name 

func (_ SipAttribute) Name() string

type SipStringParser 

type SipStringParser struct{}

attribute parsers

func (*SipStringParser) ParseKey 

func (s *SipStringParser) ParseKey(element string, key *ExtraKey) error

type StringKeyParser 

type StringKeyParser interface {
	ParseKey(element string, key *ExtraKey) error
}

func NewStringKeyParser 

func NewStringKeyParser(kind string) StringKeyParser

type StringValParser 

type StringValParser interface {
	ParseVal(element string, val *Val) error
}

func NewStringValParser 

func NewStringValParser(kind string) StringValParser

type SyslogDBWriter 

type SyslogDBWriter struct {
	// contains filtered or unexported fields
}

func NewSyslogDBWriter 

func NewSyslogDBWriter() (*SyslogDBWriter, error)

func (*SyslogDBWriter) Write 

func (s *SyslogDBWriter) Write(flowmap AggFlowMap, iface string, timestamp int64)

type TimeStringParser 

type TimeStringParser struct{}

extra attributes

func (*TimeStringParser) ParseKey 

func (t *TimeStringParser) ParseKey(element string, key *ExtraKey) error

type Val 

type Val struct {
	NBytesRcvd uint64
	NBytesSent uint64
	NPktsRcvd  uint64
	NPktsSent  uint64
}

func (*Val) String 

func (v *Val) String() string

Source Files

View all Source files

Directories

Path Synopsis
Package bigendian provides functionality to convert (u)ints encoded in big-endian to little-endian.
 Package bigendian provides functionality to convert (u)ints encoded in big-endian to little-endian.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.