database

Documentation

Index

• type DB
  • func NewDB(cfg *config.Resources) *DB
  • func (d *DB) BuildBlacklistedCollection()
  • func (d *DB) BuildConnectionsCollection()
  • func (d *DB) BuildHostnamesCollection()
  • func (d *DB) BuildHostsCollection()
  • func (d *DB) BuildHttpCollection()
  • func (d *DB) BuildScanningCollection()
  • func (d *DB) BuildTBDCollection()
  • func (d *DB) BuildUniqueConnectionsCollection()
  • func (d *DB) BuildUrlsCollection()
  • func (d *DB) BuildUserAgentCollection()
• type DBMetaInfo
• type MetaDBHandle
  • func NewMetaDBHandle(cfg *config.Resources) *MetaDBHandle
  • func (m *MetaDBHandle) AddNewDB(name string) error
  • func (m *MetaDBHandle) GetDatabases() []string
  • func (m *MetaDBHandle) GetFiles() []*PFile
  • func (m *MetaDBHandle) GetUnAnalysedDatabases() []string
  • func (m *MetaDBHandle) MarkCompleted(f *PFile) error
  • func (m *MetaDBHandle) MarkDBCompleted(name string) error
  • func (m *MetaDBHandle) UpdateFiles(files []*PFile) []*PFile
• type PFile

Types

type DB

type DB struct {
	// contains filtered or unexported fields
}

DB is the workhorse container for messing with the database

func NewDB

func NewDB(cfg *config.Resources) *DB

NewDB builds up a new data session

func (*DB) BuildBlacklistedCollection

func (d *DB) BuildBlacklistedCollection()

///////////////////////////////////////////////////////////////////////////// ////////////////// LAYER 2 COLLECTION BUILDING FUNCTIONS //////////////////// /////////////////////////////////////////////////////////////////////////////

• Name: BuildBlacklistedCollection
• Purpose: Builds the blacklisted collection
• Build Type:
• Source:
• comments:

func (*DB) BuildConnectionsCollection

func (d *DB) BuildConnectionsCollection()

BuildConnectionsCollection builds the 'conn' collection. Sourced from the bro parser.

func (*DB) BuildHostnamesCollection

func (d *DB) BuildHostnamesCollection()

* Name: BuildHostnamesCollection * Purpose: Builds the hostnames collection * Build Type: aggregation * Source: urls collection * comments: Relies on the url collection being built

func (*DB) BuildHostsCollection

func (d *DB) BuildHostsCollection()

BuildHostsCollection builds the 'host' collection for this timeframe. Note that this is a different host collection that the one found in HostsIntelDB. This host collection references only hosts found in this time frame, info from the HostsIntelDB collection can be found by following the 'intelid' field after it is populated by the cymru and blacklist modules. Runs via mongodb aggregation. Sourced from the 'conn' table. TODO: Confirm that this section of code is not faster than an aggregation from the 'uconn' table which should have less repeated data.

func (*DB) BuildHttpCollection

func (d *DB) BuildHttpCollection()

BuildHttpCollection builds the 'http' collection. Sourced from the bro parser.

func (*DB) BuildScanningCollection

func (d *DB) BuildScanningCollection()

* Name: BuildScanningCollection * Purpose: Builds the scanning collection * Build Type: aggregation * Source: connections table * comments:

func (*DB) BuildTBDCollection

func (d *DB) BuildTBDCollection()

* Name: BuildTBDCollection * Purpose: Builds the TBD collection * Build Type: * Source: * comments:

func (*DB) BuildUniqueConnectionsCollection

func (d *DB) BuildUniqueConnectionsCollection()

BuildUniqeConnectionsCollection builds the 'uconn' collection. Runs via mongodb aggreggation. Sourced from the 'conn' collection.

func (*DB) BuildUrlsCollection

func (d *DB) BuildUrlsCollection()

* Name: BuildUrlsCollection * Purpose: Builds the urls collection * Build Type: map reduce -> aggregation * Source: http table * comments:

func (*DB) BuildUserAgentCollection

func (d *DB) BuildUserAgentCollection()

* Name: BuildUserAgentCollection * Purpose: Builds the useragent collection * Build Type: aggregation * Source: http table * comments:

type DBMetaInfo

type DBMetaInfo struct {
	ID       bson.ObjectId `bson:"_id,omitempty"` // Ident
	Name     string        `bson:"name"`          // Top level name of the database
	Analysed bool          `bson:"analyzed"`      // Has this database been analyzed
}

DBMetaInfo defines some information about the database

type MetaDBHandle

type MetaDBHandle struct {
	DB      string       // Database path
	Session *mgo.Session // Session to the database
	// contains filtered or unexported fields
}

MetaDBHandle exports control for the meta database

func NewMetaDBHandle

func NewMetaDBHandle(cfg *config.Resources) *MetaDBHandle

NewMetaDBHandle takes in a configuration and returns a MetaDBHandle controller

func (*MetaDBHandle) AddNewDB

func (m *MetaDBHandle) AddNewDB(name string) error

AddNewDB adds a new database tot he DBMetaInfo table

func (*MetaDBHandle) GetDatabases

func (m *MetaDBHandle) GetDatabases() []string

GetDatabases returns a list of databases being tracked in metadb or an empty array on failure

func (*MetaDBHandle) GetFiles

func (m *MetaDBHandle) GetFiles() []*PFile

GetFiles gets a list of all PFile objects in the database if successful return a list of files from the database, in the case of failure return a zero length list of files and generat a log message.

func (*MetaDBHandle) GetUnAnalysedDatabases

func (m *MetaDBHandle) GetUnAnalysedDatabases() []string

GetUnAnalyzedDatabases builds a list of database names which have yet to be analyzed and returns

func (*MetaDBHandle) MarkCompleted

func (m *MetaDBHandle) MarkCompleted(f *PFile) error

markComplete will mark a file as having been completed in the database

func (*MetaDBHandle) MarkDBCompleted

func (m *MetaDBHandle) MarkDBCompleted(name string) error

MarkDBCompleted marks a database as having been analyzed

func (*MetaDBHandle) UpdateFiles

func (m *MetaDBHandle) UpdateFiles(files []*PFile) []*PFile

updateFiles updates the files table with all of the new files from a recent walk of the dir structure at the end of the update we return a new GetFiles array so that the parser knows which files to get to parsing.

type PFile

type PFile struct {
	ID       bson.ObjectId `bson:"_id,omitempty"`
	Path     string        `bson:"filepath"`
	Hash     string        `bson:"hash"`
	Length   int64         `bson:"length"`
	Parsed   int64         `bson:"time_complete"`
	Mod      time.Time     `bson:"modified"`
	DataBase string        `bson:"database"`
}

PFile retains everything we need to know about a given file