mig

README

MIG: Mozilla InvestiGator

Build one-liner:

$ go get mig.ninja/mig && cd $GOPATH/src/mig.ninja/mig && make

MIG is OpSec's platform for investigative surgery of remote endpoints.

MIG is composed of agents installed on all systems of an infrastructure that are be queried in real-time to investigate the file-systems, network state, memory or configuration of endpoints.

Capability	Linux	MacOS	Windows
file inspection
network inspection			(partial)
memory inspection
vuln management		(planned)	(planned)
log analysis	(planned)	(planned)	(planned)
system auditing	(planned)	(planned)	(planned)

Imagine it is 7am on a saturday morning, and someone just released a critical vulnerability for your favorite PHP application. The vuln is already exploited and security groups are releasing indicators of compromise (IOCs). Your weekend isn't starting great, and the thought of manually inspecting thousands of systems isn't making it any better.

MIG can help. The signature of the vulnerable PHP app (the md5 of a file, a regex, or just a filename) can be searched for across all your systems using the file module. Similarly, IOCs such as specific log entries, backdoor files with {md5,sha{1,256,512,3-{256,512}}} hashes, IP addresses from botnets or byte strings in processes memories can be investigated using MIG. Suddenly, your weekend is looking a lot better. And with just a few commands, thousands of systems will be remotely investigated to verify that you're not at risk.

MIG agents are designed to be lightweight, secure, and easy to deploy so you can ask your favorite sysadmins to add it to a base deployment without fear of breaking the entire production network. All parameters are built into the agent at compile time, including the list and ACLs of authorized investigators. Security is enforced using PGP keys, and even if MIG's servers are compromised, as long as our keys are safe on your investigator's laptop, no one will break into the agents.

MIG is designed to be fast, and asynchronous. It uses AMQP to distribute actions to endpoints, and relies on Go channels to prevent components from blocking. Running actions and commands are stored in a Postgresql database and on disk cache, such that the reliability of the platform doesn't depend on long-running processes.

Speed is a strong requirement. Most actions will only take a few hundreds milliseconds to run on agents. Larger ones, for example when looking for a hash in a big directory, should run in less than a minute or two. All in all, an investigation usually completes in between 10 and 300 seconds.

Privacy and security are paramount. Agents never send raw data back to the platform, but only reply to questions instead. All actions are signed by GPG keys that are not stored in the platform, thus preventing a compromise from taking over the entire infrastructure.

Technology

MIG is built in Go and uses a REST API that receives signed JSON messages distributed to agents via RabbitMQ and stored in a Postgres database.

It is:

• Massively Distributed means Fast.
• Simple to deploy and Cross-Platform.
• Secured using OpenPGP.
• Respectful of privacy by never retrieving raw data from endpoints.

Check out this 10 minutes video for a more general presentation and a demo of the console interface.

MIG was recently presented at the SANS DFIR Summit in Austin, Tx. You can watch the recording below:

Discussion

Join #mig on irc.mozilla.org

Documentation

All documentation is available in the 'doc' directory and on http://mig.mozilla.org .

• Concepts & Internal Components
• Installation & Configuration

Bug & Issue tracker

We use Bugzilla to track the work on MIG.

• List open bugs: Bugzilla MIG
• Create a new bug: New Bug in Bugzilla

Documentation

Overview

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/.

Contributor: Julien Vehent jvehent@mozilla.com [:ulfr]

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/.

Contributor: Julien Vehent jvehent@mozilla.com [:ulfr]

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/.

Contributor: Julien Vehent jvehent@mozilla.com [:ulfr]

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/.

Contributor: Julien Vehent jvehent@mozilla.com [:ulfr]

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/.

Contributor: Julien Vehent jvehent@mozilla.com [:ulfr]

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/.

Contributor: Julien Vehent jvehent@mozilla.com [:ulfr]

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/.

Contributor: Julien Vehent jvehent@mozilla.com [:ulfr]

Index

• Constants
• func GenB32ID() string
• func GenID() float64
• func ProcessLog(logctx Logging, l Log) (stop bool, err error)
• type ACL
• type Action
  • func ActionFromFile(path string) (Action, error)
  • func (a Action) PrintCounters()
  • func (a Action) Sign(keyid string, secring io.Reader) (sig string, err error)
  • func (a Action) String() (str string, err error)
  • func (a Action) ToTempFile() (filename string, err error)
  • func (a Action) Validate() (err error)
  • func (a Action) VerifyACL(acl ACL, keyring io.Reader) (err error)
  • func (a Action) VerifySignatures(keyring io.Reader) (err error)
• type ActionCounters
• type Agent
• type AgentEnv
• type AgentsStats
• type AgentsVersionsSum
• type Command
  • func CmdFromFile(path string) (cmd Command, err error)
• type Description
• type Investigator
• type Log
  • func (l Log) Alert() (mlog Log)
  • func (l Log) Crit() (mlog Log)
  • func (l Log) Debug() (mlog Log)
  • func (l Log) Emerg() (mlog Log)
  • func (l Log) Err() (mlog Log)
  • func (l Log) Info() (mlog Log)
  • func (l Log) Notice() (mlog Log)
  • func (l Log) Warning() (mlog Log)
• type Logging
  • func InitLogger(orig_logctx Logging, progname string) (logctx Logging, err error)
  • func (logctx Logging) Destroy()
• type Operation
• type Permission
• type Threat

Constants

const (
	AgtStatusOnline    string = "online"
	AgtStatusUpgraded  string = "upgraded"
	AgtStatusDestroyed string = "destroyed"
	AgtStatusOffline   string = "offline"
	AgtStatusIdle      string = "idle"
)

const (
	StatusSent      string = "sent"
	StatusSuccess   string = "success"
	StatusCancelled string = "cancelled"
	StatusExpired   string = "expired"
	StatusFailed    string = "failed"
	StatusTimeout   string = "timeout"
)

const (
	// rabbitmq exchanges and common queues
	Mq_Ex_ToAgents     = "toagents"
	Mq_Ex_ToSchedulers = "toschedulers"
	Mq_Ex_ToWorkers    = "toworkers"
	Mq_Q_Heartbeat     = "mig.agt.heartbeats"
	Mq_Q_Results       = "mig.agt.results"

	// event queues
	Ev_Q_Agt_Auth_Fail = "agent.authentication.failure"
	Ev_Q_Agt_New       = "agent.new"
	Ev_Q_Cmd_Res       = "command.results"
)

const (
	StatusActiveInvestigator   string = "active"
	StatusDisabledInvestigator string = "disabled"
)

const (
	MODE_STDOUT = 1 << iota
	MODE_FILE
	MODE_SYSLOG
)

const ActionVersion uint16 = 2

ActionVersion is the version of the syntax that is expected

Functions

func GenB32ID

func GenB32ID() string

GenHexID returns a string with an hexadecimal encoded ID

func GenID

func GenID() float64

GenID() returns a float64 ID number that is unique to this process. The ID is initialized at the number of seconds since MIG's creation date, shifted 16 bits to the right and incremented by one every time a new ID is requested. The resulting value must fit in 53 bits of precision provided by the float64 type.

func ProcessLog

func ProcessLog(logctx Logging, l Log) (stop bool, err error)

processLog receives events and perform logging and evaluationg of the log if the log is too critical, Analyze will trigger a scheduler shutdown

Types

type ACL

type ACL []Permission

type Action

type Action struct {
	ID             float64        `json:"id"`
	Name           string         `json:"name"`
	Target         string         `json:"target"`
	Description    Description    `json:"description,omitempty"`
	Threat         Threat         `json:"threat,omitempty"`
	ValidFrom      time.Time      `json:"validfrom"`
	ExpireAfter    time.Time      `json:"expireafter"`
	Operations     []Operation    `json:"operations"`
	PGPSignatures  []string       `json:"pgpsignatures"`
	Investigators  []Investigator `json:"investigators,omitempty"`
	Status         string         `json:"status,omitempty"`
	StartTime      time.Time      `json:"starttime,omitempty"`
	FinishTime     time.Time      `json:"finishtime,omitempty"`
	LastUpdateTime time.Time      `json:"lastupdatetime,omitempty"`
	Counters       ActionCounters `json:"counters,omitempty"`
	SyntaxVersion  uint16         `json:"syntaxversion,omitempty"`
}

an Action is the json object that is created by an investigator and provided to the MIG platform. It must be PGP signed.

func ActionFromFile

func ActionFromFile(path string) (Action, error)

ActionFromFile() reads an action from a local file on the file system and returns a mig.Action structure

func (Action) PrintCounters

func (a Action) PrintCounters()

PrintCounters prints the counters of an action to stderr

func (Action) Sign

func (a Action) Sign(keyid string, secring io.Reader) (sig string, err error)

Sign computes and returns the GPG signature of a MIG action in its stringified form

func (Action) String

func (a Action) String() (str string, err error)

concatenates Action components into a string

func (Action) ToTempFile

func (a Action) ToTempFile() (filename string, err error)

ToTempFile writes an action into a generated temporary file and returns its filename

func (Action) Validate

func (a Action) Validate() (err error)

Validate verifies that the Action received contained all the necessary fields, and returns an error when it doesn't.

func (Action) VerifyACL

func (a Action) VerifyACL(acl ACL, keyring io.Reader) (err error)

VerifyACL controls that an action has been issued by investigators that have the right permissions. This function looks at each operation listed in the action, and find the corresponding permission. If no permission is found, the default one `default` is used. The first permission that is found to apply to an operation, but doesn't allow the operation to run, will fail the verification globally

func (Action) VerifySignatures

func (a Action) VerifySignatures(keyring io.Reader) (err error)

VerifySignatures verifies that the Action contains valid signatures from known investigators. It does not verify permissions.

type ActionCounters

type ActionCounters struct {
	Sent      int `json:"sent,omitempty"`
	Done      int `json:"done,omitempty"`
	InFlight  int `json:"inflight,omitempty"`
	Success   int `json:"success,omitempty"`
	Cancelled int `json:"cancelled,omitempty"`
	Expired   int `json:"expired,omitempty"`
	Failed    int `json:"failed,omitempty"`
	TimeOut   int `json:"timeout,omitempty"`
}

Some counters used to track the completion of an action

type Agent

type Agent struct {
	ID              float64     `json:"id,omitempty"`
	Name            string      `json:"name"`
	QueueLoc        string      `json:"queueloc"`
	Mode            string      `json:"mode"`
	Version         string      `json:"version,omitempty"`
	PID             int         `json:"pid,omitempty"`
	StartTime       time.Time   `json:"starttime,omitempty"`
	DestructionTime time.Time   `json:"destructiontime,omitempty"`
	HeartBeatTS     time.Time   `json:"heartbeatts,omitempty"`
	Status          string      `json:"status,omitempty"`
	Authorized      bool        `json:"authorized,omitempty"`
	Env             AgentEnv    `json:"environment,omitempty"`
	Tags            interface{} `json:"tags,omitempty"`
}

Agent stores the description of an agent and serves as a canvas for heartbeat messages

type AgentEnv

type AgentEnv struct {
	Init      string   `json:"init,omitempty"`
	Ident     string   `json:"ident,omitempty"`
	OS        string   `json:"os,omitempty"`
	Arch      string   `json:"arch,omitempty"`
	IsProxied bool     `json:"isproxied"`
	Proxy     string   `json:"proxy,omitempty"`
	Addresses []string `json:"addresses,omitempty"`
	PublicIP  string   `json:"publicip,omitempty"`
}

AgentEnv stores basic information of the endpoint

type AgentsStats

type AgentsStats struct {
	Timestamp             time.Time           `json:"timestamp"`
	OnlineAgents          float64             `json:"onlineagents"`
	OnlineAgentsByVersion []AgentsVersionsSum `json:"onlineagentsbyversion"`
	OnlineEndpoints       float64             `json:"onlineendpoints"`
	IdleAgents            float64             `json:"idleagents"`
	IdleAgentsByVersion   []AgentsVersionsSum `json:"idleagentsbyversion"`
	IdleEndpoints         float64             `json:"idleendpoints"`
	NewEndpoints          float64             `json:"newendpoints"`
	MultiAgentsEndpoints  float64             `json:"multiagentsendpoints"`
	DisappearedEndpoints  float64             `json:"disappearedendpoints"`
	FlappingEndpoints     float64             `json:"flappingendpoints"`
}

type AgentsVersionsSum

type AgentsVersionsSum struct {
	Version string  `json:"version"`
	Count   float64 `json:"count"`
}

type Command

type Command struct {
	ID     float64 `json:"id"`
	Action Action  `json:"action"`
	Agent  Agent   `json:"agent"`

	// Status can be one of:
	// sent: the command has been sent by the scheduler to the agent
	// success: the command has successfully ran on the agent and been returned to the scheduler
	// cancelled: the command has been cancelled by the investigator
	// expired: the command has been expired by the scheduler
	// failed: the command has failed on the agent and been returned to the scheduler
	// timeout: module execution has timed out, and the agent returned the command to the scheduler
	Status string `json:"status"`

	Results    []modules.Result `json:"results"`
	StartTime  time.Time        `json:"starttime"`
	FinishTime time.Time        `json:"finishtime"`
}

func CmdFromFile

func CmdFromFile(path string) (cmd Command, err error)

FromFile reads a command from a local file on the file system and return the mig.Command structure

type Description

type Description struct {
	Author   string  `json:"author,omitempty"`
	Email    string  `json:"email,omitempty"`
	URL      string  `json:"url,omitempty"`
	Revision float64 `json:"revision,omitempty"`
}

a description is a simple object that contains detail about the action's author, and it's revision.

type Investigator

type Investigator struct {
	ID             float64   `json:"id,omitempty"`
	Name           string    `json:"name"`
	PGPFingerprint string    `json:"pgpfingerprint"`
	PublicKey      []byte    `json:"publickey,omitempty"`
	PrivateKey     []byte    `json:"privatekey,omitempty"`
	Status         string    `json:"status"`
	CreatedAt      time.Time `json:"createdat"`
	LastModified   time.Time `json:"lastmodified"`
}

type Log

type Log struct {
	OpID, ActionID, CommandID float64
	Sev, Desc                 string
	Priority                  syslog.Priority
}

Log defines a log entry

func (Log) Alert

func (l Log) Alert() (mlog Log)

func (Log) Crit

func (l Log) Crit() (mlog Log)

func (Log) Debug

func (l Log) Debug() (mlog Log)

func (Log) Emerg

func (l Log) Emerg() (mlog Log)

func (Log) Err

func (l Log) Err() (mlog Log)

func (Log) Info

func (l Log) Info() (mlog Log)

func (Log) Notice

func (l Log) Notice() (mlog Log)

func (Log) Warning

func (l Log) Warning() (mlog Log)

type Logging

type Logging struct {
	// configuration
	Mode, Level, File, Host, Protocol, Facility string
	Port                                        int
	// contains filtered or unexported fields
}

Logging stores the attributes needed to perform the logging

func InitLogger

func InitLogger(orig_logctx Logging, progname string) (logctx Logging, err error)

InitLogger prepares the context for logging based on the configuration in Logging

func (Logging) Destroy

func (logctx Logging) Destroy()

type Operation

type Operation struct {
	Module     string      `json:"module"`
	Parameters interface{} `json:"parameters"`
}

an operation is an object that map to an agent module. the parameters of the operation are passed to the module as argument, and thus their format depend on the module itself.

type Permission

type Permission map[string]struct {
	MinimumWeight int
	Investigators map[string]struct {
		Fingerprint string
		Weight      int
	}
}

type Threat

type Threat struct {
	Ref    string `json:"ref,omitempty"`
	Level  string `json:"level,omitempty"`
	Family string `json:"family,omitempty"`
	Type   string `json:"type,omitempty"`
}

a threat provides the investigator with an idea of how dangerous a the compromission might be, if the indicators return positive