mig

README

MIG: Mozilla InvestiGator

Build one-liner:

$ go get mig.ninja/mig && cd $GOPATH/src/mig.ninja/mig && make

MIG is OpSec's platform for investigative surgery of remote endpoints.

MIG is composed of agents installed on all systems of an infrastructure that are be queried in real-time to investigate the file-systems, network state, memory or configuration of endpoints.

Capability	Linux	MacOS	Windows
file inspection
network inspection			(partial)
memory inspection
vuln management		(planned)	(planned)
log analysis	(planned)	(planned)	(planned)
system auditing	(planned)	(planned)	(planned)

Imagine it is 7am on a saturday morning, and someone just released a critical vulnerability for your favorite PHP application. The vuln is already exploited and security groups are releasing indicators of compromise (IOCs). Your weekend isn't starting great, and the thought of manually inspecting thousands of systems isn't making it any better.

MIG can help. The signature of the vulnerable PHP app (the md5 of a file, a regex, or just a filename) can be searched for across all your systems using the file module. Similarly, IOCs such as specific log entries, backdoor files with md5 and sha1/2/3 hashes, IP addresses from botnets or byte strings in processes memories can be investigated using MIG. Suddenly, your weekend is looking a lot better. And with just a few commands, thousands of systems will be remotely investigated to verify that you're not at risk.

MIG agents are designed to be lightweight, secure, and easy to deploy so you can ask your favorite sysadmins to add it to a base deployment without fear of breaking the entire production network. All parameters are built into the agent at compile time, including the list and ACLs of authorized investigators. Security is enforced using PGP keys, and even if MIG's servers are compromised, as long as our keys are safe on your investigator's laptop, no one will break into the agents.

MIG is designed to be fast, and asynchronous. It uses AMQP to distribute actions to endpoints, and relies on Go channels to prevent components from blocking. Running actions and commands are stored in a Postgresql database and on disk cache, such that the reliability of the platform doesn't depend on long-running processes.

Speed is a strong requirement. Most actions will only take a few hundreds milliseconds to run on agents. Larger ones, for example when looking for a hash in a big directory, should run in less than a minute or two. All in all, an investigation usually completes in between 10 and 300 seconds.

Privacy and security are paramount. Agents never send raw data back to the platform, but only reply to questions instead. All actions are signed by GPG keys that are not stored in the platform, thus preventing a compromise from taking over the entire infrastructure.

Technology

MIG is built in Go and uses a REST API that receives signed JSON messages distributed to agents via RabbitMQ and stored in a Postgres database.

It is:

• Massively Distributed means Fast.
• Simple to deploy and Cross-Platform.
• Secured using OpenPGP.
• Respectful of privacy by never retrieving raw data from endpoints.

Check out this 10 minutes video for a more general presentation and a demo of the console interface.

MIG was recently presented at the SANS DFIR Summit in Austin, Tx. You can watch the recording below:

Discussion

Join #mig on irc.mozilla.org (use a web client such as mibbit ).

We also have a public mailing list at list@mig.ninja.

Documentation

All documentation is available in the 'doc' directory and on http://mig.mozilla.org .

• Concepts & Internal Components
• Installation & Configuration

Documentation

Index

• Constants
• Variables
• func GenB32ID() string
• func GenID() float64
• func ProcessLog(logctx Logging, l Log) (stop bool, err error)
• func TransformAMQPAuth(in []byte) (out []byte)
• func ValidateLoaderKey(key string) error
• type ACL
• type Action
  • func ActionFromFile(path string) (Action, error)
  • func (a Action) IndentedString() (string, error)
  • func (a Action) PrintCounters()
  • func (a Action) Sign(keyid string, secring io.Reader) (sig string, err error)
  • func (a Action) String() (str string, err error)
  • func (a Action) ToTempFile() (filename string, err error)
  • func (a Action) Validate() (err error)
  • func (a Action) VerifyACL(acl ACL, keyring io.Reader) (err error)
  • func (a Action) VerifySignatures(keyring io.Reader) (err error)
• type ActionCounters
• type Agent
• type AgentEnv
• type AgentEnvAWS
• type AgentsStats
• type AgentsVersionsSum
• type BundleDictionaryEntry
  • func GetHostBundle() ([]BundleDictionaryEntry, error)
  • func HashBundle(b []BundleDictionaryEntry) ([]BundleDictionaryEntry, error)
• type Command
  • func CmdFromFile(path string) (cmd Command, err error)
• type Description
• type Investigator
• type LoaderEntry
  • func (le *LoaderEntry) Validate() error
• type Log
  • func (l Log) Alert() (mlog Log)
  • func (l Log) Crit() (mlog Log)
  • func (l Log) Debug() (mlog Log)
  • func (l Log) Emerg() (mlog Log)
  • func (l Log) Err() (mlog Log)
  • func (l Log) Info() (mlog Log)
  • func (l Log) Notice() (mlog Log)
  • func (l Log) Warning() (mlog Log)
• type Logging
  • func InitLogger(orig_logctx Logging, progname string) (logctx Logging, err error)
  • func (logctx Logging) Destroy()
• type ManifestEntry
• type ManifestFetchResponse
• type ManifestParameters
  • func (m *ManifestParameters) Validate() error
  • func (m *ManifestParameters) ValidateFetch() error
• type ManifestRecord
  • func (m *ManifestRecord) ContentFromFile(path string) (err error)
  • func (m *ManifestRecord) ManifestObject(obj string) ([]byte, error)
  • func (m *ManifestRecord) ManifestResponse() (ManifestResponse, error)
  • func (m *ManifestRecord) Sign(keyid string, secring io.Reader) (sig string, err error)
  • func (m *ManifestRecord) Validate() (err error)
• type ManifestResponse
  • func (m *ManifestResponse) Validate() error
  • func (m *ManifestResponse) VerifySignatures(keyring io.Reader) (validcnt int, err error)
• type Operation
  • func (op *Operation) CompressOperationParam() (err error)
  • func (op *Operation) DecompressOperationParam() (err error)
• type Permission
• type RunnerResult
• type Threat

Constants

const (
	AgtStatusOnline    string = "online"
	AgtStatusUpgraded  string = "upgraded"
	AgtStatusDestroyed string = "destroyed"
	AgtStatusOffline   string = "offline"
	AgtStatusIdle      string = "idle"
)

const (
	StatusSent      string = "sent"
	StatusSuccess   string = "success"
	StatusCancelled string = "cancelled"
	StatusExpired   string = "expired"
	StatusFailed    string = "failed"
	StatusTimeout   string = "timeout"
)

const (
	// rabbitmq exchanges and common queues
	Mq_Ex_ToAgents     = "toagents"
	Mq_Ex_ToSchedulers = "toschedulers"
	Mq_Ex_ToWorkers    = "toworkers"
	Mq_Q_Heartbeat     = "mig.agt.heartbeats"
	Mq_Q_Results       = "mig.agt.results"

	// event queues
	Ev_Q_Agt_Auth_Fail = "agent.authentication.failure"
	Ev_Q_Agt_New       = "agent.new"
	Ev_Q_Cmd_Res       = "command.results"

	// dummy queue for scheduler heartbeats to the relays
	Ev_Q_Sched_Hb = "scheduler.heartbeat"
)

const (
	StatusActiveInvestigator   string = "active"
	StatusDisabledInvestigator string = "disabled"
)

const (
	MODE_STDOUT = 1 << iota
	MODE_FILE
	MODE_SYSLOG
)

const ActionVersion uint16 = 2

ActionVersion is the version of the syntax that is expected

Variables

var BundleDictionary = map[string][]BundleDictionaryEntry{
	"linux":  bundleEntryLinux,
	"darwin": bundleEntryDarwin,
}

var Version string = "20160429-0.51f628c"

Functions

func GenB32ID

func GenB32ID() string

GenHexID returns a string with an hexadecimal encoded ID

func GenID

func GenID() float64

GenID() returns a float64 ID number that is unique to this process. The ID is initialized at the number of seconds since MIG's creation date, shifted 16 bits to the right and incremented by one every time a new ID is requested. The resulting value must fit in 53 bits of precision provided by the float64 type.

func ProcessLog

func ProcessLog(logctx Logging, l Log) (stop bool, err error)

processLog receives events and perform logging and evaluationg of the log if the log is too critical, Analyze will trigger a scheduler shutdown

func TransformAMQPAuth

func TransformAMQPAuth(in []byte) (out []byte)

Transforms a byte string containing the agent configuration file, removing AMQP credentials and replacing them with a place holder. This is used during manifest comparison operations, as configuration files can be loader specific but we want to compare against the manifests template configuration.

func ValidateLoaderKey

func ValidateLoaderKey(key string) error

Types

type ACL

type ACL []Permission

type Action

type Action struct {
	ID             float64        `json:"id"`
	Name           string         `json:"name"`
	Target         string         `json:"target"`
	Description    Description    `json:"description,omitempty"`
	Threat         Threat         `json:"threat,omitempty"`
	ValidFrom      time.Time      `json:"validfrom"`
	ExpireAfter    time.Time      `json:"expireafter"`
	Operations     []Operation    `json:"operations"`
	PGPSignatures  []string       `json:"pgpsignatures"`
	Investigators  []Investigator `json:"investigators,omitempty"`
	Status         string         `json:"status,omitempty"`
	StartTime      time.Time      `json:"starttime,omitempty"`
	FinishTime     time.Time      `json:"finishtime,omitempty"`
	LastUpdateTime time.Time      `json:"lastupdatetime,omitempty"`
	Counters       ActionCounters `json:"counters,omitempty"`
	SyntaxVersion  uint16         `json:"syntaxversion,omitempty"`
}

an Action is the json object that is created by an investigator and provided to the MIG platform. It must be PGP signed.

func ActionFromFile

func ActionFromFile(path string) (Action, error)

ActionFromFile() reads an action from a local file on the file system and returns a mig.Action structure

func (Action) IndentedString

func (a Action) IndentedString() (string, error)

Return the an indented JSON string representing the action suitable for display

func (Action) PrintCounters

func (a Action) PrintCounters()

PrintCounters prints the counters of an action to stderr

func (Action) Sign

func (a Action) Sign(keyid string, secring io.Reader) (sig string, err error)

Sign computes and returns the GPG signature of a MIG action in its stringified form

func (Action) String

func (a Action) String() (str string, err error)

concatenates Action components into a string

func (Action) ToTempFile

func (a Action) ToTempFile() (filename string, err error)

ToTempFile writes an action into a generated temporary file and returns its filename

func (Action) Validate

func (a Action) Validate() (err error)

Validate verifies that the Action received contained all the necessary fields, and returns an error when it doesn't.

func (Action) VerifyACL

func (a Action) VerifyACL(acl ACL, keyring io.Reader) (err error)

VerifyACL controls that an action has been issued by investigators that have the right permissions. This function looks at each operation listed in the action, and find the corresponding permission. If no permission is found, the default one `default` is used. The first permission that is found to apply to an operation, but doesn't allow the operation to run, will fail the verification globally

func (Action) VerifySignatures

func (a Action) VerifySignatures(keyring io.Reader) (err error)

VerifySignatures verifies that the Action contains valid signatures from known investigators. It does not verify permissions.

type ActionCounters

type ActionCounters struct {
	Sent      int `json:"sent,omitempty"`
	Done      int `json:"done,omitempty"`
	InFlight  int `json:"inflight,omitempty"`
	Success   int `json:"success,omitempty"`
	Cancelled int `json:"cancelled,omitempty"`
	Expired   int `json:"expired,omitempty"`
	Failed    int `json:"failed,omitempty"`
	TimeOut   int `json:"timeout,omitempty"`
}

Some counters used to track the completion of an action

type Agent

type Agent struct {
	ID              float64     `json:"id,omitempty"`
	Name            string      `json:"name"`
	QueueLoc        string      `json:"queueloc"`
	Mode            string      `json:"mode"`
	Version         string      `json:"version,omitempty"`
	PID             int         `json:"pid,omitempty"`
	StartTime       time.Time   `json:"starttime,omitempty"`
	DestructionTime time.Time   `json:"destructiontime,omitempty"`
	HeartBeatTS     time.Time   `json:"heartbeatts,omitempty"`
	RefreshTS       time.Time   `json:"refreshts,omitempty"`
	Status          string      `json:"status,omitempty"`
	Authorized      bool        `json:"authorized,omitempty"`
	Env             AgentEnv    `json:"environment,omitempty"`
	Tags            interface{} `json:"tags,omitempty"`
}

Agent stores the description of an agent and serves as a canvas for heartbeat messages

type AgentEnv

type AgentEnv struct {
	Init      string      `json:"init,omitempty"`
	Ident     string      `json:"ident,omitempty"`
	OS        string      `json:"os,omitempty"`
	Arch      string      `json:"arch,omitempty"`
	IsProxied bool        `json:"isproxied"`
	Proxy     string      `json:"proxy,omitempty"`
	Addresses []string    `json:"addresses,omitempty"`
	PublicIP  string      `json:"publicip,omitempty"`
	AWS       AgentEnvAWS `json:"aws,omitempty"`
}

AgentEnv stores basic information of the endpoint

type AgentEnvAWS

type AgentEnvAWS struct {
	InstanceID   string `json:"instanceid,omitempty"`
	LocalIPV4    string `json:"localipv4,omitempty"`
	AMIID        string `json:"amiid,omitempty"`
	InstanceType string `json:"instancetype,omitempty"`
}

Stores AWS specific agent environment values

type AgentsStats

type AgentsStats struct {
	Timestamp             time.Time           `json:"timestamp"`
	OnlineAgents          float64             `json:"onlineagents"`
	OnlineAgentsByVersion []AgentsVersionsSum `json:"onlineagentsbyversion"`
	OnlineEndpoints       float64             `json:"onlineendpoints"`
	IdleAgents            float64             `json:"idleagents"`
	IdleAgentsByVersion   []AgentsVersionsSum `json:"idleagentsbyversion"`
	IdleEndpoints         float64             `json:"idleendpoints"`
	NewEndpoints          float64             `json:"newendpoints"`
	MultiAgentsEndpoints  float64             `json:"multiagentsendpoints"`
	DisappearedEndpoints  float64             `json:"disappearedendpoints"`
	FlappingEndpoints     float64             `json:"flappingendpoints"`
}

type AgentsVersionsSum

type AgentsVersionsSum struct {
	Version string  `json:"version"`
	Count   float64 `json:"count"`
}

type BundleDictionaryEntry

type BundleDictionaryEntry struct {
	Name          string
	Path          string
	SHA256        string
	TransformFunc func([]byte) []byte
	Perm          os.FileMode
}

The bundle dictionary is used to map tokens within the loader manifest to objects on the file system. We don't allow specification of an exact path for interrogation or manipulation in the manifest. This results in some restrictions but hardens the loader against making unauthorized changes to the file system.

If a Transform function is set on the entry, this is used to transform bytes into the data set prior to hash calculation

func GetHostBundle

func GetHostBundle() ([]BundleDictionaryEntry, error)

func HashBundle

func HashBundle(b []BundleDictionaryEntry) ([]BundleDictionaryEntry, error)

Populates a slice of BundleDictionaryEntrys, adding the SHA256 checksums from the file system

type Command

type Command struct {
	ID     float64 `json:"id"`
	Action Action  `json:"action"`
	Agent  Agent   `json:"agent"`

	// Status can be one of:
	// sent: the command has been sent by the scheduler to the agent
	// success: the command has successfully ran on the agent and been returned to the scheduler
	// cancelled: the command has been cancelled by the investigator
	// expired: the command has been expired by the scheduler
	// failed: the command has failed on the agent and been returned to the scheduler
	// timeout: module execution has timed out, and the agent returned the command to the scheduler
	Status string `json:"status"`

	Results    []modules.Result `json:"results"`
	StartTime  time.Time        `json:"starttime"`
	FinishTime time.Time        `json:"finishtime"`
}

func CmdFromFile

func CmdFromFile(path string) (cmd Command, err error)

FromFile reads a command from a local file on the file system and return the mig.Command structure

type Description

type Description struct {
	Author   string  `json:"author,omitempty"`
	Email    string  `json:"email,omitempty"`
	URL      string  `json:"url,omitempty"`
	Revision float64 `json:"revision,omitempty"`
}

a description is a simple object that contains detail about the action's author, and it's revision.

type Investigator

type Investigator struct {
	ID             float64   `json:"id,omitempty"`
	Name           string    `json:"name"`
	PGPFingerprint string    `json:"pgpfingerprint"`
	PublicKey      []byte    `json:"publickey,omitempty"`
	PrivateKey     []byte    `json:"privatekey,omitempty"`
	Status         string    `json:"status"`
	CreatedAt      time.Time `json:"createdat"`
	LastModified   time.Time `json:"lastmodified"`
	IsAdmin        bool      `json:"isadmin"`
}

type LoaderEntry

type LoaderEntry struct {
	ID        float64   // Loader ID
	Name      string    // Loader name
	Key       string    // Loader key (only populated during creation)
	AgentName string    // Loader environment, agent name
	LastSeen  time.Time // Last time loader was used
	Enabled   bool      // Loader entry is active
}

Describes a loader entry stored in the database

func (*LoaderEntry) Validate

func (le *LoaderEntry) Validate() error

type Log

type Log struct {
	OpID, ActionID, CommandID float64
	Sev, Desc                 string
	Priority                  syslog.Priority
}

Log defines a log entry

func (Log) Alert

func (l Log) Alert() (mlog Log)

func (Log) Crit

func (l Log) Crit() (mlog Log)

func (Log) Debug

func (l Log) Debug() (mlog Log)

func (Log) Emerg

func (l Log) Emerg() (mlog Log)

func (Log) Err

func (l Log) Err() (mlog Log)

func (Log) Info

func (l Log) Info() (mlog Log)

func (Log) Notice

func (l Log) Notice() (mlog Log)

func (Log) Warning

func (l Log) Warning() (mlog Log)

type Logging

type Logging struct {
	// configuration
	Mode, Level, File, Host, Protocol, Facility string
	Port                                        int
	MaxFileSize                                 int64
	// contains filtered or unexported fields
}

Logging stores the attributes needed to perform the logging

func InitLogger

func InitLogger(orig_logctx Logging, progname string) (logctx Logging, err error)

InitLogger prepares the context for logging based on the configuration in Logging

func (Logging) Destroy

func (logctx Logging) Destroy()

type ManifestEntry

type ManifestEntry struct {
	Name   string `json:"name"`   // Corresponds to a bundle name
	SHA256 string `json:"sha256"` // SHA256 of entry
}

Describes individual file elements within a manifest

type ManifestFetchResponse

type ManifestFetchResponse struct {
	Data []byte `json:"data"`
}

The response to a manifest object fetch

type ManifestParameters

type ManifestParameters struct {
	AgentIdentifier Agent  `json:"agent"`  // Agent context information
	Object          string `json:"object"` // Object being requested
}

Manifest parameters are sent from the loader to the API as part of a manifest request.

func (*ManifestParameters) Validate

func (m *ManifestParameters) Validate() error

Validate parameters included in a manifest request

func (*ManifestParameters) ValidateFetch

func (m *ManifestParameters) ValidateFetch() error

Validate parameters included in a manifest request with an object fetch component

type ManifestRecord

type ManifestRecord struct {
	ID         float64   `json:"id"`                // Manifest record ID
	Name       string    `json:"name"`              // The name of the manifest record
	Content    string    `json:"content,omitempty"` // Full data contents of record
	Timestamp  time.Time `json:"timestamp"`         // Record timestamp
	Status     string    `json:"status"`            // Record status
	Target     string    `json:"target"`            // Targetting parameters for record
	Signatures []string  `json:"signatures"`        // Signatures applied to the record
}

Describes a manifest record stored within the MIG database

func (*ManifestRecord) ContentFromFile

func (m *ManifestRecord) ContentFromFile(path string) (err error)

Load manifest content from a file on the file system (a gzip'd tar file), primarily utilized by mig-console during manifest creation operations.

func (*ManifestRecord) ManifestObject

func (m *ManifestRecord) ManifestObject(obj string) ([]byte, error)

Returns the requested file object as a gzip compressed byte slice from the manifest record

func (*ManifestRecord) ManifestResponse

func (m *ManifestRecord) ManifestResponse() (ManifestResponse, error)

Convert a manifest record into a manifest response

func (*ManifestRecord) Sign

func (m *ManifestRecord) Sign(keyid string, secring io.Reader) (sig string, err error)

Sign a manifest record

func (*ManifestRecord) Validate

func (m *ManifestRecord) Validate() (err error)

Validate an existing manifest record

type ManifestResponse

type ManifestResponse struct {
	LoaderName string          `json:"loader_name"`
	Entries    []ManifestEntry `json:"entries"`
	Signatures []string        `json:"signatures"`
}

The response to a standard manifest request

func (*ManifestResponse) Validate

func (m *ManifestResponse) Validate() error

Validate a manifest response

func (*ManifestResponse) VerifySignatures

func (m *ManifestResponse) VerifySignatures(keyring io.Reader) (validcnt int, err error)

Validates signatures stored in the manifest against keys in keyring, returns the number of valid signature matches

type Operation

type Operation struct {
	Module     string      `json:"module"`
	Parameters interface{} `json:"parameters"`

	// If WantCompressed is set in the operation, the parameters
	// will be compressed in PostAction() when the client sends the
	// action to the API. This will also result in IsCompressed being
	// marked as true, so the receiving agent knows it must decompress
	// the parameter data.
	IsCompressed   bool `json:"is_compressed,omitempty"`
	WantCompressed bool `json:"want_compressed,omitempty"`
}

an operation is an object that maps to an agent module. the parameters of the operation are passed to the module as an argument, and thus their format depends on the module itself.

func (*Operation) CompressOperationParam

func (op *Operation) CompressOperationParam() (err error)

Compress the parameters stored within an operation

func (*Operation) DecompressOperationParam

func (op *Operation) DecompressOperationParam() (err error)

Decompress the parameters stored within an operation

type Permission

type Permission map[string]struct {
	MinimumWeight int
	Investigators map[string]struct {
		Fingerprint string
		Weight      int
	}
}

type RunnerResult

type RunnerResult struct {
	Action     Action    `json:"action"`
	Commands   []Command `json:"commands"`
	EntityName string    `json:"name"`
	UsePlugin  string    `json:"plugin"`
}

Describes results that are produced by mig-runner

type Threat

type Threat struct {
	Ref    string `json:"ref,omitempty"`
	Level  string `json:"level,omitempty"`
	Family string `json:"family,omitempty"`
	Type   string `json:"type,omitempty"`
}

a threat provides the investigator with an idea of how dangerous a the compromission might be, if the indicators return positive