minesweeper

README

minesweeper

Minesweeper is a tool to detect websites infected with drive-by malware.

Shopify uses Minesweeper to protect its 100,000+ online stores from web-based malware infections.

Minesweeper deals with not only the problem of detecting the malware, but also attributing an infection back to a particular asset. It is designed for high speed, parallel operation.

Install

1. Install PhantomJS

On Mac
brew install phantomjs
On Ubuntu
apt-get install phantomjs
Binary Install
http://phantomjs.org/download.html

1. Install minesweeper

Download Release v0.1.0 "Hello World" for Mac 64bit or Linux 64bit

1. Optional - Grab a Google API key

Setup a Google API key by following these intructions.
Add the key as an ENV variable e.g. export MINESWEEPER_GOOGLE_API_KEY="<YOUR_KEY>"

1. Optional - Install Suricata

Instructions and sample config coming soon.

Usage

$ minesweeper [options...] <url>

Minesweeper will browse the URL, perform a security analysis and produce a verdict.

A JSON report is returned if the verdict is suspicious. Nothing is returned if it's ok.

A suspicious verdict means that a module has produced a positive (blacklist) hit or (IDS) alert.

Currently, there are 3 modules: malwaredomains, google and suricata.

Example

$ minesweeper ianfette.org
 {
  "Url": "http://ianfette.org",
  "CreatedAt": "Wed Sep  3 22:27:40 UTC 2014",
  "RunDir": "/var/folders/dg/m668qw1x3szdyhmt8qs0qq3w0000gn/T/minesweeper458550870",
  "Resources": [
    {
      "Method": "GET",
      "Url": "http://ianfette.org/",
      "Status": 200,
      "ContentType": "text/html",
      "ContentLength": 43,
      "MinesweeperSha256": "0e7d00142cf0f74c7e4d5b2469c016bd421837ce692cd6a276fce2f3d5fc3a06",
      "MinesweeperSniffedMime": "text/html; charset=utf-8",
      "MinesweeperHostAddr": "173.201.140.128"
    }
  ],
  "Changes": null,
  "Hits": [
    {
      "BlacklistName": "google",
      "Url": "http://ianfette.org/",
      "Domain": "ianfette.org",
      "Type": "malware",
      "Ref": "https://developers.google.com/safe-browsing/developers_guide_v3"
    }
  ],
  "Alerts": null,
  "Verdict": "suspicious"
}

How does it work?

Minesweeper scans websites using PhantomJS through a local MITM proxy

• It records:
  • HTTP requests for resources such as Javascript and CSS files
    • URL, Method, Status, Content-Type
    • A sha256 sum for a file - useful for submitting to VirusTotal
    • A MIME-sniffed Content-Type determined using DetectContentType
  • Javascript calls to document.write()
    • The HTML to be written is recorded
    • A stack trace is captured to attribute this back to an exact line in the source!
  • Javascript DOMSubtreeModified Mutation Events where the target is either HTMLScriptElement or HTMLIFrameElement
    • The outer HTML of the modification is recorded

It captures the traffic between PhantomJS and the local MITM proxy

• Capturing here means that the IDS has a chance to alert on HTTP request content that will never be seen in a non-proxied environment e.g. a <script> tag referencing a malicious domain that has been DNS blacklisted.
• In order to enable parallel scans, a separate proxy is setup for each scan.
  • Free ports to listen on are chosen by the operating system by binding to port 0
• By running with a BPF filter for a specific port, we can capture the traffic of each scan separately.

It grabs any alerts from the Suricata IDS which is listening on localhost

• Alerts written in the fast format are supported /var/log/suricata/fast.log
• Minesweeper also greps through your rules files /etc/suricata/rules/*.rules to find the full rule text and adds this to each alert

It also checks all URLs browsed against domain blacklists

• Currently, there are 2 blacklists:
  • google - Google Safe Browsing Lookup API
  • malwaredomains - malwaredomains.com

It produces a JSON report

• If there are IDS alerts or Blacklist hits, the website is deemed suspicious, otherwise it's ok.

Questions

Please contact falsenegative