yarascan

README

yarascan

A set of composable tools for incremental filesystem scanning using YARA rules.

The tools can be combined with the likes of find or rdup to provide full or incremental scanning and reporting. The suite includes a centralized server with a database.

Installation

To install from source you can use the following command:

$ go install git.autistici.org/ai3/tools/yarascan/cmd/yarascan

which will install the yarascan binary in $GOPATH/bin.

Packages

The yarascan tool can be packaged either as a Debian package (it requires at least buster to build), or as a Docker image. This is due to the libyara3 dependency.

Usage

The yarascan utility supports a few different commands. The most important are scan and report, which are meant to be combined together in a pipeline, e.g.:

$ rdup -F '%n\n' /dev/null /home \
    | yarascan scan --rules=... \
    | yarascan report --report-url=...

scan

The scan subcommand loads YARA rules and applies them to a set of files. It expects a list of files on standard input, one per line. It will not follow symbolic links or descend into directories. It can be easily combined with the find or rdup utilities. For rdup, the -F '%n\\n' option is required to make it output just the file names.

The --rules option can be used to specify the location of the YARA rules to load. Rules will be compiled when the program starts. The engine will execute on a number of concurrent threads specified by the --num-workers option, by default equal to the number of available cores.

The output of the scan command consists of JSON-encoded detection objects, one for each of the matches returned by the YARA engine. Every detection is serialized to JSON on a separate line, a "streaming" form of JSON record encoding which is suitable to be read by the yarascan report command.

Incremental usage

It's possible to incrementally scan a filesystem using rdup with the -m option, eventually combining it with the --retry-file option to the report command (see below) to improve reliability:

$ rdup -F '%n\n' -m /var/lib/backups/list /home \
    | yarascan scan --rules=... \
    | yarascan report --retry-file=/var/lib/backups/retry --report-url=...

report

The report subcommand reads YARA detections and reports them over HTTP to a centralized database server via its submission API.

If the --retry-file option is specified, the tool will maintain in the given file a list of the failed records, to be retried on the next run. This way it's possible to implement something close to reliable delivery, even in face of long-term failures of the database servers.

server

The server command runs a small HTTP(S) server offering an API to the centralized database of detections. Multiple hosts can feed their data to this centralized service. Authentication is handled by the TLS layer (ref.). The only database backend available is SQLite.

Documentation

Overview

Most code lifted from https://github.com/botherder/kraken.

Index

• Variables
• type Detection
  • func NewDetection(path, signature string) *Detection
• type FindDetectionsBySiteRequest
• type FindDetectionsBySiteResponse
• type FindUnresolvedDetectionPathsRequest
• type FindUnresolvedDetectionPathsResponse
• type PathDetection
• type ResolveEntry
• type ResolveRequest
• type Scanner
  • func NewScanner(rulesPath string) (*Scanner, error)
  • func (s *Scanner) ScanFile(filePath string) ([]*yara.Rule, error)
• type SubmissionRequest

Variables

var Hostname string

Hostname (current machine).

Types

type Detection

type Detection struct {
	Host      string    `json:"host"`
	Path      string    `json:"path"`
	Signature string    `json:"signature"`
	Timestamp time.Time `json:"timestamp"`
}

Detection entry.

func NewDetection

func NewDetection(path, signature string) *Detection

NewDetection builds a new Detection object for the current host.

type FindDetectionsBySiteRequest

type FindDetectionsBySiteRequest struct {
	Site string `json:"site"`
}

FindDetectionsBySiteRequest is the request type for /api/search/by_site.

type FindDetectionsBySiteResponse

type FindDetectionsBySiteResponse struct {
	Detections []*PathDetection `json:"detections"`
}

FindDetectionsBySiteResponse is the result type for FindDetectionsBySiteRequest.

type FindUnresolvedDetectionPathsRequest

type FindUnresolvedDetectionPathsRequest struct {
	Host string `json:"host"`
}

type FindUnresolvedDetectionPathsResponse

type FindUnresolvedDetectionPathsResponse struct {
	Paths []string `json:"paths"`
}

type PathDetection

type PathDetection struct {
	Host       string    `json:"host"`
	Path       string    `json:"path"`
	Site       string    `json:"site"`
	Signatures []string  `json:"signatures"`
	FirstStamp time.Time `json:"first_stamp"`
	LastStamp  time.Time `json:"last_stamp"`
}

PathDetection summarizes the detections by path, aggregating all non-resolved detections over time.

type ResolveEntry

type ResolveEntry struct {
	Site string `json:"site"`
	Path string `json:"path"`
}

type ResolveRequest

type ResolveRequest struct {
	Host    string          `json:"host"`
	Entries []*ResolveEntry `json:"entries"`
}

type Scanner

type Scanner struct {
	// contains filtered or unexported fields
}

Scanner can apply YARA rules to files.

func NewScanner

func NewScanner(rulesPath string) (*Scanner, error)

NewScanner loads YARA rules from rulesPath and returns a new Scanner.

func (*Scanner) ScanFile

func (s *Scanner) ScanFile(filePath string) ([]*yara.Rule, error)

ScanFile scans a file path with the provided Yara rules.

type SubmissionRequest

type SubmissionRequest struct {
	Detections []*Detection `json:"detections"`
}

SubmissionRequest is the request type for /api/submission.