yapscan

README

yapscan

Yapscan is a YAra based Process SCANner, aimed at giving more control about what to scan and giving detailed reports on matches.

The report format is now versioned and a stable version 1.1.0 is released with compatibility guarantees, see the report format documentation.

Features

You can use yapscan to selectively scan the memory of running processes as well as files in local hard drives and/or mounted shares. The most notable differences to stock yara are (see section Usage),

• Supports loading yara-rules from an encrypted zip file to prevent anti-virus software from detecting rules as malicious.
• Multiple yara rules can also be loaded recursively from a directory.
• Can suspend processes to be scanned (use with care, may crash your system).
• Allows for filtering of memory segments to be scanned based on size, type (image, mapped, private), state (commit, free, reserve), and permissions.
• Allows for easy scanning of all running processes, local drives and/or mounted shares.
• Comes with extensive reporting features to allow later analysis of efficacy of rules.
• Matched memory segments can even be automatically dumped and stored as part of the

Other quality-of-life features include

• Statically built, dependency free exe for Windows
• Listing running processes
• Listing and dumping memory segments of a specific process
• Compiling yara rules and compressing them into an encrypted zip.
• Provides an "executable DLL" for locked down environments such as VDIs (see section Executable DLL)
• Anonymization of reports with either a predefined, or a randomly generated salt

Yapscan comes with support for both Windows and Linux, however Windows is the primary and most thoroughly tested target OS.

Usage

I'll write a proper man page soon. For now, use yapscan --help and yapscan <command> --help for usage information.

COMMANDS:
   list-processes, ps, lsproc  lists all running processes
   list-process-memory, lsmem  lists all memory segments of a process
   dump                        dumps memory of a process
   scan                        scans processes or paths with yara rules
   anonymize                   anonymize reports
   zip-rules                   creates an encrypted zip containing compiled yara rules
   join                        joins dumps with padding
   crash-process, crash        crash a process
   as-service                  executes yapscan as a windows service (windows only)
   help, h                     Shows a list of commands or help for one command

> yapscan scan --help
NAME:
   yapscan scan - scans processes or paths with yara rules

USAGE:
   yapscan scan [command options] [pid/path...]

OPTIONS:
   --rules value, -r value, -C value                   path to yara rules file or directory, if it's a file it can be a yara rules file or a zip containing a rules file encrypted with password "infected"
   --rules-recurse, --recurse-rules, --rr              if --rules specifies a directory, compile rules recursively (default: false)
   --all-processes, --all-p                            scan all running processes (default: false)
   --all-drives, --all-d                               scan all files in all local drives, implies --recurse (default: false)
   --all-shares, --all-s                               scan all files in all mounted net-shares, implies --recurse (default: false)
   --file-extensions value, -e value                   list of file extensions to scan, use special extension "-" as no extension, use --file-extensions "" to allow any (default: "-", "so", "exe", "dll", "sys")
   --threads value, -t value                           number of threads (goroutines) used for scanning files (default: 6)
   --full-report                                       create a full report (default: false)
   --scan-mapped-files                                 when encountering memory-mapped files also scan the backing file on disk (default: false)
   --report-dir value                                  the directory to which the report archive will be written (default: current working directory)
   --store-dumps                                       store dumps of memory regions that match rules, implies --full-report, the report will be encrypted with --password (default: false)
   --password value                                    setting this will encrypt the report with the given password; ignored without --full-report
   --pgpkey value                                      setting this will encrypt the report with the public key in the given file; ignored without --full-report
   --anonymize                                         anonymize any output, hashing any usernames, hostnames and IPs with a salt (default: false)
   --salt value                                        the salt (base64 string) to use for anonymization, ignored unless --anonmyize is provided (default: random salt)
   --verbose, -v                                       show more information about rule matches (default: false)
   --filter-permissions value, --f-perm value          only consider segments with the given permissions or more, examples: "rw" includes segments with rw, rc and rwx
   --filter-permissions-exact value, --f-perm-e value  comma separated list of permissions to be considered, supported permissions: r, rw, rc, rwx, rcx
   --filter-type value, --f-type value                 comma separated list of considered types, supported types: image, mapped, private
   --filter-state value, --f-state value               comma separated list of considered states, supported states: free, commit, reserve (default: "commit")
   --filter-size-max value, --f-size-max value         maximum size of memory segments to be considered, can be absolute (e.g. "1.5GB"), percentage of total RAM (e.g. "10%T") or percentage of free RAM (e.g. "10%F") (default: "10%F")
   --filter-size-min value, --f-size-min value         minimum size of memory segments to be considered
   --filter-rss-ratio-min value, --f-rss-min value     minimum RSS/Size ratio of memory segments to eb considered
   --suspend, -s                                       suspend the process before reading its memory (default: false)
   --force, -f                                         don't ask before suspending a process (default: false)
   --help, -h                                          show help (default: false)

Here are some additional example usages

# Create rules zip (optional)
yapscan zip-rules --output rules.zip rules.yara

# Scan a process with PID 423 with default filters
yapscan scan -r rules.zip 423
# Scan all processes with default filters
yapscan scan -r rules.zip --all-processes
# Scan all processes and all local drives with default filters
yapscan scan -r rules.zip --all-processes --all-drives
# Scan everything with default filters
yapscan scan -r rules.zip --all-processes --all-drives --all-shares

# Only scan memory segments with execute permission or more
yapscan scan -r rules.zip --filter-permissions x --all-processes
# Only scan memory segments with exactly read and execute (not write)
yapscan scan -r rules.zip --filter-permissions-exact rx --all-processes

# Enable logging, reporting and auto dumping of matched segments
yapscan --log-level debug --log-path yapscan.log scan -r rules.zip --full-report --store-dumps --all-processes

Running as Service

Yapscan can be run as a windows service in order to gain SYSTEM privileges. This allows you to crash even other windows services, using the crash command. Running as service is currently an experimental feature.

For memory scanning this should not be necessary. In my experiments it has been sufficient to run yapscan as administrator in order to read the memory of any process. If you find a process that yapscan cannot scan with administrator privileges but that can be scanned as a service, please let me know in the issues.

In order to use yapscan as a service just prepend the as-service command to the command (and flags) you wish to execute. Example:

# Normal mode
.\yapscan.exe crash 42
# Service mode
.\yapscan.exe as-service crash 42

The output of the windows service is transmitted to the terminal via two TCP connections. If this breaks a warning will be emitted. In such a case the service may still be running, you just won't see any output. Also CTRL-C will break the proxy command, preventing you from seeing any output, but will not affect the running service. If you want to kill the service, you'll have to use the windows service manager for now.

Executable DLL

The DLL built by this project is not a usual DLL, meant for importing functions from. Instead it acts similarly to the exe with two exported, high-level entry points:

// start acts same as you would expect a main function to act.
// It assumes a terminal with stdout/stderr and stdin has already
// been allocated.
extern int start(int argc, char** argv);

// run is meant for use with rundll32.
// It opens a new console window via AllocConsole(), then parses the
// lpCmdLine to extract the arguments and calls starts yapscan
// with the extracted arguments. 
extern void run(HWND hWnd, HINSTANCE hInst, LPTSTR lpCmdLine, int nCmdShow);

Some environments like VDIs (Virtual Desktop Infrastructure) may prevent the execution of arbitrary exe-files but still allows for use of arbitrary DLLs. If you gain access to a command line terminal in such an environment you can call yapscan via the built DLL like so.

rundll32.exe yapscan.dll,run scan -r rules.zip --all-processes

NOTE: This feature is still experimental! There very likely are quirks with the argument parsing.

State of this project

BETA, FeatureFreeze

Right now yapscan is in a beta period. I am currently working on a stable 1.0 release. This release will have a defined and documented stable API and cli interface that will remain backwards compatible for all 1.x versions. Before releasing 1.0 I also want to have at least a reasonable amount of test-coverage.

Feel free to use and test it and open issues for any bugs you may find. If you would like to have some additional features you can also open an issue with a feature request, but until the 1.0 release I will not be working on or merging any new features.

Contributing

Found a bug or want a feature, but you can't code or don't have the time? Feel free to open an issue! Please look for duplicates first.

If you want to contribute code, be it fixes or features, please open a pull request on the develop branch and mention any related issues.

In the rapid-development phase I have only used the master branch, but will switch very shortly to the git-flow workflow. This means the master branch will represent the latest stable release at any point in time and any work-in-progress is to be merged into the develop branch.

Scanning Technique

The actual scanning is left to the yara library. In case of file scanning, the high level yara library function yr_rules_scan_file is used. This function memory-maps the given file. Scanning process memory, on the other hand, is done on a lower level. Yapscan copies one memory segment at a time into a buffer in its own memory and then uses yr_rules_scan_mem in order to scan this buffer.

Building Yapscan

To build natively on Linux, for Linux you need install Go and the yara library. Once you have installed the dependencies it's as easy as:

# Install Golang and libyara
git clone https://github.com/fkie-cad/yapscan
cd yapscan/cmd/yapscan
go build

If you want to build on Linux for Windows, all you need installed is docker.

# Install docker
git clone https://github.com/fkie-cad/yapscan
cd yapscan/cicd/
./crossBuildForWindows.sh

The resulting binaries will be placed in cicd/build/.

Building natively on Windows, using MSYS2 follow these instructions

1. Install Go
2. Install MSYS2 and follow the first steps on the MSYS2 Website of updating via pacman.
3. Install build dependencies pacman --needed -S base-devel git autoconf automake libtool mingw-w64-{x86_64,i686}-{gcc,make,pkgconf}
4. Open PowerShell in the cicd/ directory and execute .\buildOnWindows.ps1 -MsysPath <msys_path> -BuildDeps where <msys_path> is the install directory for MSYS2, default is C:\msys64. NOTE: You'll have to press Enter on the MSYS window, once the dependencies are finished.
5. Enjoy the built files in cicd/build/

If you want to run tests on Windows, you have to run .\cicd\buildOnWindows.ps1 -BuildDeps only once. Then you open PowerShell and execute .\cicd\enableMingw.ps1 -MsysPath <msys_path> to set the appropriate environment variables. Now it's as easy as go test -tags yara_static ./.... The -tags yara_static is necessary if you use the build scripts, as they do not install any windows DLLs but only the static libraries.

NOTE: You might get inexplicable failures with -race on Windows. According to the golang release notes for 1.14, the new pointer arithmetic checks are somewhat overzealous on Windows. In Golang v1.15 it seems this may not have been fixed, but the checks are enabled automatically. You can deactivate them like this go test -tags yara_static -race -gcflags=all=-d=checkptr=0 ./....

You don't have to rely on the powershell/bash scripts, but they are intended to make things as easy as possible at the cost of control over the compilation. If you want more control, take a look at the scripts use and modify them or execute the commands individually. The scripts perform the following tasks.

1. Start "MSYS2 MinGW 64-bit"
   1. Download OpenSSL from github
   2. Static-Build OpenSSL and install the development files
   3. Download libyara from github
   4. Static-Build libyara and install it
2. Set some environment variables in powershell, to allow the use of the mingw toolchain
3. Call the go build command with the appropriate build tag for static builds

Thanks to @hillu (author of go-yara), for pointing me in the right direction for building natively on windows. See #7 and the links therein if you want some more details.

Documentation

Overview

Package yapscan provides high-level features on top of the popular virus-scanner yara.

Index

• Constants
• Variables
• func AddressesFromMatches(matches []yara.MatchString, offset uint64) []uint64
• func FormatSlice(format string, slice interface{}, args ...interface{}) []string
• func IsYaraRulesFile(name string) bool
• func Join(parts []string, defaultGlue, finalGlue string) string
• func LoadYaraRules(path string, recurseIfDir bool) (*yara.Rules, error)
• type FilterMatch
• type MemoryScanProgress
• type MemoryScanner
• type MemorySegmentFilter
  • func NewAndFilter(filters ...MemorySegmentFilter) MemorySegmentFilter
  • func NewFilterFromFunc(filter MemorySegmentFilterFunc, parameter interface{}, reasonTemplate string, ...) MemorySegmentFilter
  • func NewMaxSizeFilter(size uintptr) MemorySegmentFilter
  • func NewMinSizeFilter(size uintptr) MemorySegmentFilter
  • func NewPermissionsFilter(perm procio.Permissions) MemorySegmentFilter
  • func NewPermissionsFilterExact(perms []procio.Permissions) MemorySegmentFilter
  • func NewRSSRatioFilter(ratio float64) MemorySegmentFilter
  • func NewStateFilter(states []procio.State) MemorySegmentFilter
  • func NewTypeFilter(types []procio.SegmentType) MemorySegmentFilter
• type MemorySegmentFilterFunc
• type ProcessScanner
  • func NewProcessScanner(proc procio.Process, filter MemorySegmentFilter, scanner MemoryScanner) *ProcessScanner
  • func (s *ProcessScanner) EncounteredMemoryMappedFiles() []string
  • func (s *ProcessScanner) Scan() (<-chan *MemoryScanProgress, error)
• type ProfilingInformation
• type Rules
• type ScanningStatistics
  • func NewScanningStatistics() *ScanningStatistics
  • func (s *ScanningStatistics) Finalize()
  • func (s *ScanningStatistics) IncrementFilesScanned(numOfBytes uint64)
  • func (s *ScanningStatistics) IncrementMemorySegmentsScanned(numOfBytes uint64)
  • func (s *ScanningStatistics) IncrementNumberOfProcessesScanned()
  • func (s *ScanningStatistics) StartProfiler(ctx context.Context, scanInterval time.Duration)
• type YaraScanner
  • func NewYaraScanner(rules Rules) (*YaraScanner, error)
  • func (s *YaraScanner) ScanFile(filename string) ([]yara.MatchRule, error)
  • func (s *YaraScanner) ScanMem(buf []byte) ([]yara.MatchRule, error)
  • func (s *YaraScanner) Statistics() *ScanningStatistics

Examples

• Join

Constants

const RulesZIPPassword = "infected"

RulesZIPPassword is the password yapscan uses to de-/encrypt the rules zip file.

Variables

var DefaultYaraRulesNamespace = ""

DefaultYaraRulesNamespace is the default namespace when compiling rules.

var ErrSkipped = errors.New("skipped")

ErrSkipped is returned, when a memory segment is skipped due to the applied filter.

var YaraRulesFileExtensions = []string{
	".yar",
	".yara",
}

YaraRulesFileExtensions are the file extensions yapscan expects rules files to have. This is used when loading files from a directory.

Functions

func AddressesFromMatches

func AddressesFromMatches(matches []yara.MatchString, offset uint64) []uint64

AddressesFromMatches returns one value for each given yara.MatchString. The returned values are equal to the given offset plus the Offset field of each yara.MatchString.

func FormatSlice

func FormatSlice(format string, slice interface{}, args ...interface{}) []string

FormatSlice calls fmt.Sprintf(format, element, args...) for each element in the given slice. The returned string slice contains the formatted output.

func IsYaraRulesFile

func IsYaraRulesFile(name string) bool

IsYaraRulesFile returns true, if the given filename has one of the extensions in YaraRulesFileExtensions.

func Join

func Join(parts []string, defaultGlue, finalGlue string) string

Join joins all elements of a string slice, using the defaultGlue for all but the last two elements.

Example

parts := []string{"life", "the universe", "everything"}
fmt.Println(Join(parts, ", ", " and "))

Output:

life, the universe and everything

func LoadYaraRules

func LoadYaraRules(path string, recurseIfDir bool) (*yara.Rules, error)

LoadYaraRules loads yara.Rules from a file (or files) and compiles if necessary. The given path can be a path to a directory, a compiled rules-file, a plain text file containing rules, or an encrypted zip file containing rules.

If the path is a directory, all files with one of the file extensions in YaraRulesFileExtensions are loaded (recursively if recurseIfDir is true). All files are assumed to be uncompiled and will be compiled. Loading multiple already compiled files into one yara.Rules object is not supported. Each file will be compiled with the namespace equal to its filename, relative to the given path.

If the path is a single file, it may be compiled, uncompiled or a zip file. An uncompiled file will be compiled with the namespace `DefaultYaraRulesNamespace+"/"+filename`. A zip file will be opened and decrypted with the RulesZIPPassword. The contents of the zip file will be treated similar to the way a directory is treated (see above), however *all* files are assumed to be rules-files, recursion is always enabled and there may be either a single compiled file or arbitrarily many uncompiled files in the zip.

Types

type FilterMatch

type FilterMatch struct {
	Result bool
	MSI    *procio.MemorySegmentInfo
	Reason string // Reason for filter mismatch, if Result is false
}

FilterMatch contains information about the matching of a MemorySegmentFilter.

type MemoryScanProgress

type MemoryScanProgress struct {
	// Process contains information about the process being scanned.
	Process procio.Process
	// MemorySegment contains information about the specific memory segment which was just scanned.
	MemorySegment *procio.MemorySegmentInfo
	// Dump contains the raw contents of the memory segment.
	Dump []byte
	// Matches contains the yara.MatchRule results.
	Matches []yara.MatchRule
	// Error contains the encountered error or nil, if no error was encountered.
	Error error
}

MemoryScanProgress contains all information, generated during scanning.

type MemoryScanner

type MemoryScanner interface {
	ScanMem(buf []byte) (results []yara.MatchRule, err error)
}

MemoryScanner is a yara.Rules compatible interface, defining the subset of functions required for scanning memory buffers.

type MemorySegmentFilter

type MemorySegmentFilter interface {
	Filter(info *procio.MemorySegmentInfo) *FilterMatch
	Description() string
}

MemorySegmentFilter describes an interface, capable of filtering *procio.MemorySegmentInfo instances.

func NewAndFilter

func NewAndFilter(filters ...MemorySegmentFilter) MemorySegmentFilter

NewAndFilter creates a new filter, which is the logical AND-combination of all given MemorySegmentFilter instances.

func NewFilterFromFunc

func NewFilterFromFunc(filter MemorySegmentFilterFunc, parameter interface{}, reasonTemplate string, description string) MemorySegmentFilter

NewFilterFromFunc creates a new filter from a given MemorySegmentFilterFunc.

func NewMaxSizeFilter

func NewMaxSizeFilter(size uintptr) MemorySegmentFilter

NewMaxSizeFilter creates a new filter, matching *procio.MemorySegmentInfo with the given maximum size.

func NewMinSizeFilter

func NewMinSizeFilter(size uintptr) MemorySegmentFilter

NewMinSizeFilter creates a new filter, matching *procio.MemorySegmentInfo with the given minimum size.

func NewPermissionsFilter

func NewPermissionsFilter(perm procio.Permissions) MemorySegmentFilter

NewPermissionsFilter creates a new filter, matching *procio.MemorySegmentInfo with procio.Permissions equal to or more permissive than the given perm.

func NewPermissionsFilterExact

func NewPermissionsFilterExact(perms []procio.Permissions) MemorySegmentFilter

NewPermissionsFilterExact creates a new filter, matching *procio.MemorySegmentInfo with procio.Permissions exactly equal to one of the given perms.

func NewRSSRatioFilter

func NewRSSRatioFilter(ratio float64) MemorySegmentFilter

NewRSSRatioFilter creates a new filter, matching *procio.MemorySegmentInfo with RSS/Size ratio equal or greater than the given value.

func NewStateFilter

func NewStateFilter(states []procio.State) MemorySegmentFilter

NewStateFilter creates a new filter, matching *procio.MemorySegmentInfo with a procio.State equal to one of the given states.

func NewTypeFilter

func NewTypeFilter(types []procio.SegmentType) MemorySegmentFilter

NewTypeFilter creates a new filter, matching *procio.MemorySegmentInfo with a procio.SegmentType equal to one of the given types.

type MemorySegmentFilterFunc

type MemorySegmentFilterFunc func(info *procio.MemorySegmentInfo) bool

MemorySegmentFilterFunc is a callback, used to filter *procio.MemorySegmentInfo instances.

type ProcessScanner

type ProcessScanner struct {
	// contains filtered or unexported fields
}

ProcessScanner implements scanning of memory segments, allocated by a process. This scanning is done using an underlying MemoryScanner on segments, matching a MemorySegmentFilter.

func NewProcessScanner

func NewProcessScanner(proc procio.Process, filter MemorySegmentFilter, scanner MemoryScanner) *ProcessScanner

NewProcessScanner create a new ProcessScanner with for the given procio.Process. It uses the given MemoryScanner in order to scan memory segments of the process, which match the given MemoryScanner.

func (*ProcessScanner) EncounteredMemoryMappedFiles

func (s *ProcessScanner) EncounteredMemoryMappedFiles() []string

func (*ProcessScanner) Scan

func (s *ProcessScanner) Scan() (<-chan *MemoryScanProgress, error)

Scan starts an asynchronous scan. The returned unbuffered channel will yield MemoryScanProgress instances every time a memory segment has been processed. The channel will be closed when all segments have been processed.

type ProfilingInformation

type ProfilingInformation struct {
	Time                  report.Time `json:"time"`
	FreeRAM               uintptr     `json:"freeRAM"`
	FreeSwap              uintptr     `json:"freeSwap"`
	LoadAvgOneMinute      float64     `json:"loadAvgOneMinute"`
	LoadAvgFiveMinutes    float64     `json:"loadAvgFiveMinutes"`
	LoadAvgFifteenMinutes float64     `json:"loadAvgFifteenMinutes"`
}

type Rules

type Rules interface {
	ScanFile(filename string, flags yara.ScanFlags, timeout time.Duration, cb yara.ScanCallback) (err error)
	ScanMem(buf []byte, flags yara.ScanFlags, timeout time.Duration, cb yara.ScanCallback) (err error)
}

Rules are a yara.Rules compatible interface, defining the functions required by yapscan. The choice of an interface over the concrete struct yara.Rules is mostly to make testing easier.

type ScanningStatistics

type ScanningStatistics struct {
	Start                      report.Time             `json:"start"`
	End                        report.Time             `json:"end"`
	NumberOfProcessesScanned   uint64                  `json:"numberOfProcessesScanned"`
	NumberOfSegmentsScanned    uint64                  `json:"numberOfSegmentsScanned"`
	NumberOfMemoryBytesScanned uint64                  `json:"numberOfMemoryBytesScanned"`
	NumberOfFileBytesScanned   uint64                  `json:"numberOfFileBytesScanned"`
	NumberOfFilesScanned       uint64                  `json:"numberOfFilesScanned"`
	ProfilingInformation       []*ProfilingInformation `json:"profilingInformation"`
	// contains filtered or unexported fields
}

ScanningStatistics holds statistic information about a scan.

func NewScanningStatistics

func NewScanningStatistics() *ScanningStatistics

func (*ScanningStatistics) Finalize

func (s *ScanningStatistics) Finalize()

Finalize finalizes the statistics, stopping the memory profile routine if its running. Use this function before processing the statistics further. This function is thread safe.

func (*ScanningStatistics) IncrementFilesScanned

func (s *ScanningStatistics) IncrementFilesScanned(numOfBytes uint64)

IncrementFilesScanned increments the number of files scanned as well as the number of bytes scanned. This function is thread safe.

func (*ScanningStatistics) IncrementMemorySegmentsScanned

func (s *ScanningStatistics) IncrementMemorySegmentsScanned(numOfBytes uint64)

IncrementMemorySegmentsScanned increments the number of segments scanned as well as the number of bytes scanned. This function is thread safe.

func (*ScanningStatistics) IncrementNumberOfProcessesScanned

func (s *ScanningStatistics) IncrementNumberOfProcessesScanned()

IncrementNumberOfProcessesScanned increments the number of scanned processes. This function is thread safe.

func (*ScanningStatistics) StartProfiler

func (s *ScanningStatistics) StartProfiler(ctx context.Context, scanInterval time.Duration)

StartProfiler starts a goroutine, regularly saving information about free memory, free swap and CPU load.

type YaraScanner

type YaraScanner struct {
	// contains filtered or unexported fields
}

YaraScanner is a wrapper for yara.Rules, with a more go-like interface.

func NewYaraScanner

func NewYaraScanner(rules Rules) (*YaraScanner, error)

NewYaraScanner creates a new YaraScanner from the given yara.Rules.

func (*YaraScanner) ScanFile

func (s *YaraScanner) ScanFile(filename string) ([]yara.MatchRule, error)

ScanFile scans the file with the given filename. This function simply calls ScanFile on the underlying yara.Rules object.

func (*YaraScanner) ScanMem

func (s *YaraScanner) ScanMem(buf []byte) ([]yara.MatchRule, error)

ScanMem scans the given buffer. This function simply calls ScanMem on the underlying yara.Rules object.

func (*YaraScanner) Statistics

func (s *YaraScanner) Statistics() *ScanningStatistics

Statistics returns the mutable statistics of the scanner.