fswalker

README

fswalker

A simple and fast file system integrity checking tool in Go.

Overview

fswalker consists of two parts:

• Walker: The walker collects information about the target machine's file system and writes the collected list out in binary proto format. The walker policy defines which directories to include and exclude.

• Reporter: The reporter is a tool which runs outside of the target machine and compares two runs (aka Walks) with each other and reports the diffs, if any. The report config defines which directories to include and exclude.

Note: The walker and the reporter have two separate definitions of directories to include and exclude. This is done on purpose so more information can be collected than what is later reviewed. If something suspicious comes up, it is always possible to see more changes than the ones deemed "interesting" in the first place.

Why using fswalker instead of using existing solutions such as Tripwire, AIDE, Samhain, etc?

• It's opensource and actively developed.
• All data formats used are open as well and thus allow easy imports and exports.
• It's easily expandable with local modifications.
• No dependencies on non-standard Go libraries outside github.com/google.

Installation

go get github.com/google/fswalker/cmd/walker
go get github.com/google/fswalker/cmd/reporter

Configuration

Walker Policy

The Walker policy specifies how a file system is walked and what to write to the output file. Most notably, it contains a list of includes and excludes.

• include: Includes are starting points for the file walk. All includes are walked simultaneously.

• exclude_pfx: Excludes are specified as prefixes. They are literal string prefix matches. To make this more clear, let's assume we have an include of "/" and an exclude_pfx of "/home". When the walker evaluates "/home", it will skip it because the prefix matches. However, it also skips "/homeofme/important.file".

Refer to the proto buffer description to see a complete reference of all options and their use.

The following constitutes a functional example for Ubuntu:

policy.textpb

version: 1
max_hash_file_size: 1048576
walk_cross_device: true
ignore_irregular_files: false
include: "/"
exclude_pfx: "/usr/local/"
exclude_pfx: "/usr/src/"
exclude_pfx: "/usr/share/"
exclude_pfx: "/var/backups/"
exclude_pfx: "/var/cache/"
exclude_pfx: "/var/log/"
exclude_pfx: "/var/mail/"
exclude_pfx: "/var/spool/"
exclude_pfx: "/var/tmp/"

Reporter Config

The reporter allows to specify fewer things in its config, notably excludes. The reason to have additional excludes in the reporter is simple: It allows recording more details in the walks and fewer to be reported. If something suspicious is ever found, it allows going back to previous walks however and check what the status was back then.

• exclude_pfx: Excludes are specified as prefixes. They are literal string prefix matches. To make this more clear, let's assume we have an include of "/" and an exclude_pfx of "/home". When the walker evaluates "/home", it will skip it because the prefix matches. However, it also skips "/homeofme/important.file".

The following constitutes a functional example for Ubuntu:

config.textpb

version: 1
exclude_pfx: "/root/"
exclude_pfx: "/home/"
exclude_pfx: "/tmp/"

Refer to the proto buffer description to see a complete reference of all options.

Review File

The following constitutes a functional example:

reviews.textpb

review: {
  key: "some-host.google.com"
  value: {
    walk_id: "457ab084-2426-4ca8-b54c-cefdce543042"
    walk_reference: "/tmp/some-host.google.com-20181205-060000-fswalker-state.pb"
    fingerprint: {
      method: SHA256
      value: "0bfb7506e44dbca14914c3250b2d4d5be005d0de4460c9f298f227bac096f642"
    }
  }
}

Refer to the proto buffer description to see a complete reference of all options.

Examples

The following examples show how to run both the walker and the reporter.

Note that there are libraries for each which can be used independently if so desired. See the implementations of walker and reporter main for a reference on how to use the libraries.

Walker

Once you have a policy as described above, you can run the walker:

walker \
  -policy-file=policy.textpb \
  -output-file-pfx="/tmp"

Add -verbose to see more details about what's going on.

Reporter

Once you have a config as described above and more than one Walk file, you can run the reporter.

Add -verbose to see more details about what's going on.

To allow for easier reviews, -paginate allows to invoke $PAGER (or less if $PAGER is not set) to page through the results.

Direct Comparison

The simplest way to run it is to directly specify two Walk files to compare against each other:

reporter \
  -config-file=config.textpb \
  -before-file=/tmp/some-host.google.com-20181205-060000-fswalker-state.pb \
  -after-file=/tmp/some-host.google.com-20181206-060000-fswalker-state.pb \
  -paginate

Note that you can also run with just -after-file specified which will basically list all files as newly added. This is only really useful with a new machine.

Review File Based

Contrary to the above example, reporter would normally be run with a review file:

reporter \
  -config-file=config.textpb \
  -review-file=reviews.textpb \ # this needs to be writeable!
  -walk-path=/tmp \
  -hostname=some-host.google.com \
  -paginate

The reporter runs, displays all diffs and when deemed ok, updates the review file with the latest "known good" information.

The idea is that the review file contains a set of "known good" states and is under version control and four-eye principle / reviews.

Development

Protocol Buffer

If you change the protocol buffer, ensure you generate a new Go library based on it:

go generate

(The rules for go generate are in fswalker.go.)

License

Apache 2.0

This is not an officially supported Google product

Documentation

Overview

Package fswalker contains functionality to walk a file system and compare the differences.

Index

• Variables
• func NormalizePath(path string, isDir bool) string
• func WalkFilename(hostname string, t time.Time) string
• type ActionData
• type Report
  • func (r *Report) Empty() bool
• type Reporter
  • func ReporterFromConfigFile(ctx context.Context, path string, verbose bool) (*Reporter, error)
  • func (r *Reporter) Compare(before, after *fspb.Walk) (*Report, error)
  • func (r *Reporter) PrintDiffSummary(out io.Writer, report *Report)
  • func (r *Reporter) PrintReportSummary(out io.Writer, report *Report)
  • func (r *Reporter) PrintRuleSummary(out io.Writer, report *Report)
  • func (r *Reporter) ReadLastGoodWalk(ctx context.Context, hostname, reviewFile string) (*WalkFile, error)
  • func (r *Reporter) ReadLatestWalk(ctx context.Context, hostname, walkPath string) (*WalkFile, error)
  • func (r *Reporter) ReadWalk(ctx context.Context, path string) (*WalkFile, error)
  • func (r *Reporter) UpdateReviewProto(ctx context.Context, walkFile *WalkFile, reviewFile string) error
• type WalkCallback
• type WalkFile
• type Walker
  • func WalkerFromPolicyFile(ctx context.Context, path string) (*Walker, error)
  • func (w *Walker) Run(ctx context.Context) error

Variables

var Glob = func(_ context.Context, pattern string) ([]string, error) {
	return filepath.Glob(pattern)
}

Glob returns the names of all files matching pattern or nil if there is no matching file.

var ReadFile = func(_ context.Context, filename string) ([]byte, error) {
	return ioutil.ReadFile(filename)
}

ReadFile reads the file named by filename and returns the contents.

var WriteFile = func(_ context.Context, filename string, data []byte, perm os.FileMode) error {
	return ioutil.WriteFile(filename, data, perm)
}

WriteFile writes data to a file named by filename.

Functions

func NormalizePath

func NormalizePath(path string, isDir bool) string

NormalizePath returns a cleaned up path with a path separator at the end if it's a directory. It should always be used when printing or comparing paths.

func WalkFilename

func WalkFilename(hostname string, t time.Time) string

WalkFilename returns the appropriate filename for a Walk for the given host and time. If time is not provided, it returns a file pattern to glob by.

Types

type ActionData

type ActionData struct {
	Before *fspb.File
	After  *fspb.File
	Diff   string
	Err    error
}

ActionData contains a diff between two files in different Walks.

type Report

type Report struct {
	Added      []ActionData
	Deleted    []ActionData
	Modified   []ActionData
	Errors     []ActionData
	Counter    *metrics.Counter
	WalkBefore *fspb.Walk
	WalkAfter  *fspb.Walk
}

Report contains the result of the comparison between two Walks.

func (*Report) Empty

func (r *Report) Empty() bool

Empty returns true if there are no additions, no deletions, no modifications and no errors.

type Reporter

type Reporter struct {

	// Verbose, when true, makes Reporter print more information for all diffs found.
	Verbose bool
	// contains filtered or unexported fields
}

Reporter compares two Walks against each other based on the config provided and prints a list of diffs between the two.

func ReporterFromConfigFile

func ReporterFromConfigFile(ctx context.Context, path string, verbose bool) (*Reporter, error)

ReporterFromConfigFile creates a new Reporter based on a config path.

func (*Reporter) Compare

func (r *Reporter) Compare(before, after *fspb.Walk) (*Report, error)

Compare two Walks and returns the diffs.

func (*Reporter) PrintDiffSummary

func (r *Reporter) PrintDiffSummary(out io.Writer, report *Report)

PrintDiffSummary prints the diffs found in a Report.

func (*Reporter) PrintReportSummary

func (r *Reporter) PrintReportSummary(out io.Writer, report *Report)

PrintReportSummary prints a few key information pieces around the Report.

func (*Reporter) PrintRuleSummary

func (r *Reporter) PrintRuleSummary(out io.Writer, report *Report)

PrintRuleSummary prints the configs and policies involved in creating the Walk and Report.

func (*Reporter) ReadLastGoodWalk

func (r *Reporter) ReadLastGoodWalk(ctx context.Context, hostname, reviewFile string) (*WalkFile, error)

ReadLastGoodWalk reads the designated review file and attempts to find an entry matching the given hostname. Note that if it can't find one but the review file itself was read successfully, it will return an empty Walk and no error. It returns the file path it ended up reading, the Walk it read and the fingerprint for it.

func (*Reporter) ReadLatestWalk

func (r *Reporter) ReadLatestWalk(ctx context.Context, hostname, walkPath string) (*WalkFile, error)

ReadLatestWalk looks for the latest Walk in a given folder for a given hostname. It returns the file path it ended up reading, the Walk it read and the fingerprint for it.

func (*Reporter) ReadWalk

func (r *Reporter) ReadWalk(ctx context.Context, path string) (*WalkFile, error)

ReadWalk reads a file as marshaled proto in fspb.Walk format.

func (*Reporter) UpdateReviewProto

func (r *Reporter) UpdateReviewProto(ctx context.Context, walkFile *WalkFile, reviewFile string) error

UpdateReviewProto updates the reviews file to the reviewed version to be "last known good".

type WalkCallback

type WalkCallback func(context.Context, *fspb.Walk) error

WalkCallback is called by Walker at the end of the Run. The callback is typically used to dump the walk to disk and/or perform any other checks. The error return value is propagated back to the Run callers.

type WalkFile

type WalkFile struct {
	Path        string
	Walk        *fspb.Walk
	Fingerprint *fspb.Fingerprint
}

WalkFile contains info about a Walk file.

type Walker

type Walker struct {

	// Function to call once the Walk is complete i.e. to inspect or write the Walk.
	WalkCallback WalkCallback

	// Verbose, when true, makes Walker print file metadata to stdout.
	Verbose bool

	// Counter records stats over all processed files, if non-nil.
	Counter *metrics.Counter
	// contains filtered or unexported fields
}

Walker is able to walk a file structure starting with a list of given includes as roots. All paths starting with any prefix specified in the excludes are ignored. The list of specific files in the hash list are read and a hash sum built for each. Note that this is expensive and should not be done for large files or a large number of files.

func WalkerFromPolicyFile

func WalkerFromPolicyFile(ctx context.Context, path string) (*Walker, error)

WalkerFromPolicyFile creates a new Walker based on a policy path.

func (*Walker) Run

func (w *Walker) Run(ctx context.Context) error

Run is the main function of Walker. It discovers all files under included paths (minus excluded ones) and processes them. This does NOT follow symlinks - fortunately we don't need it either.