scalibr

README

OSV-SCALIBR

Note: The code in this repo is subject to change in the near future as we're merging SCALIBR with OSV-scanner to provide a single tool that unifies the two scanners' extraction and vuln scanning capabilities.

SCALIBR (Software Composition Analysis Library) is an extensible file system scanner used to extract software inventory data (e.g. installed language packages) and detect vulnerabilities.

The scanner can either be used as a standalone binary to scan the local machine or as a library with a custom wrapper to perform scans on e.g. container images or remote hosts. It comes with built-in plugins for inventory extraction and vulnerability detection and it also allows users to run their custom plugins.

See the list of currently supported software inventory types.

Prerequisites

To build SCALIBR, you'll need to have go installed. Follow https://go.dev/doc/install.

How to use

As a standalone binary

Note: This binary is a thin wrapper around the OSV-SCALIBR library. In the near future the osv-scanner CLI will make all capabilities of OSV-SCALIBR available with convenience features such as vuln matching through OSV.dev.

1. go install github.com/google/osv-scalibr/binary/scalibr@latest
2. scalibr --result=result.textproto

See the result proto definition for details about the scan result format.

Run scalibr --help for a list of additional CLI args.

As a library:

1. Import github.com/google/osv-scalibr into your Go project
2. Create a new scalibr.ScanConfig struct, configure the extraction and detection plugins to run
3. Call scalibr.New().Scan() with the config
4. Parse the returned scalibr.ScanResults

See below for an example code snippet.

On a container image

Add the --remote-image flag to scan a remote container image. Example:

scalibr --result=result.textproto --remote-image=alpine@sha256:0a4eaa0eecf5f8c050e5bba433f58c052be7587ee8af3e8b3910ef9ab5fbe9f5

SPDX generation

SCALIBR supports generating the result of inventory extraction as an SPDX v2.3 file in json, yaml or tag-value format. Example usage:

scalibr -o spdx23-json=result.spdx.json

Some fields in the generated SPDX can be overwritten:

scalibr -spdx-document-name="Custom name" --spdx-document-namespace="Custom-namespace" --spdx-creators=Organization:Google -o spdx23-json=result.spdx.json

Running built-in plugins

With the standalone binary

The binary runs SCALIBR's "recommended" internal plugins by default. You can enable more plugins with the --extractors= and --detectors= flags. See the definition files for a list of all built-in plugins and their CLI flags (extractors (fs), detectors).

With the library

A collection of all built-in plugin modules can be found in the definition files (extractors, detectors). To enable them, just import the module and add the appropriate plugins to the scan config, e.g.

import (
  scalibr "github.com/google/osv-scalibr"
  el "github.com/google/osv-scalibr/extractor/filesystem/list"
  dl "github.com/google/osv-scalibr/detector/list"
)
cfg := &scalibr.ScanConfig{
  Root:                 "/",
  FilesystemExtractors: el.Python,
  Detectors:            dl.CIS,
}
results := scalibr.New().Scan(context.Background(), cfg)

Creating + running custom plugins

Custom plugins can only be run when using SCALIBR as a library.

1. Create an implementation of the SCALIBR Extractor or Detector interface.
2. Add the newly created struct to the scan config and run the scan, e.g.

import (
  "github.com/google/osv-scalibr/extractor/filesystem"
  scalibr "github.com/google/osv-scalibr"
)
cfg := &scalibr.ScanConfig{
  Root:                 "/",
  FilesystemExtractors: []extractor.Extractor{&myExtractor{}},
}
results := scalibr.New().Scan(context.Background(), cfg)

A note on cross-platform

SCALIBR is compatible with Linux and has experimental support for Windows and Mac. When a new plugin is implemented for SCALIBR, we need to ensure that it will not break other platforms. Our runners will generally catch compatibility issue, but to ensure everything is easy when implementing a plugin, here are a few recommendations to keep in mind:

• Ensure you work with file paths using the filepath library. For example, avoid using /my/path but prefer filepath.Join('my', 'path') instead.
• If the plugin can only support one system (e.g. a windows-specific detector), the layout will generally be to have two versions of the file:
  • file_system.go: where system is the targeted system (e.g. file_windows.go) that contains the code specific to the target system. It must also contain the adequate go build constraint.
  • file_dummy.go: contains the code for every other system. It generally does nothing and just ensures that the code compiles on that system;
• Because of the way our internal automation works, we generally require unit tests to be defined for every platform and be filtered out dynamically if not compatible. In other words, a test should be filtered in/out using if runtime.GOOS rather than a //go:build constraint. Here is an example.

Custom logging

You can make the SCALIBR library log using your own custom logger by passing an implementation of the log.Logger interface to log.SetLogger():

import (
  customlog "path/to/custom/log"
  "github.com/google/osv-scalibr/log"
  scalibr "github.com/google/osv-scalibr"
)
cfg := &scalibr.ScanConfig{ScanRoot: "/"}
log.SetLogger(&customlog.Logger{})
results := scalibr.New().Scan(context.Background(), cfg)
log.Info(results)

Contributing

Read how to contribute to SCALIBR.

To build and test your local changes, run make and make test. A local scalibr binary will be generated in the repo base.

Some of your code contributions might require regenerating protos. This can happen when, say, you want to contribute a new inventory type. For such cases, you'll need install a few dependencies

• protoc: Install the appropriate precompiled protoc binary.
  • For Mac, you can also install via HomeBrew.
• protoc-gen-go: Run go install google.golang.org/protobuf/cmd/protoc-gen-go

and then run make protos or ./build_protos.sh.

Disclaimers

SCALIBR is not an official Google product.

Documentation

Overview

Package scalibr provides an interface for running software inventory extraction and security finding detection on a machine.

Index

• func CmpInventories(a, b *extractor.Inventory) int
• type ScanConfig
  • func (cfg *ScanConfig) EnableRequiredExtractors() error
  • func (cfg *ScanConfig) ValidatePluginRequirements() error
• type ScanResult
• type Scanner
  • func New() *Scanner
  • func (Scanner) Scan(ctx context.Context, config *ScanConfig) (sr *ScanResult)
  • func (s Scanner) ScanContainer(ctx context.Context, img *image.Image, config *ScanConfig) (sr *ScanResult, err error)

Functions

func CmpInventories

func CmpInventories(a, b *extractor.Inventory) int

CmpInventories is a comparison helper fun to be used for sorting Inventory structs.

Types

type ScanConfig

type ScanConfig struct {
	FilesystemExtractors []filesystem.Extractor
	StandaloneExtractors []standalone.Extractor
	Detectors            []detector.Detector
	// Capabilities that the scanning environment satisfies, e.g. whether there's
	// network access. Some plugins can only run if certain requirements are met.
	Capabilities *plugin.Capabilities
	// ScanRoots contain the list of root dir used by file walking during extraction.
	// All extractors and detectors will assume files are relative to these dirs.
	// Example use case: Scanning a container image or source code repo that is
	// mounted to a local dir.
	ScanRoots []*scalibrfs.ScanRoot
	// Optional: Individual files to extract inventory from. If specified, the
	// extractors will only look at these files during the filesystem traversal.
	// Note that on real filesystems these are not relative to the ScanRoots and
	// thus need to be in sub-directories of one of the ScanRoots.
	FilesToExtract []string
	// Optional: Directories that the file system walk should ignore.
	// Note that on real filesystems these are not relative to the ScanRoots and
	// thus need to be in sub-directories of one of the ScanRoots.
	// TODO(b/279413691): Also skip local paths, e.g. "Skip all .git dirs"
	DirsToSkip []string
	// Optional: If the regex matches a directory, it will be skipped.
	SkipDirRegex *regexp.Regexp
	// Optional: If the glob matches a directory, it will be skipped.
	SkipDirGlob glob.Glob
	// Optional: stats allows to enter a metric hook. If left nil, no metrics will be recorded.
	Stats stats.Collector
	// Optional: Whether to read symlinks.
	ReadSymlinks bool
	// Optional: Limit for visited inodes. If 0, no limit is applied.
	MaxInodes int
	// Optional: By default, inventories stores a path relative to the scan root. If StoreAbsolutePath
	// is set, the absolute path is stored instead.
	StoreAbsolutePath bool
	// Optional: If true, print a detailed analysis of the duration of each extractor.
	PrintDurationAnalysis bool
	// Optional: If true, fail the scan if any permission errors are encountered.
	ErrorOnFSErrors bool
}

ScanConfig stores the config settings of a scan run such as the plugins to use and the dir to consider the root of the scanned system.

func (*ScanConfig) EnableRequiredExtractors

func (cfg *ScanConfig) EnableRequiredExtractors() error

EnableRequiredExtractors adds those extractors to the config that are required by enabled detectors but have not been explicitly enabled.

func (*ScanConfig) ValidatePluginRequirements

func (cfg *ScanConfig) ValidatePluginRequirements() error

ValidatePluginRequirements checks that the scanning environment's capabilities satisfy the requirements of all enabled plugin.

type ScanResult

type ScanResult struct {
	Version   string
	StartTime time.Time
	EndTime   time.Time
	// Status of the overall scan.
	Status *plugin.ScanStatus
	// Status and versions of the inventory+vuln plugins that ran.
	PluginStatus []*plugin.Status
	Inventories  []*extractor.Inventory
	Findings     []*detector.Finding
}

ScanResult stores the software inventory and security findings that a scan run found.

type Scanner

type Scanner struct{}

Scanner is the main entry point of the scanner.

func New

func New() *Scanner

New creates a new scanner instance.

func (Scanner) Scan

func (Scanner) Scan(ctx context.Context, config *ScanConfig) (sr *ScanResult)

Scan executes the extraction and detection using the provided scan config.

func (Scanner) ScanContainer

func (s Scanner) ScanContainer(ctx context.Context, img *image.Image, config *ScanConfig) (sr *ScanResult, err error)

ScanContainer scans the provided container image for inventory and security findings using the provided scan config. It populates the LayerDetails field of the inventory with the origin layer details. Functions to create an Image from a tarball, remote name, or v1.Image are available in the artifact/image/layerscanning/image package.