claircore

README

ClairCore

ClairCore provides a set of go modules which handle scanning container layers for installed packages and reporting any discovered vulnerabilities.
ClairCore is designed to be embedded into a service wrapper.

Usage

Two packages exist libindex and libvuln.
These modules export the methods for scanning and image for packages and matching the results of the scan to vulnerabilities respectively.

libindex usage

Creating an instance

opts := &libindex.Opts{
    DataStore: libindex.Postgres,
    ConnString: "postgres://host:port",
    ScanLock: libindex.PostgresSL,
    // see definition for more configuration options
}
lib := libindex.New(opts)

call libindex with a populated Manifest

m := &claircore.Manifest{
    ...
}

ir, err := lib.Index(m)
if err != nil {
    log.Printf("%v", err)
}
if ir.State == "ScanError" {
    log.Printf("scan failed: %s", sr.Err)
}

libvuln usage

creating an instance  

opts := &libvuln.Opts{ DataStore: libvuln.Postgres, ConnString: "postgres://host:port", ScanLock: libindex.PostgresSL, // see definition for more configuration option } lib := libvuln.New(opts)

call libvuln with a populated IndexReport  

sr := &claircore.IndexReport{ ... } vr, err := libvuln.Scan(sr) if err != nil { log.Printf("%v", err) }

Libvuln will first initialize all updaters before returning from its constructor.  
Controlling how many updaters initialize in parallel is provided via the libvuln.Opts struct  

# Local development and testing

The included makefile has targets for spawning a libindex and a libvuln database instance.  

make libindex-db-restart make libvuln-db-restart

After both targets are ran you may run integration tests  

make integration

unit tests do not require a database  

make unit

## Dev servers

ClairCore provides two http servers for local development and quick testing/hacking.  
You may build these from `./cmd/libindexhttp` and `./cmd/libvulnhttp`  

## Running libindex on darwin/MacOS

Layer stacking will fail on Darwin/MacOS with a file permissions issue and subsequently fail the scanner.   
In order to get around this the layer stacking integration test has a build tag "unix".  
The makefile target `integration-unix` runs tests that will only pass on a unix env.   
You may use the make target 'docker-shell' to drop into a linux shell where `make integration-unix` may be ran.  

# Deeper dives

[Highlevel Architecture](./docs/highlevel_arch.md)  
[Matching Architecture](./docs/matching_arch.md)  
[Vulnerability Matching](./docs/matching_vulns.md)  
[Vulnerability Tombstoning](./docs/tombstoning.md)  
[Content-Addressability](./docs/content_addressability.md)  
[Libindex Data Model](./docs/scanner_data_model.md)  
[Indexer States](./docs/indexer_states.md)  
[Local Development](./docs/local-dev.md)  

Documentation

Index

• type Distribution
• type Environment
• type IndexRecord
• type IndexReport
  • func (report *IndexReport) IndexRecords() []*IndexRecord
• type Layer
  • func (l *Layer) Files(paths []string) (map[string][]byte, error)
  • func (l *Layer) Reader() (io.ReadCloser, error)
• type Manifest
• type Package
• type RemotePath
• type Repository
• type Vulnerability
• type VulnerabilityReport

Types

type Distribution

type Distribution struct {
	// unique ID of this distribution. this will be created as discovered by the library
	// and used for persistence and hash map indexes.
	ID int `json:"id"`
	// A lower-case string (no spaces or other characters outside of 0–9, a–z, ".", "_" and "-") identifying the operating system, excluding any version information
	// and suitable for processing by scripts or usage in generated filenames. Example: "DID=fedora" or "DID=debian".
	DID string `json:"did"`
	// A string identifying the operating system.
	// example: "Ubuntu"
	Name string `json:"name"`
	// A string identifying the operating system version, excluding any OS name information,
	// possibly including a release code name, and suitable for presentation to the user.
	// example: "16.04.6 LTS (Xenial Xerus)"
	Version string `json:"version"`
	// A lower-case string (no spaces or other characters outside of 0–9, a–z, ".", "_" and "-") identifying the operating system release code name,
	// excluding any OS name information or release version, and suitable for processing by scripts or usage in generated filenames
	// example: "xenial"
	VersionCodeName string `json:"version_code_name"`
	// A lower-case string (mostly numeric, no spaces or other characters outside of 0–9, a–z, ".", "_" and "-")
	// identifying the operating system version, excluding any OS name information or release code name,
	// example: "16.04"
	VersionID string `json:"version_id"`
	// A string identifying the OS architecture
	// example: "x86_64"
	Arch string `json:"arch"`
	// Optional common platform enumeration identifier
	CPE string `json:"cpe"`
	// A pretty operating system name in a format suitable for presentation to the user.
	// May or may not contain a release code name or OS version of some kind, as suitable. If not set, defaults to "PRETTY_NAME="Linux"".
	// example: "PRETTY_NAME="Fedora 17 (Beefy Miracle)"".
	PrettyName string `json:"pretty_name"`
}

Distribution is the accompanying system context of a package. this information aides in CVE detection.

Distribution is modeled after the os-release file found in all linux distributions.

type Environment

type Environment struct {
	// the package database the associated package was discovered in
	PackageDB string `json:"package_db"`
	// the layer in which the associated package was introduced
	IntroducedIn string `json:"introduced_in"`
	// the ID of the distribution the package was discovered on
	DistributionID int `json:"distribution_id,string"`
	// the ID of the repository where this package was downloaded from (currently not used)
	RepositoryID int `json:"repository_id,string"`
}

Environment describes the surrounding environment a package was discovered in.

Environment must be accompanied by a parent structure which maps IDs to data models in order to have meaning. In our case this is IndexReport or VulnerabilityReport.

type IndexRecord

type IndexRecord struct {
	Package      *Package
	Distribution *Distribution
	Repository   *Repository
}

IndexRecord is an entry in the IndexReport.

IndexRecords provide full access to contextual package structures such as Distribution and Repository.

A list of these can be thought of as an "unpacked" IndexReport

type IndexReport

type IndexReport struct {
	// the manifest hash this IndexReport is describing
	Hash string `json:"manifest_hash"`
	// the current state of the index operation
	State string `json:"state"`
	// all discovered packages in this manifest key'd by package id
	Packages map[int]*Package `json:"packages"`
	// all discovered distributions in this manifest key'd by distribution id
	Distributions map[int]*Distribution `json:"distributions"`
	// all discovered repositories in this manifest key'd by repository id
	Repositories map[int]*Repository `json:"repository"`
	// a list of environment details a package was discovered in key'd by package id
	Environments map[int][]*Environment `json:"environments"`
	// whether the index operation finished successfully
	Success bool `json:"success"`
	// an error string in the case the index did not succeed
	Err string `json:"err"`
}

IndexReport provides a database for discovered artifacts in an image.

IndexReports make heavy usage of lookup maps to associate information without repetition.

func (*IndexReport) IndexRecords

func (report *IndexReport) IndexRecords() []*IndexRecord

IndexRecords returns a list of IndexRecords derived from the IndexReport

type Layer

type Layer struct {
	// content addressable hash unequally identifying this layer. libindex will treat layers with this same
	// hash as identical.
	Hash string `json:"hash"`
	// format of the archived layer. currently we support tar with Gzip, Bzip2, and Xz compression. compression
	// format will be determined via moby library.
	Format string `json:"format"`
	// the format of this image. typically this is the container technology which created the image.
	ImageFormat string `json:"image_format"`
	// uncompressed tar archive of the layer's content read into memory
	Bytes []byte `json:"-"`
	// path to local file containing uncompressed tar archive of the layer's content
	LocalPath string `json:"-"`
	// the URI and header information for retrieving a layer via http
	RemotePath RemotePath `json:"remote_path"`
}

Layer is an containers image filesystem layer. Layers are stacked ontop of each other to comprise the final filesystem of the container image.

func (*Layer) Files

func (l *Layer) Files(paths []string) (map[string][]byte, error)

Files retrieves specific files from the tar archive. concurrency safe as we only read. if file is not found an empty byte array will be returned as value in the map key'd by the path name.

func (*Layer) Reader

func (l *Layer) Reader() (io.ReadCloser, error)

Reader returns a ReadCloser of the layer.

It should also implement io.Seeker, and should be a tar stream.

type Manifest

type Manifest struct {
	// content addressable hash. should be able to be computed via
	// the hashes of all included layers
	Hash string `json:"hash"`
	// an array of filesystem layers indexed in the same order as the cooresponding image
	Layers []*Layer `json:"layers"`
}

Manifest represents a docker image. Layers array MUST be indexed in the order that image layers are stacked.

type Package

type Package struct {
	// unique ID of this package. this will be created as discovered by the library
	// and used for persistence and hash map indexes
	ID int `json:"id"`
	// the name of the package
	Name string `json:"name"`
	// the version of the package
	Version string `json:"version"`
	// type of package. currently expectations are binary or source
	Kind string `json:"kind,omitempty"`
	// if type is a binary package a source package maybe present which built this binary package.
	// must be a pointer to support recursive type:
	Source *Package `json:"source,omitempty"`
	// the file system path or prefix where this package resides
	PackageDB string `json:"-"`
	// a hint on which repository this package was downloaded from
	RepositoryHint string `json:"-"`
}

type RemotePath

type RemotePath struct {
	URI     string              `json:"uri"`
	Headers map[string][]string `json:"headers"`
}

RemotePath provides http retrieval information about a layer.

type Repository

type Repository struct {
	ID   int    `json:"id"`
	Name string `json:"name"`
	Key  string `json:"key"`
	URI  string `json:"uri"`
}

Repository is a package repository

type Vulnerability

type Vulnerability struct {
	// unique ID of this vulnerability. this will be created as discovered by the library
	// and used for persistence and hash map indexes
	ID int `json:"id"`
	// the updater that discovered this vulnerability
	Updater string `json:"updater"`
	// the name of the vulnerability. for example if the vulnerability exists in a CVE database this
	// would the unique CVE name such as CVE-2017-11722
	Name string `json:"name"`
	// the description of the vulnerability
	Description string `json:"description"`
	// any links to more details about the vulnerability
	Links string `json:"links"`
	// the severity of the vulnerability
	Severity string `json:"severity"`
	// the package information associated with the vulnerability. ideally these fields can be matched
	// to packages discovered by libindex PackageScanner structs.
	Package *Package `json:"-"`
	// the distribution information associated with the vulnerability.
	Dist *Distribution `json:"-"`
	// the repository information associated with the vulnerability
	Repo *Repository `json:"-"`
	// a string specifying the package version the fix was relased in
	FixedInVersion string `json:"fixed_in_version"`
}

type VulnerabilityReport

type VulnerabilityReport struct {
	// the manifest hash this vulnerability report is describing
	Hash string `json:"manifest_hash"`
	// all discovered packages in this manifest keyed by package id
	Packages map[int]*Package `json:"packages"`
	// all discovered distributions in this manifest keyed by distribution id
	Distributions map[int]*Distribution `json:"distributions"`
	// all discovered repositories in this manifest keyed by repository id
	Repositories map[int]*Repository `json:"repository"`
	// a list of environment details a package was discovered in keyed by package id
	Environments map[int][]*Environment `json:"environments"`
	// all discovered vulnerabilities affecting this manifest
	Vulnerabilities map[int]*Vulnerability `json:"vulnerabilities"`
	// a lookup table associating package ids with 1 or more vulnerability ids. keyed by package id
	PackageVulnerabilities map[int][]string `json:"package_vulnerabilities"`
}

VulnerabilityReport provides a report of packages and their associated vulnerabilities.