claircore

README

ClairCore

ClairCore provides a set of go modules which handle scanning container layers for installed packages and reporting any discovered vulnerabilities.
ClairCore is designed to be embedded into a service wrapper.

For a full overview see: ClairCore Book

Local development and testing

The following targets start and stop a local development environment

make local-dev-up
make local-dev-down

If you modify libvuln or libindex code the following make targets will restart the services with your changes

make libindexhttp-restart
make libvulnhttp-restart

With the local development environment up the following make target runs all tests including integration

make integration

The following make target runs unit tests which do not require a database or local development environment

make unit

Documentation

Overview

Package claircore has foundational types for the claircore module.

Additional documentation can be found at http://quay.github.io/claircore/

Index

• Constants
• Variables
• func VersionSort(vs []Version) func(int, int) bool
• type AffectedManifests
  • func NewAffectedManifests() AffectedManifests
  • func (a *AffectedManifests) Add(v *Vulnerability, digests ...Digest)
  • func (a *AffectedManifests) Sort()
• type ArchOp
  • func (o ArchOp) Cmp(a, b string) bool
  • func (o ArchOp) MarshalText() (text []byte, err error)
  • func (o *ArchOp) Scan(i interface{}) error
  • func (i ArchOp) String() string
  • func (o *ArchOp) UnmarshalText(text []byte) error
  • func (o ArchOp) Value() (driver.Value, error)
• type CheckVulnernableFunc
• type Digest
  • func MustParseDigest(digest string) Digest
  • func NewDigest(algo string, sum []byte) (Digest, error)
  • func ParseDigest(digest string) (Digest, error)
  • func (d Digest) Algorithm() string
  • func (d Digest) Checksum() []byte
  • func (d Digest) Hash() hash.Hash
  • func (d Digest) MarshalText() ([]byte, error)
  • func (d *Digest) Scan(i interface{}) error
  • func (d Digest) String() string
  • func (d *Digest) UnmarshalText(t []byte) error
  • func (d Digest) Value() (driver.Value, error)
• type DigestError
  • func (e *DigestError) Error() string
  • func (e *DigestError) Unwrap() error
• type Distribution
• type Environment
• type File
• type FileKind
• type IndexRecord
• type IndexReport
  • func (report *IndexReport) IndexRecords() []*IndexRecord
• type Layer
  • func (l *Layer) Close() error
  • func (l *Layer) FS() (fs.FS, error)
  • func (l *Layer) Fetched() bool
  • func (l *Layer) Files(paths ...string) (map[string]*bytes.Buffer, error)
  • func (l *Layer) Init(ctx context.Context, desc *LayerDescription, r io.ReaderAt) error
  • func (l *Layer) Reader() (ReadAtCloser, error)
  • func (l *Layer) SetLocal(_ string) error
• type LayerDescription
• type Manifest
• type Package
• type Range
  • func (r *Range) Contains(v *Version) bool
• type ReadAtCloser
• type Repository
• type Severity
  • func (s *Severity) MarshalText() ([]byte, error)
  • func (s *Severity) Scan(i interface{}) error
  • func (i Severity) String() string
  • func (s *Severity) UnmarshalText(b []byte) error
  • func (s Severity) Value() (driver.Value, error)
• type Version
  • func FromSemver(v *semver.Version) (out Version)
  • func (v *Version) Compare(x *Version) int
  • func (v *Version) MarshalText() ([]byte, error)
  • func (v *Version) String() string
  • func (v *Version) UnmarshalText(text []byte) (err error)
• type Vulnerability
• type VulnerabilityReport

Constants

const (
	SHA256 = "sha256"
	SHA512 = "sha512"
)

const (
	BINARY = "binary"
	SOURCE = "source"
)

const (
	FileKindWhiteout = FileKind("whiteout")
)

Variables

var ErrNotFound = errors.New("claircore: unable to find any requested files")

ErrNotFound is returned by Layer.Files if none of the requested files are found.

Deprecated: The Layer.Files method is deprecated.

Functions

func VersionSort

func VersionSort(vs []Version) func(int, int) bool

VersionSort returns a function suitable for passing to sort.Slice or sort.SliceStable.

Types

type AffectedManifests

type AffectedManifests struct {

	// map of vulnerabilities keyed by the vulnerability's ID
	Vulnerabilities map[string]*Vulnerability `json:"vulnerabilities"`
	// map associating a list of vulnerability ids keyed by the
	// manifest hash they affect.
	VulnerableManifests map[string][]string `json:"vulnerable_manifests"`
	// contains filtered or unexported fields
}

AffectedManifests describes a set of manifests affected by a set of Vulnerabilities.

func NewAffectedManifests

func NewAffectedManifests() AffectedManifests

NewAffectedManifests initializes a new AffectedManifests struct.

func (*AffectedManifests) Add

func (a *AffectedManifests) Add(v *Vulnerability, digests ...Digest)

Add will add the provided Vulnerability and Manifest digest to the necessary maps.

Add is safe to use by multiple goroutines.

func (*AffectedManifests) Sort

func (a *AffectedManifests) Sort()

Sort will sort each array in the VulnerableManifests map by Vulnerability.NormalizedSeverity in Desc order.

Sort is safe to use by multiple goroutines.

type ArchOp

type ArchOp uint

const (
	OpEquals       ArchOp // equals
	OpNotEquals           // not equals
	OpPatternMatch        // pattern match
)

func (ArchOp) Cmp

func (o ArchOp) Cmp(a, b string) bool

func (ArchOp) MarshalText

func (o ArchOp) MarshalText() (text []byte, err error)

func (*ArchOp) Scan

func (o *ArchOp) Scan(i interface{}) error

func (ArchOp) String

func (i ArchOp) String() string

func (*ArchOp) UnmarshalText

func (o *ArchOp) UnmarshalText(text []byte) error

func (ArchOp) Value

func (o ArchOp) Value() (driver.Value, error)

type CheckVulnernableFunc

type CheckVulnernableFunc func(ctx context.Context, record *IndexRecord, vuln *Vulnerability) (bool, error)

CheckVulnernableFunc takes a vulnerability and an indexRecord and checks if the record is vulnerable to the vulnerability, it is by the Querier.AffectedManifests method and allows a backdoor to introduce application filtering logic into the DB layer.

type Digest

type Digest struct {
	// contains filtered or unexported fields
}

Digest is a type representing the hash of some data.

It's used throughout claircore packages as an attempt to remain independent of a specific hashing algorithm.

func MustParseDigest

func MustParseDigest(digest string) Digest

MustParseDigest works like ParseDigest but panics if the provided string is not well-formed.

func NewDigest

func NewDigest(algo string, sum []byte) (Digest, error)

NewDigest constructs a Digest.

func ParseDigest

func ParseDigest(digest string) (Digest, error)

ParseDigest constructs a Digest from a string, ensuring it's well-formed.

func (Digest) Algorithm

func (d Digest) Algorithm() string

Algorithm returns a string representation of the algorithm used for this digest.

func (Digest) Checksum

func (d Digest) Checksum() []byte

Checksum returns the checksum byte slice.

func (Digest) Hash

func (d Digest) Hash() hash.Hash

Hash returns an instance of the hashing algorithm used for this Digest.

func (Digest) MarshalText

func (d Digest) MarshalText() ([]byte, error)

MarshalText implements encoding.TextMarshaler.

func (*Digest) Scan

func (d *Digest) Scan(i interface{}) error

Scan implements sql.Scanner.

func (Digest) String

func (d Digest) String() string

func (*Digest) UnmarshalText

func (d *Digest) UnmarshalText(t []byte) error

UnmarshalText implements encoding.TextUnmarshaler.

func (Digest) Value

func (d Digest) Value() (driver.Value, error)

Value implements driver.Valuer.

type DigestError

type DigestError struct {
	// contains filtered or unexported fields
}

DigestError is the concrete type backing errors returned from Digest's methods.

func (*DigestError) Error

func (e *DigestError) Error() string

Error implements error.

func (*DigestError) Unwrap

func (e *DigestError) Unwrap() error

Unwrap enables errors.Unwrap.

type Distribution

type Distribution struct {
	// unique ID of this distribution. this will be created as discovered by the library
	// and used for persistence and hash map indexes.
	ID string `json:"id"`
	// A lower-case string (no spaces or other characters outside of 0–9, a–z, ".", "_" and "-") identifying the operating system, excluding any version information
	// and suitable for processing by scripts or usage in generated filenames. Example: "DID=fedora" or "DID=debian".
	DID string `json:"did"`
	// A string identifying the operating system.
	// example: "Ubuntu"
	Name string `json:"name"`
	// A string identifying the operating system version, excluding any OS name information,
	// possibly including a release code name, and suitable for presentation to the user.
	// example: "16.04.6 LTS (Xenial Xerus)"
	Version string `json:"version"`
	// A lower-case string (no spaces or other characters outside of 0–9, a–z, ".", "_" and "-") identifying the operating system release code name,
	// excluding any OS name information or release version, and suitable for processing by scripts or usage in generated filenames
	// example: "xenial"
	VersionCodeName string `json:"version_code_name"`
	// A lower-case string (mostly numeric, no spaces or other characters outside of 0–9, a–z, ".", "_" and "-")
	// identifying the operating system version, excluding any OS name information or release code name,
	// example: "16.04"
	VersionID string `json:"version_id"`
	// A string identifying the OS architecture
	// example: "x86_64"
	Arch string `json:"arch"`
	// Optional common platform enumeration identifier
	CPE cpe.WFN `json:"cpe"`
	// A pretty operating system name in a format suitable for presentation to the user.
	// May or may not contain a release code name or OS version of some kind, as suitable. If not set, defaults to "PRETTY_NAME="Linux"".
	// example: "PRETTY_NAME="Fedora 17 (Beefy Miracle)"".
	PrettyName string `json:"pretty_name"`
}

Distribution is the accompanying system context of a package. this information aides in CVE detection.

Distribution is modeled after the os-release file found in all linux distributions.

type Environment

type Environment struct {
	// the package database the associated package was discovered in
	PackageDB string `json:"package_db"`
	// the layer in which the associated package was introduced
	IntroducedIn Digest `json:"introduced_in"`
	// the ID of the distribution the package was discovered on
	DistributionID string `json:"distribution_id"`
	// the ID of the repository where this package was downloaded from (currently not used)
	RepositoryIDs []string `json:"repository_ids"`
}

Environment describes the surrounding environment a package was discovered in.

Environment must be accompanied by a parent structure which maps IDs to data models in order to have meaning. In our case this is IndexReport or VulnerabilityReport.

type File

type File struct {
	// Path is where in the layer filesystem the file is located.
	Path string
	// Kind is what kind of file was found.
	Kind FileKind
}

File represents interesting files that are found in the layer.

type FileKind

type FileKind string

FileKind is used to determine what kind of file was found.

type IndexRecord

type IndexRecord struct {
	Package      *Package
	Distribution *Distribution
	Repository   *Repository
}

IndexRecord is an entry in the IndexReport.

IndexRecords provide full access to contextual package structures such as Distribution and Repository.

A list of these can be thought of as an "unpacked" IndexReport

type IndexReport

type IndexReport struct {
	// the manifest hash this IndexReport is describing
	Hash Digest `json:"manifest_hash"`
	// the current state of the index operation
	State string `json:"state"`
	// all discovered packages in this manifest key'd by package id
	Packages map[string]*Package `json:"packages"`
	// all discovered distributions in this manifest key'd by distribution id
	Distributions map[string]*Distribution `json:"distributions"`
	// all discovered repositories in this manifest key'd by repository id
	Repositories map[string]*Repository `json:"repository"`
	// a list of environment details a package was discovered in key'd by package id
	Environments map[string][]*Environment `json:"environments"`
	// whether the index operation finished successfully
	Success bool `json:"success"`
	// an error string in the case the index did not succeed
	Err string `json:"err"`
	// Files doesn't end up in the json report but needs to be available at post-coalesce
	Files map[string]File `json:"-"`
}

IndexReport provides a database for discovered artifacts in an image.

IndexReports make heavy usage of lookup maps to associate information without repetition.

func (*IndexReport) IndexRecords

func (report *IndexReport) IndexRecords() []*IndexRecord

IndexRecords returns a list of IndexRecords derived from the IndexReport

type Layer

type Layer struct {

	// Hash is a content addressable hash uniquely identifying this layer.
	// Libindex will treat layers with this same hash as identical.
	Hash Digest `json:"hash"`
	// URI is a URI that can be used to fetch layer contents.
	//
	// Deprecated: This is exported for historical reasons and may stop being
	// populated in the future.
	URI string `json:"uri"`
	// Headers is additional request headers for fetching layer contents.
	//
	// Deprecated: This is exported for historical reasons and may stop being
	// populated in the future.
	Headers map[string][]string `json:"headers"`
	// contains filtered or unexported fields
}

Layer is an internal representation of a container image file system layer. Layers are stacked on top of each other to create the final file system of the container image.

This type being in the external API of the github.com/quay/claircore/libindex.Libindex type is a historical accident.

Previously, it was OK to use Layer literals. This is no longer allowed and the Layer.Init method must be called. Any methods besides Layer.Init called on an uninitialized Layer will report errors and may panic.

func (*Layer) Close

func (l *Layer) Close() error

Close releases held resources by this Layer.

Not calling Close may cause the program to panic.

func (*Layer) FS

func (l *Layer) FS() (fs.FS, error)

FS returns an fs.FS reading from an initialized layer.

func (*Layer) Fetched

func (l *Layer) Fetched() bool

Fetched reports whether the layer blob has been fetched locally.

Deprecated: Layers should now only be constructed by the code that does the fetching. That is, merely having a valid Layer indicates that the blob has been fetched.

func (*Layer) Files

func (l *Layer) Files(paths ...string) (map[string]*bytes.Buffer, error)

Files retrieves specific files from the layer's tar archive.

An error is returned only if none of the requested files are found.

The returned map may contain more entries than the number of paths requested. All entries in the map are keyed by paths that are relative to the tar-root. For example, requesting paths of "/etc/os-release", "./etc/os-release", and "etc/os-release" will all result in any found content being stored with the key "etc/os-release".

Deprecated: Callers should instead use fs.WalkDir with the fs.FS returned by Layer.FS.

func (*Layer) Init

func (l *Layer) Init(ctx context.Context, desc *LayerDescription, r io.ReaderAt) error

Init initializes a Layer in-place. This is provided for flexibility when constructing a slice of Layers.

func (*Layer) Reader

func (l *Layer) Reader() (ReadAtCloser, error)

Reader returns a ReadAtCloser of the layer.

It should also implement io.Seeker, and should be a tar stream.

func (*Layer) SetLocal

func (l *Layer) SetLocal(_ string) error

SetLocal is a namespacing wart.

Deprecated: This function unconditionally errors and does nothing. Use the Layer.Init method instead.

type LayerDescription

type LayerDescription struct {
	// Digest is a content addressable checksum uniquely identifying this layer.
	Digest string
	// URI is a URI that can be used to fetch layer contents.
	URI string
	// MediaType is the [OCI Layer media type] for this layer. Any [Indexer]
	// instance will support the OCI-defined media types, and may support others
	// based on its configuration.
	//
	// [OCI Layer media type]: https://github.com/opencontainers/image-spec/blob/main/layer.md
	MediaType string
	// Headers is additional request headers for fetching layer contents.
	Headers map[string][]string
}

LayerDescription is a description of a container layer. It should contain enough information to fetch the layer.

Unlike the Layer type, this type does not have any extra state or access to the contents of the layer.

type Manifest

type Manifest struct {
	// content addressable hash. should be able to be computed via
	// the hashes of all included layers
	Hash Digest `json:"hash"`
	// an array of filesystem layers indexed in the same order as the corresponding image
	Layers []*Layer `json:"layers"`
}

Manifest represents a docker image. Layers array MUST be indexed in the order that image layers are stacked.

type Package

type Package struct {
	// unique ID of this package. this will be created as discovered by the library
	// and used for persistence and hash map indexes
	ID string `json:"id"`
	// the name of the package
	Name string `json:"name"`
	// the version of the package
	Version string `json:"version"`
	// type of package. currently expectations are binary or source
	Kind string `json:"kind,omitempty"`
	// if type is a binary package a source package maybe present which built this binary package.
	// must be a pointer to support recursive type:
	Source *Package `json:"source,omitempty"`
	// the file system path or prefix where this package resides
	PackageDB string `json:"-"`
	// a location in the layer where the package is located, this is useful for language packages.
	Filepath string `json:"-"`
	// a hint on which repository this package was downloaded from
	RepositoryHint string `json:"-"`
	// NormalizedVersion is a representation of a version string that's
	// correctly ordered when compared with other representations from the same
	// producer.
	NormalizedVersion Version `json:"normalized_version,omitempty"`
	// Module and stream which this package is part of
	Module string `json:"module,omitempty"`
	// Package architecture
	Arch string `json:"arch,omitempty"`
	// CPE name for package
	CPE cpe.WFN `json:"cpe,omitempty"`
}

type Range

type Range struct {
	Lower Version `json:"["`
	Upper Version `json:")"`
}

Range is a half-open interval of two Versions.

In the usual notation, it is: [Lower, Upper)

func (*Range) Contains

func (r *Range) Contains(v *Version) bool

Contains reports whether the Version falls within the Range.

type ReadAtCloser

type ReadAtCloser interface {
	io.ReadCloser
	io.ReaderAt
}

ReadAtCloser is an io.ReadCloser and also an io.ReaderAt.

type Repository

type Repository struct {
	ID   string  `json:"id,omitempty"`
	Name string  `json:"name,omitempty"`
	Key  string  `json:"key,omitempty"`
	URI  string  `json:"uri,omitempty"`
	CPE  cpe.WFN `json:"cpe,omitempty"`
}

Repository is a package repository

type Severity

type Severity uint

const (
	Unknown Severity = iota
	Negligible
	Low
	Medium
	High
	Critical
)

func (*Severity) MarshalText

func (s *Severity) MarshalText() ([]byte, error)

func (*Severity) Scan

func (s *Severity) Scan(i interface{}) error

func (Severity) String

func (i Severity) String() string

func (*Severity) UnmarshalText

func (s *Severity) UnmarshalText(b []byte) error

func (Severity) Value

func (s Severity) Value() (driver.Value, error)

type Version

type Version struct {
	Kind string
	V    [10]int32
}

Version describes a revision of some sort that is ordered correctly within its "Kind".

Versions of different kinds do not have any sensible ordering.

func FromSemver

func FromSemver(v *semver.Version) (out Version)

FromSemver is the SemVer to claircore.Version mapping used by this package.

func (*Version) Compare

func (v *Version) Compare(x *Version) int

Compare returns an integer describing the relationship of two Versions.

The result will be 0 if a==b, -1 if a < b, and +1 if a > b. If the Versions are of different kinds, the Kinds will be compared lexographically.

func (*Version) MarshalText

func (v *Version) MarshalText() ([]byte, error)

MarshalText implments encoding.TextMarshaler.

func (*Version) String

func (v *Version) String() string

func (*Version) UnmarshalText

func (v *Version) UnmarshalText(text []byte) (err error)

UnmarshalText implments encoding.TextUnmarshaler.

type Vulnerability

type Vulnerability struct {
	// unique ID of this vulnerability. this will be created as discovered by the library
	// and used for persistence and hash map indexes
	ID string `json:"id"`
	// the updater that discovered this vulnerability
	Updater string `json:"updater"`
	// the name of the vulnerability. for example if the vulnerability exists in a CVE database this
	// would the unique CVE name such as CVE-2017-11722
	Name string `json:"name"`
	// the description of the vulnerability
	Description string `json:"description"`
	// the timestamp when vulnerability was issued
	Issued time.Time `json:"issued"`
	// any links to more details about the vulnerability
	Links string `json:"links"`
	// the severity string retrieved from the security database
	Severity string `json:"severity"`
	// a normalized Severity type providing client guaranteed severity information
	NormalizedSeverity Severity `json:"normalized_severity"`
	// the package information associated with the vulnerability. ideally these fields can be matched
	// to packages discovered by libindex PackageScanner structs.
	Package *Package `json:"package"`
	// the distribution information associated with the vulnerability.
	Dist *Distribution `json:"distribution,omitempty"`
	// the repository information associated with the vulnerability
	Repo *Repository `json:"repository,omitempty"`
	// a string specifying the package version the fix was released in
	FixedInVersion string `json:"fixed_in_version"`
	// Range describes the range of versions that are vulnerable.
	Range *Range `json:"range,omitempty"`
	// ArchOperation indicates how the affected Package's "arch" should be
	// compared.
	ArchOperation ArchOp `json:"arch_op,omitempty"`
}

type VulnerabilityReport

type VulnerabilityReport struct {
	// the manifest hash this vulnerability report is describing
	Hash Digest `json:"manifest_hash"`
	// all discovered packages in this manifest keyed by package id
	Packages map[string]*Package `json:"packages"`
	// all discovered distributions in this manifest keyed by distribution id
	Distributions map[string]*Distribution `json:"distributions"`
	// all discovered repositories in this manifest keyed by repository id
	Repositories map[string]*Repository `json:"repository"`
	// a list of environment details a package was discovered in keyed by package id
	Environments map[string][]*Environment `json:"environments"`
	// all discovered vulnerabilities affecting this manifest
	Vulnerabilities map[string]*Vulnerability `json:"vulnerabilities"`
	// a lookup table associating package ids with 1 or more vulnerability ids. keyed by package id
	PackageVulnerabilities map[string][]string `json:"package_vulnerabilities"`
	// a map of enrichments keyed by a type.
	Enrichments map[string][]json.RawMessage `json:"enrichments"`
}

VulnerabilityReport provides a report of packages and their associated vulnerabilities.