common

README

Analyzers Common Library

This repository contains Go packages you may use to create analyzers.

Current State - maintenance mode

With the work in GitLab#211819 it is recommended to use the following dedicated modules:

• analyzers/command - the command interface (replacing common/command)
• analyzers/report - Report and Finding structs for marshalling JSON reports (replacing common/issue)
• analyzers/template - template project for new analyzers (replacing common/template)

This project is in maintenance mode. As we transition away from a centralized module and towards separate projects, any future improvements should take into consideration whether the affected module should be extracted. In their current state, many of the modules are considered feature-complete and do not require updates, thereby not requiring extraction. This list includes the following modules:

• common/cacert
• common/logutil
• common/pathfilter
• common/plugin
• common/search
• common/walk

Logging Level

The default logging level is set to "info". You can control the level at which our logs are output by setting the SECURE_LOG_LEVEL env var to one of the following string values:

• panic
• fatal
• error
• warn
• info
• debug
• trace

Setting the log level allows that level and the other levels above it to output into the log. For example, "panic" is the most restrictive setting and "trace" the most permissive. If you were to set it to "error", then it would only output error, fatal, and panic level logs.

How to use the analyzers

Analyzers are shipped as Docker images. All analyzers based on this library can be used the same way. Here's an example using the spotbugs Docker image:

1. cd into the directory of the source code you want to scan

2. Run the Docker image:

   docker run \
     --interactive --tty --rm \
     --volume "$PWD":/tmp/app \
     --env CI_PROJECT_DIR=/tmp/app \
     registry.gitlab.com/gitlab-org/security-products/analyzers/spotbugs:${VERSION:-2} /analyzer run
   

   VERSION must be replaced with one of the available releases, see versioning.

3. The Docker container generates a report in the mounted project directory with a report filename corresponding to the analyzer category:

   Analyzer Category	Report Filename
   SAST	gl-sast-report.json
   Secret Detection	gl-secret-detection-report.json
   Dependency Scanning	gl-dependency-scanning-report.json

Analyzers development

To update the analyzer:

1. Modify the Go source code.
2. Compile a Linux binary.
3. Build a new Docker image.
4. Run the analyzer against its test project.
5. Compare the generated report with what's expected.

Here's how to create a Docker image named analyzer:

GOOS=linux go build -o analyzer
docker build -t analyzer.

docker run --rm \
  --volume "$PWD"/test/fixtures:/tmp/project \
  --env CI_PROJECT_DIR=/tmp/project \
  analyzer \
  /analyzer run

Then you can test it:

• for SAST analyzers:

  wget https://gitlab.com/gitlab-org/security-products/ci-templates/-/raw/master/scripts/compare_reports.sh
  sh ./compare_reports.sh sast test/fixtures/gl-sast-report.json test/expect/gl-sast-report.json \
  | patch -Np1 test/expect/gl-sast-report.json && git commit -m 'Update expectation' test/expect/gl-sast-report.json
  rm compare_reports.sh
  

• for Secret Detection:

  wget https://gitlab.com/gitlab-org/security-products/ci-templates/-/raw/master/scripts/compare_reports.sh
  sh ./compare_reports.sh sd test/fixtures/gl-secret-detection-report.json test/expect/gl-secret-detection-report.json \
  | patch -Np1 test/expect/gl-secret-detection-report.json && git commit -m 'Update expectation' test/expect/gl-secret-detection-report.json
  rm compare_reports.sh
  

• for Dependency Scanning analyzers:

  wget https://gitlab.com/gitlab-org/security-products/ci-templates/-/raw/master/scripts/compare_reports.sh
  sh ./compare_reports.sh ds test/fixtures/gl-dependency-scanning-report.json test/expect/gl-dependency-scanning.json
  | patch -Np1 test/expect/gl-dependency-scanning-report.json && git commit -m 'Update expectation' test/expect/gl-dependency-scanning-report.json
  rm compare_reports.sh
  

You can also compile the binary for your own environment and run it locally but analyze and run probably won't work since the runtime dependencies of the analyzer are missing.

Here's an illustration based on find-sec-bugs:

go build -o analyzer
./analyzer search test/fixtures
./analyzer convert test/fixtures/app/spotbugsXml.Xml > ./gl-sast-report.json

Testing Local Changes

If you want to test local changes in common from an analyzer you can do this by using the go mod replace command so that it loads common with your local changes instead of using the version of common that has been tagged remotely. Ex:

go mod edit -replace gitlab.com/gitlab-org/security-products/analyzers/common/v3=/local/path/to/common

Alternatively you can achieve the same result by manually updating the go.mod file:

module gitlab.com/gitlab-org/security-products/analyzers/awesome-analyzer/v2

replace gitlab.com/gitlab-org/security-products/analyzers/common/v3 => /path/to/common

require (
    ...
    gitlab.com/gitlab-org/security-products/analyzers/common/v3 v2.19.0
)

Testing Local Changes in Docker

If you want to use docker with replace in the go.mod file follow these steps:

1. Copy the contents of common into the directory of the analyzer. cp -r /path/to/common path/to/analyzer/common.
2. Add a copy statement in the analyzer's Dockerfile: COPY common /common.
3. Update the replace statement to make sure it matches the destination of the COPY statement in the step above: replace gitlab.com/gitlab-org/security-products/analyzers/common/v3 => /common

How to use the library

Analyzer relies on the command Go package to implement a command line.

How to tests the analyzers

Video walkthrough of how Dependency Scanning analyzers are using multi project pipeline feature to test analyzers using test projects. We create test projects by following the procedure defined in test/commons

Versioning and release process

Analyzers are independent projects that follow their own versioning. Though to ensure compatibility between analyzers, common API and orchestrators that uses them, they all share the same MAJOR version number. Minor and Patch can be bumped as long as backward compatibility with the common API is kept. Patch version bumps tend to correspond to a Minor version bump of the underlying tools (i.e. bandit), allowing us greater flexibility in reserving Minor bumps for more significant changes to our scanners. In case of breaking changes imposed by the wrapped scanner, creating a new analyzer on a separate repository must be considered.

One important outcome of common API and analyzers sharing the same MAJOR version number is that when a new analyzer project is started, its MAJOR version should be equal to the current one of the common library.

The analyzers are released as Docker images following this scheme:

• each push to the master branch will override the edge image tag
• each push to any awesome-feature branch will generate a matching awesome-feature image tag
• each git tag will generate the corresponding Major.Minor.Patch image tag. A manual job allows to override the corresponding Major and the latest image tags to point to this Major.Minor.Patch.

To release a new analyzer Docker image, there are two different options:

1. Manual release process
2. Automatic release process

Manual release process

1. Ensure that the CHANGELOG.md entry for the new analyzer is correct.
2. Ensure that the release source (typical the master or main branch) has a passing pipeline.
3. Create a new release for the analyzer project by selecting the Deployments menu on the left-hand side of the project window, then selecting the Releases sub-menu.
4. Click the New release button to open the New Release page.
   1. In the Tag name drop down, enter the same version used in the CHANGELOG.md, for example v2.4.2 and select the option to create the tag (Create tag v2.4.2 here).
   2. In the Release title enter the same version used above, for example v2.4.2.
   3. In the Release notes field, copy and paste the notes from the corresponding version in the CHANGELOG.md.
   4. Leave all other settings as the default values.
   5. Click on the Create release button.

After following the above process and creating a new release, a new git tag will be created with the Tag name provided above. This will trigger a new pipeline with the given tag version and a new analyzer Docker image will be built.

If the analyzer uses the analyzer.yml template, then the pipeline triggered as part of the New release process above will automatically tag and deploy a new version of the analyzer Docker image.

If the analyzer does not use the analyzer.yml template, you'll need to manually tag and deploy a new version of the analyzer Docker image using these steps:

1. Click on the CI/CD menu on the left-hand side of the project window, then select the Pipelines sub-menu.
2. A new pipeline should currently be running with the same tag used previously, for example v2.4.2
3. Once the pipeline has completed, it will be in a blocked state.
4. Click on the Manual job play button on the right hand side of the window and select tag version to tag and deploy a new version of the analyzer Docker image

Use your best judgment to decide when to create a git tag, which will then trigger the release job. If you can't decide, then ask for other's input.

Automatic release process

The following must be performed before the automatic release process can be used:

1. Configure CREATE_GIT_TAG: true as a CI/CD environment variable.
2. Check the Variables in the CI/CD project settings. Unless the project already inherits the GITLAB_TOKEN environment variable from the project group, create a project access token with complete read/write access to the API and configure GITLAB_TOKEN as a CI/CD environment variable which refers to this token.

Once the above steps have been completed, the automatic release process executes as follows:

1. A project maintainer merges an MR into the default branch.
2. The default pipeline is triggered, and the upsert git tag job is executed.
   • If the most recent version in the CHANGELOG.md matches one of the git tags, the job is a no-op.
   • Else, this job automatically creates a new release and git tag using the releases API. The version and message is obtained from the most recent entry in the CHANGELOG.md file for the project.
3. A pipeline is automatically triggered for the new git tag. This pipeline releases the latest, major, minor and patch Docker images of the analyzer.

Steps to perform after releasing an analyzer

1. Once a new version of the analyzer Docker image has been tagged and deployed, please test it with the corresponding test project.

2. Announce the release on the relevant group slack channel. Example message:

   FYI I've just released ANALYZER_NAME ANALYZER_VERSION. LINK_TO_RELEASE

Never delete a git tag that has been pushed as there is a good chance that the tag will be used and/or cached by the Go package registry.

Contributing

Contributions are welcome, see CONTRIBUTING.md for more details.

License

This code is distributed under the MIT Expat license, see the LICENSE file.