Skip to content

Automatically check component security in a build and generate a report

Context

If composition analysis runs only manually, vulnerabilities and policy violations are easy to notice too late, after a failed build or even after a release. Experienced teams embed this kind of check directly into the build pipeline so every run immediately shows the component set, detected vulnerabilities, and policy matches.

This is what the Johnny CLI agent is used for: it analyzes dependencies, saves results to CodeScoring, and saves an SBOM and reports for further work.

The scenario below uses GitLab CI as an example, but the same idea works for other pipelines where the Johnny agent can run as part of the build.

What you will get

By the end of this scenario, the .gitlab-ci.yml file will contain a dedicated sca job that:

  • runs the Johnny agent against the repository contents;
  • saves the results to a CodeScoring project;
  • produces bom.json and report.sarif as build artifacts.

Requirements

Before you start, make sure you have:

  • a GitLab repository with a working GitLab CI pipeline;
  • GitLab CI variables JOHNNY_API_URL and JOHNNY_API_TOKEN;
  • a CodeScoring license with the SCA module enabled;
  • access to CodeScoring where a project for Johnny results can be created automatically with --create-project or an existing project can be used;
  • a clear understanding of which policies should be applied to Johnny runs at the build stage.

Steps

Step 1. Prepare the agent binary in the build environment

Before adding the job to the pipeline, make sure the agent itself is available in the environment where the build runs.

  1. Open the [platform-url]/download/ page in your CodeScoring installation.
  2. If needed, check the current agent version at [platform-url]/download/johnny_version.
  3. Download the required executable file into the build environment, for example to /usr/local/bin/johnny.
  4. Allow it to run:
chmod +x /usr/local/bin/johnny

If you need a more detailed GitLab CI-specific example, use the Johnny Agent GitLab CI guide.

After that, the pipeline can call johnny as a regular executable command.

Step 2. Add a dedicated SCA job to .gitlab-ci.yml

The goal of this step is to move dependency checking into a standalone job and define which files should remain after the build finishes.

Example job in .gitlab-ci.yml

stages:
  - test

sca:
  stage: test

  script:
    - >
      johnny scan dir .
      --api_token $JOHNNY_API_TOKEN
      --api_url $JOHNNY_API_URL
      --project "billing-service-cli"
      --save-results
      --create-project
      --stage build
      --localization ru
      --format "coloredtable,sarif>>report.sarif"
      --ignore .git

  artifacts:
    paths:
      - bom.json
      - report.sarif
    when: always
    expire_in: 1 week

Key points in this job:

  • scan dir . runs analysis against the repository directory;
  • --project, --save-results, and --create-project save results to CodeScoring;
  • --stage build applies policies intended for Johnny Agent runs;
  • --format "coloredtable,sarif>>report.sarif" keeps console output and writes a sarif report;
  • the full list of flags and launch modes is available in the Johnny Agent launch guide;
  • bom.json is generated automatically by the agent and stored as an artifact together with report.sarif.

After this change, the pipeline gets a dedicated job that not only runs analysis but also leaves files that are ready for reporting and automation.

Step 3. Run the pipeline with the new job

Now the first real run should confirm three things at once: the command works, the artifacts are created, and the results are sent to the platform.

  1. Save the changes in .gitlab-ci.yml.
  2. Commit them and push the branch to GitLab.
  3. Wait for the sca job to start.

How to read the agent exit code

Exit code 1 means the Johnny Agent completed the scan and found issues that match security policies. It does not mean the agent crashed. The full description of exit codes and output formats is available in the Johnny agent launch guide.

After the first run, it becomes clear whether the current variables and permissions are enough for the job to reach the analysis and reporting steps.

Step 4. Review how a typical Johnny run looks and which files it leaves behind

Below is an interactive demo with an example job and a typical agent run.

After such a run, the build already leaves behind an SBOM and a machine-readable sarif report.

Step 5. Verify that the results are immediately usable

Automatic checking is only useful when the result can be consumed right away by the build, the platform, and later reporting steps.

Check that after the job finishes:

  1. The GitLab job artifacts contain:
    • bom.json;
    • report.sarif.
  2. A CodeScoring project named billing-service-cli exists if it was not created earlier.
  3. The latest run results are saved in that project.

Why these two files are enough

bom.json works well as an SBOM artifact for the build, while report.sarif is convenient for CI/CD consumers, application security tooling, and later automation.

At this stage, the build does not just perform a check; it leaves behind a practical set of outputs for analysis and reporting.

Result

The scenario is complete if:

  • .gitlab-ci.yml contains a dedicated sca job that launches Johnny;
  • the job produces bom.json and report.sarif as artifacts;
  • the Johnny Agent saves results to a CodeScoring project through --save-results;
  • the team understands that exit code 1 means policy matches were found rather than an agent crash.

After that, component security is checked automatically during the build, and the reports remain available both in CI and in the platform.

What next

Страница была полезна?