BIPI
BIPI

Container Image Scanning: Drowning in CVEs, Missing the Real Risk

Cloud Security

Trivy and Grype will happily report 500 CVEs per base image. Most are unreachable. VEX, reachability analysis, and base image rationalisation get the noise down to what actually matters.

By Arjun Raghavan, Security & Systems Lead, BIPI · February 16, 2024 · 7 min read

#container-security#vulnerability-management#trivy

A fresh ubuntu:22.04 image scanned with Trivy in early 2024 returns roughly 80 vulnerabilities. A typical Node.js application image based on node:20-alpine returns 150. A Python data science image with scikit-learn and pandas can hit 500. If your CI pipeline blocks on 'no critical or high CVEs', you are blocking every build forever and your developers stop reading the output.

Why scanners over-report

Image scanners (Trivy, Grype, Snyk Container, Aqua Trivy, AWS Inspector v2) work by enumerating installed packages and looking up known CVEs against package versions. They flag every CVE that has been published against a package version, regardless of whether the vulnerable code path is reachable in your application.

Three sources of false positives we see constantly:

  • Distro packages: openssl in the base image has 5 CVEs, but your application uses Python's hashlib and never touches openssl directly.
  • Transitive language deps: lodash inside a deeply nested webpack chunk for a feature flag library that is not even bundled into production.
  • Old upstream not yet patched in your distro: a Debian package has not picked up the upstream fix yet, so the scanner reports it as vulnerable even though the maintainer applied a backport.

Reachability analysis cuts the noise

Reachability tools (Snyk Reachable Vulns, Endor Labs, Apiiro, Semgrep Supply Chain) parse the application's call graph and only flag CVEs where the vulnerable function is actually called from application code. The reduction is dramatic. A 500-CVE image with 300 'high' severity findings typically drops to 5-20 actually reachable ones.

Reachability is not free. Static call graph analysis is slow (minutes per image) and has its own false positives and negatives (dynamic dispatch, reflection, plugin loading). But the signal-to-noise improvement makes it the highest-leverage thing you can add to a container security pipeline.

VEX is finally getting traction

VEX (Vulnerability Exploitability eXchange) is a format for saying 'CVE-X applies to component Y but is not exploitable in product Z for reason R'. Vendors publish VEX documents alongside SBOMs, and scanners that consume VEX suppress the suppressed findings.

Adoption is still patchy in early 2024. Red Hat publishes VEX for their UBI images. Microsoft publishes VEX for some products. Most upstream open source does not yet. But the format is now standardized (CSAF VEX, OpenVEX), and Trivy 0.49+ consumes VEX documents. We are starting to publish VEX for our own product images so customers' scanners stop flagging us for unreachable transitive CVEs.

Base image rationalisation

The single biggest reduction in CVE count comes from picking the right base image. Comparison for a Node.js application image:

150
CVEs on node:20-alpine
80
CVEs on node:20-slim (Debian)
12
CVEs on chainguard/node (distroless)
3
CVEs on cgr.dev/chainguard/node:latest

Chainguard Images (and Google's distroless, and Bitnami minideb) ship with the bare minimum needed to run the application. No shell, no package manager, no man pages. The attack surface and CVE count drop accordingly. Migration is moderately painful (debugging requires kubectl debug with an ephemeral container) but the maintenance cost drops dramatically.

Policies that ship, not block

The policy we land on with most clients:

  1. Scan every image at build time. Publish results to a central dashboard.
  2. Block builds only on critical CVEs with known active exploitation (CISA KEV list).
  3. Track high and critical CVEs with reachability. Open a ticket if a reachable finding ages past 14 days.
  4. Auto-rebuild base images weekly. Most CVEs disappear on the next base image update.
  5. Quarterly base image rationalisation. Reduce the matrix of images in use.

Runtime is the other half

Image scanning catches vulnerabilities in known packages. It does not catch misconfigured containers, secrets baked into layers, or runtime behavior. Pair image scanning with admission controllers (Kyverno, OPA Gatekeeper) for configuration, secret scanners (TruffleHog, Gitleaks on image layers) for embedded secrets, and runtime detection (Falco, Tetragon) for behavior. No one tool covers the whole problem.

Read more field notes, explore our services, or get in touch at info@bipi.in. Privacy Policy · Terms.