Want to Be a DevSecOps Engineer? Then You Need This Tool Stack!
Update (15 Oct 2026): On 19 March 2026, an attacker using compromised credentials published a malicious Trivy release (v0.69.4) and rewrote 76 of the 77 tags of
aquasecurity/trivy-action, plus all seven tags ofaquasecurity/setup-trivy, with credential-stealing code (GHSA-69fq-xp46-6x23, CVE-2026-33634). Any pipeline that used a rewritten tag during the exposure window ran that code with access to the job’s secrets. Separately, Kyverno 1.17 published thepolicies.kyverno.io/v1API and announced the retirement ofClusterPolicy; according to the policy types table,ImageValidatingPolicyis stable from 1.18,ClusterPolicyis deprecated in 1.19 and will be removed in 1.20.
Suppose a pentest two days before a release turns up three things: an AWS key in the Git history, a transitive dependency with a critical CVE, and an S3 bucket whose policy grants access to "Principal": "*". All three could have been caught automatically when they entered the repository, back when fixing them meant changing a few lines. Manual pentesting and DAST are still necessary, but at the end of the cycle they arrive too late for this kind of flaw.
My rule is to assign each control to the stage where it catches the flaw with the least noise, and to write down whether it blocks or only warns.
The stack by stage
| Stage | What it covers | Tools | Policy |
|---|---|---|---|
| Pre-commit (local) | Secrets in the change | gitleaks | Blocks the commit, but can be bypassed |
| Pull request | Secrets; SAST on new code | gitleaks, Semgrep, Opengrep, SonarQube | Blocks secrets and new high-severity SAST findings |
| Build | SBOM and dependency CVEs | Trivy, Syft, Grype, OSV-Scanner, Snyk | Blocks fixable HIGH and CRITICAL |
| Image | Base OS and package CVEs | Trivy, Grype | Blocks fixable HIGH and CRITICAL |
| IaC and manifests | Insecure configuration | trivy config, Checkov, KICS | Blocks HIGH and CRITICAL |
| Publish (registry) | Signature and attestation by digest | Cosign, attest-build-provenance | Required for admission |
| Admission and deploy | Signature, Pod Security, drift from Git | Kyverno, PSA, Argo CD, Flux | Admission blocks; drift is reverted only with selfHeal |
| Staging | Headers, cookies, web configuration | ZAP | Blocks only rules marked FAIL; everything else warns |
| Runtime | Anomalous process behavior | Falco, Tetragon | Alerting and response |
Warning means annotating the PR or opening an alert with an owner, without stopping the pipeline. A blocking gate needs very few false positives: every unjustified block teaches the team to work around it.
Secrets: gitleaks before the commit and in the PR
gitleaks searches Git patches for secrets. Locally, you install it as a hook with pre-commit, pinned to a commit (pre-commit autoupdate --freeze writes the hash instead of the tag):
repos:
- repo: https://github.com/gitleaks/gitleaks
rev: 83d9cd684c87d95d656c1458ef04895a7f1cbd8e # frozen: v8.30.1
hooks:
- id: gitleaks
Every developer has to run pre-commit install, and anyone can skip the hook with git commit --no-verify or SKIP=gitleaks. The hook gives fast feedback; the control that counts is the CI job that scans only the PR’s commits, plus a scheduled (schedule:) scan of the full history, which finds what got in before the gate was turned on or without going through a PR. Old findings that have already been rotated are excluded by their fingerprint in .gitleaksignore. If a secret has already reached a pushed commit, rotate it: rewriting history does not revoke the credential.
SAST on new code
Semgrep with --baseline-commit only reports findings that did not exist at the base commit, --severity ERROR limits the report to the highest-severity rules, and --error makes it exit with code 1 if it finds any. That way you block what the PR introduces without inheriting the debt of the whole repository. Opengrep is an LGPL-2.1 fork of Semgrep that accepts the same rules.
Under the October 2024 product names, pull request analysis is available in SonarQube Cloud and in SonarQube Server from Developer Edition up. In Jenkins, waitForQualityGate abortPipeline: true needs a webhook from SonarQube to <jenkins>/sonarqube-webhook/ so the pipeline can fail on the quality gate.
The main channel for findings is the PR itself: a SARIF file uploaded to code scanning shows up as an annotation on the affected line. On private repositories, code scanning requires GitHub Code Security. Jira is for accepted risk.
Dependencies, SBOM and image
An SBOM is an inventory of components, vulnerable or not; the vulnerability report comes from matching it against an advisory database, and it changes even when the SBOM does not. That is why you keep the SBOM for every release: it lets you answer the next Log4Shell by rescanning archived SBOMs instead of pulling every image. How to do that, and how to sign and attest the SBOM with Cosign, is covered in the supply chain post.
Trivy generates the SBOM with --format cyclonedx; in that mode it does not look for vulnerabilities, so the scan runs afterwards with trivy sbom. --ignore-unfixed shows only CVEs that have a fix, which keeps an unpatched CVE in the base image from blocking a build where the team cannot fix anything. Those CVEs are still there: they do not block the build, they go to triage with CISA KEV and EPSS, and you reduce them with a minimal base image pinned by digest. With Grype, the equivalent gate on the same SBOM is grype "sbom:./sbom.cdx.json" --fail-on high --only-fixed.
When the vulnerable package is a transitive dependency, the fix is usually to upgrade the direct dependency that pulls it in; if no version resolves it, overrides in package.json forces the version in npm.
Infrastructure as code
The tfsec repository recommends migrating to Trivy, which uses the same engine for Terraform. trivy config checks Terraform, CloudFormation, Dockerfiles, Kubernetes manifests and Helm charts. Checkov and KICS are alternatives.
Since April 2023, new buckets are created with S3 Block Public Access enabled and ACLs disabled (AWS announcement). Today, a public bucket takes at least two deliberate changes: setting block_public_policy = false and restrict_public_buckets = false in aws_s3_bucket_public_access_block, and attaching a policy with "Principal": "*". Trivy flags the first with checks AWS-0087 and AWS-0093, both HIGH. Both checks also fire if aws_s3_bucket_public_access_block is missing; I prefer to declare it explicitly rather than ignore the rule.
Publish, admission and GitOps
The image is signed by digest with Cosign in keyless mode, and whoever consumes it verifies the signer’s identity with cosign verify --certificate-identity https://github.com/org/repo/.github/workflows/release.yml@refs/heads/main --certificate-oidc-issuer https://token.actions.githubusercontent.com registry.example.com/app@sha256:<digest>. A signature proves who signed a digest; it does not prove how or where the image was built, which is what SLSA provenance covers. The same release.yml generates it with actions/attest-build-provenance, and you verify it with gh attestation verify.
At admission, a Kyverno policy (requires Kyverno 1.18 or later) rejects images without a valid signature from that identity. The full policy is in Hardening the supply chain. Pod Security Admission enforces the restricted profile per namespace. PSA has been stable since Kubernetes 1.25, the release that removed PodSecurityPolicy:
apiVersion: v1
kind: Namespace
metadata:
name: payments
labels:
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/enforce-version: v1.37
pod-security.kubernetes.io/warn: restricted
pod-security.kubernetes.io/audit: restricted
Admission evaluates the request when the pod is created or modified; it does not see what the process does afterwards. That is runtime, and I cover it in the Kubernetes post, along with the reason for pinning enforce-version to the cluster’s minor version.
Argo CD and Flux reconcile the cluster with Git. In Argo CD, syncPolicy.automated.selfHeal: true reverts changes made directly in the cluster; it is off by default, and drift persists until the next sync. For Git to be the auditable record of changes, also take cluster write access away from people via RBAC and protect the branch with required reviews; without protection, a force-push rewrites history. In the standard install, the Argo CD controller’s ClusterRole grants * on all resources, so its access belongs in the threat model.
Staging and runtime
In staging, ZAP (known as ZAP by Checkmarx since September 2024) runs the baseline scan from its official image:
#!/usr/bin/env bash
set -euo pipefail
# Baseline: spider + passive scan, no attacks.
# Only rules set to FAIL in .zap/rules.tsv exit 1; -I keeps WARN from failing.
docker run --rm -t -v "$(pwd)/.zap:/zap/wrk/:rw" \
ghcr.io/zaproxy/zaproxy:2.17.0@sha256:7aaa659b0d43078febd82e29bad112285c370727e86ab8340444220e17d9f0d2 \
zap-baseline.py -t https://staging.example.com -c rules.tsv -I
The baseline finds missing security headers and configuration errors; it does not test for injection, because it does not attack. With -c you pass a rules file, mounted in /zap/wrk, that sets each rule ID to IGNORE, WARN or FAIL, tab-separated; -g gen.conf generates a starting file with every rule set to WARN. Only rules marked FAIL block: the script exits with 1 if any of them fires, and -I keeps WARNs from returning 2. zap-full-scan.py does launch active attacks: run it only against staging, and only with authorization.
At runtime, Falco captures syscalls (by default with its modern eBPF probe) and raises alerts based on rules; it does not block, and response is left to other systems. Tetragon can also act in the kernel: Override stops the call and returns an error, and SIGKILL kills the process, although on its own it does not always stop the operation from completing.
A minimal workflow for pull requests
This workflow runs the PR gates as three parallel jobs; the weekly schedule: trigger runs only the secrets job, over the full history. Signing and publishing belong in the release pipeline.
name: security-gates
on:
pull_request:
branches: [main]
schedule:
- cron: "0 3 * * 1" # weekly full-history secret scan
permissions:
contents: read
env:
GITLEAKS_VERSION: "8.30.1"
GITLEAKS_SHA256: "551f6fc83ea457d62a0d98237cbad105af8d557003051f41f3e7ca7b3f2470eb"
SEMGREP_VERSION: "1.181.0"
TRIVY_VERSION: "0.75.0"
TRIVY_SHA256: "c6e65abddb348e25f10549df887045629cf28cc72453cd1c63acb717316b3f3f"
jobs:
secrets:
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
fetch-depth: 0
persist-credentials: false
- name: Scan with gitleaks (PR range, or full history on schedule)
env:
BASE_SHA: ${{ github.event.pull_request.base.sha }}
run: |
set -euo pipefail
f="gitleaks_${GITLEAKS_VERSION}_linux_x64.tar.gz"
curl -sSfL -o "$RUNNER_TEMP/$f" \
"https://github.com/gitleaks/gitleaks/releases/download/v${GITLEAKS_VERSION}/$f"
echo "${GITLEAKS_SHA256} $RUNNER_TEMP/$f" | sha256sum -c -
tar -xzf "$RUNNER_TEMP/$f" -C "$RUNNER_TEMP" gitleaks
if [ "$GITHUB_EVENT_NAME" = "pull_request" ]; then
"$RUNNER_TEMP/gitleaks" git --redact --verbose --log-opts="${BASE_SHA}..HEAD" .
else
"$RUNNER_TEMP/gitleaks" git --redact --verbose .
fi
sast:
if: github.event_name == 'pull_request'
runs-on: ubuntu-24.04
permissions:
contents: read
security-events: write # upload SARIF to code scanning
actions: read # needed by upload-sarif on private repos
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
fetch-depth: 0
persist-credentials: false
- name: Semgrep on new code only
env:
BASE_SHA: ${{ github.event.pull_request.base.sha }}
run: |
set -euo pipefail
python3 -m venv "$RUNNER_TEMP/semgrep"
"$RUNNER_TEMP/semgrep/bin/pip" install --quiet "semgrep==${SEMGREP_VERSION}"
"$RUNNER_TEMP/semgrep/bin/semgrep" scan --config p/default \
--baseline-commit "$BASE_SHA" --severity ERROR --error \
--sarif-output=semgrep.sarif
- name: Upload SARIF
if: ${{ !cancelled() }}
uses: github/codeql-action/upload-sarif@24c54180a607b1449ed407dd24f251e4e9147c8d # v4.38.3
with:
sarif_file: semgrep.sarif
category: semgrep
iac-image-sbom:
if: github.event_name == 'pull_request'
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Install Trivy
run: |
set -euo pipefail
f="trivy_${TRIVY_VERSION}_Linux-64bit.tar.gz"
curl -sSfL -o "$RUNNER_TEMP/$f" \
"https://github.com/aquasecurity/trivy/releases/download/v${TRIVY_VERSION}/$f"
echo "${TRIVY_SHA256} $RUNNER_TEMP/$f" | sha256sum -c -
tar -xzf "$RUNNER_TEMP/$f" -C "$RUNNER_TEMP" trivy
- name: IaC and manifests
run: |
set -euo pipefail
"$RUNNER_TEMP/trivy" config --severity HIGH,CRITICAL --exit-code 1 .
- name: Build image, generate SBOM, gate on vulnerabilities
run: |
set -euo pipefail
docker build -t "app:${GITHUB_SHA}" .
"$RUNNER_TEMP/trivy" image --format cyclonedx --output sbom.cdx.json "app:${GITHUB_SHA}"
"$RUNNER_TEMP/trivy" sbom --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 sbom.cdx.json
Design choices:
- Every Action is pinned to a full commit SHA, with the version in a comment. A tag is mutable: that is what attackers exploited in the tj-actions/changed-files attack (CVE-2025-30066) in March 2025 and in the
trivy-actionattack in 2026. Pinning by SHA does not protect you from whatever the Action itself calls by tag: according to the advisory, atrivy-actionpinned to a commit from before 9 April 2025 could load the compromisedsetup-trivy. - The SHA-256 written into the workflow detects a tampered download, not a malicious release; that is why you pin a known version instead of
latest. Trivy also publishes Sigstore bundles that you can check withcosign verify-blob. permissionsstarts atcontents: read, and only the SAST job addssecurity-events: write. The PR trigger ispull_request, notpull_request_target: on a PR from a fork, GitHub does not pass secrets to the runner andGITHUB_TOKENis read-only. Runner attacks, OIDC federation and verification withgh attestation verifyare covered in the pipeline hijacking post.
What blocks, what warns and how exceptions expire
Start with a few high-confidence rules, block only new findings, and expand once the false-positive rate is low.
An exception without a date ends up being permanent. Trivy loads .trivyignore by default and accepts an expiry date per entry (exp:); after that date, the entry no longer applies and the finding blocks again:
# Owner: team-payments. Reviewed: 2026-10-15.
# Not exploitable here: requires a setuid program using ncurses; the image has none.
CVE-2023-29491 exp:2026-12-31
Limitations
- Dependency and image scanners only see what already has an advisory; an unpublished vulnerability gets past all of them.
- SAST detects patterns: logic and authorization flaws tend to slip through, and the ZAP baseline does not attack.
- If the runner is compromised, the attacker can bypass the gates.
- A verified signature does not prove the build was intact, and admission does not observe runtime.
- ML pipelines add datasets and models that this stack does not inspect; I cover those in the MLOps post, and the usual adoption mistakes are in the post on common DevSecOps mistakes.