Hands-On Tutorial: Hardening MLOps – Building a Secure Machine Learning Pipeline with DevSecOps

Update (15 Oct 2026): Kyverno 1.17 (February 2026) published the policies.kyverno.io/v1 API and announced the retirement of ClusterPolicy, the resource that holds verifyImages rules. According to the policy types table, ImageValidatingPolicy is stable since 1.18, ClusterPolicy was deprecated in 1.19 and will be removed in 1.20 (expected in November 2026). This tutorial verifies signatures with ImageValidatingPolicy.

In February 2024, JFrog found around a hundred malicious models on Hugging Face; one of them opened a reverse shell when loaded with torch.load(). A year later, ReversingLabs described two models that got past Picklescan, the platform’s scanner, by using a deliberately broken pickle, a technique it named nullifAI: the payload ran before the file raised an error.

A typical CI/CD pipeline doesn’t usually inspect those files. An ML pipeline handles two extra artifacts: the dataset, which determines how the model behaves, and the model itself, which in pickle-based formats is executable code. My rule is to treat both as code. Nothing goes into training without validation, and nothing reaches production without a verifiable signature. That follows NIST’s SSDF profile SP 800-218A, written for generative AI, whose data and model integrity practices apply here too.

Threat model by stage

Data poisoning manipulates the training set (OWASP ML02). Model tampering alters the artifact after training: modified weights, a backdoor or a swapped file (OWASP ML10). They are different attacks, and you stop them at different stages (the AML.* IDs are MITRE ATLAS techniques).

Stage	Threat	Control
Ingestion	Data poisoning (ML02, AML.T0020)	Schema and distribution checks before versioning
Storage	Dataset swapped in the bucket, or .dvc pointer replaced	DVC with verify, Object Lock, read-only IAM, review of *.dvc files
Training	Compromised dependency (ML06; Ultralytics, Dec 2024)	pip install --require-hashes, no cache, no id-token in the training job
Artifact	Unsafe deserialization (AML.T0011.000); tampered model (ML10, AML.T0018)	safetensors or skops, scanning, signing with model-signing
Build and publish	Image with known CVEs	SBOM, grype --fail-on high --only-fixed, signing and provenance by digest
Admission	Unsigned image or privileged pod	Kyverno ImageValidatingPolicy, Pod Security Admission
Runtime	Model extraction (ML05) or membership inference (ML04); code execution on the server	Gateway quotas, differential privacy during training, deny-all NetworkPolicy, detection

Step 1: validate the data before versioning it

DVC can only tell you that a dataset hasn’t changed since you registered it, even if it was already poisoned when it arrived. That is why validation comes first, at ingestion, and runs again before training. This pandera script exits with an error if anything doesn’t add up:

import sys

import pandas as pd
import pandera.pandas as pa

schema = pa.DataFrameSchema(
    {
        "age": pa.Column(int, pa.Check.in_range(18, 100)),
        "income": pa.Column(float, pa.Check.ge(0)),
        "debt_ratio": pa.Column(float, pa.Check.in_range(0, 5)),
        "label": pa.Column(int, pa.Check.isin([0, 1])),
    },
    strict=True,
    coerce=True,
)

df = schema.validate(pd.read_csv(sys.argv[1]), lazy=True)

# Range taken from a reviewed baseline, not from the batch itself
default_rate = df["label"].mean()
if not 0.02 <= default_rate <= 0.15:
    sys.exit(f"label rate {default_rate:.3f} outside reviewed baseline [0.02, 0.15]")

strict=True rejects columns that aren’t in the schema, and lazy=True collects every failure into a single SchemaErrors. The script catches schema breaks and gross shifts, such as labels flipped in bulk. A targeted attack that poisons a handful of samples without moving the statistics will get through; NIST AI 100-2 describes those variants.

DVC: versioning that exposes changes

DVC keeps a pointer file (.dvc) in Git with the file’s hash and size, and stores the content in a remote (S3 in this example). The ingestion identity, the only one with write access, runs these commands:

#!/usr/bin/env bash
set -euo pipefail

pip install --require-hashes -r requirements-data.lock # dvc[s3]==3.67.1

dvc init
dvc remote add -d storage s3://ml-data-example/dvc
dvc remote modify storage verify true

dvc add data/train_data.csv
dvc push

git add .dvc data/.gitignore data/train_data.csv.dvc
git commit -m "Add training dataset v1"

With verify true, dvc pull and dvc fetch recompute the hash of whatever they download and compare it with the one in the pointer. There are two limits. The hash is MD5, which is not collision-resistant, and anyone who can commit a new .dvc file gets to redefine what “correct” means. DVC shows you what changed; to make changing it expensive, add the following:

• Versioning and S3 Object Lock on the bucket. Object Lock protects each version against deletion and overwrite for the retention period (in compliance mode, not even the root account can shorten it), but it doesn’t stop anyone from uploading new versions.
• A protected branch and mandatory review of the pointer files, with *.dvc @org/ml-data-owners in CODEOWNERS.
• A training role that can only read the DVC prefix, with no s3:PutObject or s3:DeleteObject:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::ml-data-example/dvc/*"
    },
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::ml-data-example",
      "Condition": { "StringLike": { "s3:prefix": "dvc/*" } }
    }
  ]
}

Step 2: train and save without pickle

The Python documentation warns that unpickling malicious data can execute arbitrary code. PyTorch 2.6 switched the torch.load default to weights_only=True (announcement), and CVE-2025-32434 showed that in versions up to 2.5.1, code could run even in that mode. Restricting an executable format reduces the risk; a format that only holds tensors takes code execution out of the loading path.

For PyTorch, save the weights as safetensors and rebuild the architecture from your own code:

from pathlib import Path

import torch
from safetensors.torch import load_model, save_model


def build_model() -> torch.nn.Module:
    return torch.nn.Sequential(
        torch.nn.Linear(3, 16),
        torch.nn.ReLU(),
        torch.nn.Linear(16, 1),
    )


# Training
model = build_model()
# ... training loop ...
Path("model").mkdir(exist_ok=True)
save_model(model, "model/model.safetensors")

# Serving: architecture from code, weights from the file
served = build_model()
load_model(served, "model/model.safetensors")
served.eval()

load_model uses strict=True by default and fails if the file has extra or missing keys. For scikit-learn, skops avoids pickle: skops.io.get_untrusted_types() lists the types in the file that aren’t trusted by default, and skops.io.load(..., trusted=[...]) fails if one you haven’t reviewed turns up.

If you start from a third-party model in pickle format, scan it with ModelScan (modelscan -p models/base/ exits with an error if it finds issues or can’t scan a file) and convert it to safetensors in an isolated environment with no credentials and no network access, because converting it means loading it. nullifAI shows the limit of this step: scanning is detection, and detection can be evaded.

Step 3: train and sign in separate jobs

In .github/workflows/train.yml, only fetch-data and sign get id-token: write. train installs the training stack and runs your code without being able to request an OIDC token, so a compromised dependency there can’t assume the AWS role or obtain a Sigstore certificate in the name of train.yml. Each job installs only what it needs from a lock file with exact versions and hashes (requirements-data.lock, requirements.lock with pandera==0.34.1 and so on, requirements-sign.lock with model-signing==1.1.1) generated with pip-compile --generate-hashes, and pip rejects any package whose hash doesn’t match (hash-checking mode). Signing uses OpenSSF model-signing in keyless mode; in 1.1.1, its flags use underscores.

name: train-and-sign

on:
  push:
    branches: [main]
    paths:
      - "data/*.dvc"
      - "src/**"
      - "requirements*.lock"
  workflow_dispatch:

permissions:
  contents: read

jobs:
  fetch-data:
    runs-on: ubuntu-24.04
    permissions:
      contents: read
      id-token: write
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false
      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
        with:
          python-version: "3.12"
      - uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6.3.0
        with:
          role-to-assume: arn:aws:iam::123456789012:role/ml-train-read-only
          aws-region: eu-west-1
      - name: Pull dataset (verify=true)
        run: |
          pip install --require-hashes -r requirements-data.lock
          dvc pull
      - uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
        with:
          name: dataset
          path: data/train_data.csv
          retention-days: 1
          if-no-files-found: error

  train:
    needs: fetch-data
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false
      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
        with:
          python-version: "3.12"
      - uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
        with:
          name: dataset
          path: data
      - name: Validate and train
        run: |
          pip install --require-hashes -r requirements.lock
          python src/validate_data.py data/train_data.csv
          python src/train.py
          cp data/train_data.csv.dvc model/
      - uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
        with:
          name: model
          path: model/
          retention-days: 1
          if-no-files-found: error

  sign:
    needs: train
    runs-on: ubuntu-24.04
    permissions:
      contents: read
      id-token: write
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false
      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
        with:
          python-version: "3.12"
      - uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
        with:
          name: model
          path: model
      - name: Sign model directory
        run: |
          pip install --require-hashes -r requirements-sign.lock
          model_signing sign sigstore model --signature model.sig --use_ambient_credentials
      - uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
        with:
          name: signed-model
          path: |
            model/
            model.sig
          if-no-files-found: error

• The AWS role’s trust policy pins sub to repo:org/repo:ref:refs/heads/main (the pattern is in the pipeline hijacking post). sign uses its token for Sigstore only.
• There is no dependency cache. In the Ultralytics attack (December 2024), the attacker used a pull_request_target workflow and an injection through a branch name to poison the GitHub Actions cache used by the release build (PyPI’s analysis). A lock file won’t help if you regenerate it while a malicious release is live.
• Anyone with read access to the repository can download a run’s artifacts, which is why dataset and model expire after one day.
• model_signing signs a manifest containing the hash of every file in the directory and writes a Sigstore bundle to model.sig. Because the .dvc file sits inside model/, the signature also covers the dataset version the workflow says it used.

The signature proves which identity signed and that the files haven’t changed since. It doesn’t prove that training was correct or that the data was clean: if someone compromises train, sign will sign whatever it receives. Splitting the jobs limits who gets tokens, not what gets signed, so protecting the workflow and the branch is part of the control.

Model registry: who writes and who reads

If you store models outside the image, for example under an S3 models/ prefix, add a step to sign that uploads model/ and model.sig to s3://ml-models-example/models/${GITHUB_SHA}/ with aws s3 cp --recursive. That step assumes a role whose only permission is s3:PutObject on arn:aws:s3:::ml-models-example/models/*; serving uses a different role with only s3:GetObject on the same ARN, following the JSON in step 1. With versioning and Object Lock, uploading the same key again creates a new version and the previous one stays protected.

This requires separate roles for writing and reading, and no person with write access to the prefix. Its limit: it tells you who could write, not what was trained. The signature and the .dvc pointer it covers prove that, which is why serving verifies before loading (step 5).

Step 4: a serving image with the verified model

The model ships inside the image, so the image signature covers the weights too. The base image is pinned by digest and the process runs as a non-root user:

FROM python:3.12-slim@sha256:a6e34c598f2467ed0e9a8d349809fcd8b5c603269512df273a0bb1784edc11b1

WORKDIR /app
COPY requirements-serve.lock .
RUN pip install --no-cache-dir --require-hashes -r requirements-serve.lock

COPY src/serve.py .
COPY model/ ./model/

USER 10001:10001
EXPOSE 8080
CMD ["python", "serve.py"]

serve.py loads the model as in step 2. Publish the image as registry.example.com/app with the release.yml from Hardening the supply chain, which scans, signs and attests it by digest with the identity that admission requires. Here it starts when train-and-sign finishes on main, and build-scan, which has no id-token, verifies the model before the build. Change on: and the start of that job:

on:
  workflow_run:
    workflows: [train-and-sign]
    types: [completed]
    branches: [main]

jobs:
  build-scan:
    if: github.event.workflow_run.conclusion == 'success'
    runs-on: ubuntu-24.04
    permissions:
      contents: read
      actions: read
    outputs:
      digest: ${{ steps.build.outputs.digest }}
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          ref: ${{ github.event.workflow_run.head_sha }}
          persist-credentials: false
      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
        with:
          python-version: "3.12"
      - uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
        with:
          name: signed-model
          run-id: ${{ github.event.workflow_run.id }}
          github-token: ${{ github.token }}
      - name: Verify model signature
        env:
          SIGNER: https://github.com/org/repo/.github/workflows/train.yml@refs/heads/main
        run: |
          pip install --require-hashes -r requirements-sign.lock
          model_signing verify sigstore model --signature model.sig \
            --identity "$SIGNER" \
            --identity_provider https://token.actions.githubusercontent.com
      # From here on, the existing build-scan steps: tool install, login, build, SBOM and gate

The order is the control: if the signature doesn’t come from train.yml on main, model_signing verify fails and nothing gets built. A workflow triggered by workflow_run has access to secrets even if the triggering one didn’t; here the trigger only runs on main, and verification rejects a tampered artifact. Reading artifacts from another run requires github-token and actions: read.

That workflow’s SBOM lists the model file thanks to Syft’s safetensors cataloger, which is on by default for images, but it says nothing about the data the model was trained on. The gate is --fail-on high --only-fixed. The full policy is in Hardening the supply chain: without a signature and attestation from that release.yml, admission rejects the Pod.

Step 5: least-privilege serving on Kubernetes

This inference server has no need to talk to the Kubernetes API or to open outbound connections:

apiVersion: v1
kind: Namespace
metadata:
  name: ml-serving
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: v1.37
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: credit-model
  namespace: ml-serving
automountServiceAccountToken: false
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: credit-model
  namespace: ml-serving
spec:
  selector:
    matchLabels:
      app: credit-model
  template:
    metadata:
      labels:
        app: credit-model
    spec:
      serviceAccountName: credit-model
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        runAsGroup: 10001
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: server
          image: registry.example.com/app@sha256:<digest>
          ports:
            - containerPort: 8080
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          volumeMounts:
            - name: tmp
              mountPath: /tmp
      volumes:
        - name: tmp
          emptyDir: {}
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: ml-serving
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-gateway-to-model
  namespace: ml-serving
spec:
  podSelector:
    matchLabels:
      app: credit-model
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: gateway
      ports:
        - protocol: TCP
          port: 8080

• The namespace enforces the Restricted profile of Pod Security Admission, pinned to the cluster’s minor version; the reasoning behind each field is in the Kubernetes defense post. It checks the pod when it is created, not what it does afterwards.
• The dedicated ServiceAccount, with automountServiceAccountToken: false, leaves the process without an API token. Restricted doesn’t require readOnlyRootFilesystem; I add it so a payload can’t rewrite the code or the model.
• default-deny-all blocks egress as well, so a reverse shell like the one in the JFrog case can’t connect out. It requires a CNI plugin that enforces NetworkPolicy; without one, the resource has no effect. It also cuts off DNS, so open it explicitly if you need it.

If the model is downloaded at startup from the registry in step 3 instead, run model_signing verify with the same identity in an initContainer: if verification fails, the server never starts. That variant needs egress to the model store and, unless you supply the trust root with --trust_config, to Sigstore as well. Open only those destinations.

For model extraction (OWASP ML05), NIST AI 100-2 cites limiting queries and detecting suspicious ones, which a well-resourced attacker can work around; in practice, that means authentication and per-client quotas on the gateway in front of the pod. For membership inference (ML04), training with differential privacy bounds the attacker’s success at some cost in accuracy. Returning only the predicted class isn’t enough, because some attacks rely on labels alone. Runtime detection is covered in the Kubernetes defense post.

Out of scope

This post doesn’t cover adversarial inputs (OWASP ML01), poisoning by someone with legitimate access to the data source, or risks specific to LLMs, such as prompt injection.

References

• OWASP Machine Learning Security Top Ten
• MITRE ATLAS
• NIST AI 100-2 E2025: Adversarial Machine Learning
• NIST SP 800-218A: SSDF community profile for generative AI
• OpenSSF model-signing (sigstore/model-transparency)
• Kubernetes: Pod Security Standards