Over the past decade, software teams learned to ask where their code came from. Dependency scanning, signed builds and software bills of materials became standard practice after a series of high-profile supply-chain compromises. AI systems need the same discipline, and they add new kinds of artifacts that most existing tools were not built to track.
The OWASP Top 10 for LLM Applications lists supply chain risk as its own category, and data and model poisoning as another. MITRE ATLAS documents real-world techniques for compromising machine learning supply chains. This post sets out a practical approach for security and AI teams working together.
What is in an AI supply chain
A deployed AI system typically depends on:
- Base model weights, often downloaded from a public model hub or provided by a vendor.
- Adapters and fine-tuned weights, produced in-house or by a partner.
- Training and fine-tuning datasets, from internal systems, licensed sources or public collections.
- Retrieval corpora and embeddings, which shape what the model sees at run time.
- Tokenizers, configuration files and prompt templates, which change behavior as much as weights do.
- Serving software, including inference servers, GPU drivers and the libraries beneath them.
- Third-party tools and plugins that agents can call.
Each one is a place where something can be altered, substituted or poisoned.
Risk 1: weights that execute code
Some common model file formats are built on Python's pickle serialization, which can execute arbitrary code when a file is loaded. A model file downloaded from an untrusted source can therefore behave like an untrusted executable. This is not theoretical; security researchers have repeatedly found malicious model files on public hubs.
What to do:
- Prefer formats designed to store tensors without executable content, such as safetensors.
- Scan model files before they enter your environment, and load untrusted files only in isolated sandboxes.
- Maintain an internal model registry. Production systems pull only from that registry, never directly from the internet.
Risk 2: poisoned or tampered models
A model can be modified to behave normally on most inputs and differently on specific triggers. Such backdoors are hard to detect by inspection alone. A model that claims to be a known release may also simply be a different file.
What to do:
- Verify cryptographic hashes against values published by the original source.
- Sign models that you produce, and verify signatures at deployment. Open tooling from the Sigstore project and similar efforts supports signing ML artifacts.
- Evaluate every model on your own test sets before promotion, including adversarial and trigger-search tests, and compare behavior against a known-good baseline.
Risk 3: data you cannot account for
Training and fine-tuning data shape model behavior directly. Poisoned examples can introduce bias, backdoors or quiet errors. Data with unclear licensing or consent can create legal exposure. Data that contains secrets or personal information can leak through the model later.
What to do:
- Record lineage for every dataset: source, collection date, license, transformations applied and who approved its use.
- Version datasets the way you version code, so any model can be traced to the exact data that produced it.
- Scan training data for secrets and personal information before use, and document the result.
- Apply the same controls to retrieval corpora. An attacker who can add a document to your knowledge base can influence answers without touching the model.
Risk 4: the serving stack
Inference servers, model runtimes and GPU software are software, with ordinary vulnerabilities. They often run with broad permissions and access to sensitive data.
What to do:
- Include the serving stack in standard vulnerability management, with container scanning and patch timelines.
- Run model servers with least privilege and network isolation.
- Pin versions and build from verified sources, following practices from the NIST Secure Software Development Framework (SP 800-218) and its generative AI community profile (SP 800-218A).
An AI bill of materials
A software bill of materials lists the components in an application. An AI bill of materials extends that list to models and data. Both leading SBOM standards now cover this ground: CycloneDX includes support for machine learning components, and SPDX 3.0 adds AI and dataset profiles.
A useful AI bill of materials records, for each deployed system:
- The base model, its source, version and hash.
- Any adapters or fine-tuned weights, with the datasets and configuration used to produce them.
- Retrieval sources and their update process.
- Tokenizer, prompt templates and system configuration versions.
- Serving software and its dependencies.
- Evaluation results that justified promotion to production.
When a vulnerability is disclosed in a model family, a dataset or a library, the bill of materials tells you in minutes which systems are affected.
Putting it into practice
You do not need to solve everything at once. A sensible order of work:
- Stand up an internal model registry and route all production pulls through it.
- Require hash verification and a safe serialization format for every model entering the registry.
- Add dataset lineage records for anything used in fine-tuning.
- Generate an AI bill of materials for each production system, and store it with the deployment record.
- Add signing and signature verification to your model promotion pipeline.
- Extend red-team and evaluation work to cover poisoning and backdoor scenarios.
The principle underneath
You cannot secure what you cannot name. Supply-chain security for AI is mostly about knowing, at any moment, exactly which model, data and code are running, where each came from and who approved it. That knowledge is also what auditors and regulators increasingly expect.
Our AI governance and security practice helps teams build model registries, AI bills of materials and verification pipelines that fit existing DevSecOps tooling. Get in touch to review your current model supply chain.