AI and model security: a reference architecture

Executive summary

Enterprises are moving AI from experiments into systems that read sensitive data, make recommendations and, increasingly, take actions. Each step widens the attack surface. Traditional security controls still apply, but they do not cover the new risks on their own: prompt injection, model and data poisoning, sensitive information disclosure through model output, insecure handling of model output by downstream systems, and agents with more authority than their task requires.

This paper sets out a reference architecture for AI and model security. It organizes controls into six layers, from governance down to operations, and maps each layer to three widely used public frameworks:

• The NIST AI Risk Management Framework (AI RMF 1.0, published January 2023), with its four functions: Govern, Map, Measure and Manage. NIST has also published a companion Generative AI Profile (NIST AI 600-1).
• The OWASP Top 10 for LLM Applications, a community-maintained list of the most significant risks in applications built on large language models.
• MITRE ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems), a knowledge base of adversary tactics, techniques and case studies against AI systems, modeled on MITRE ATT&CK.

The architecture is vendor-neutral. It is meant to help CISOs, heads of AI and engineering leaders agree on a common structure, find gaps, and decide what to build first. It is not legal advice, and it does not replace a risk assessment specific to your organization.

Why AI needs its own security architecture

Three properties of modern AI systems change the security picture.

Instructions and data share a channel. A language model receives instructions and content as the same stream of text. Any content that reaches the model, including documents, web pages, emails and tool responses, can attempt to change its behavior. There is no equivalent of parameterized queries that fully separates the two.

Behavior is learned, not written. A model's behavior comes from its weights and training data, which cannot be reviewed line by line like source code. Tampering with data or weights can produce systems that behave normally until a specific trigger appears.

Agents hold authority. When a model can call tools, it becomes a privileged actor in your environment. Its credentials, scope and actions need the same scrutiny as a service account or an administrator, with the added complication that its decisions can be influenced by untrusted input.

These properties do not make AI uniquely dangerous. They make it different enough that controls must be designed for it on purpose.

The three frameworks and how they fit together

The frameworks complement each other. We recommend using them at different levels of the program.

NIST AI RMF: the program structure

The AI RMF is voluntary and outcome-oriented. Its Govern function covers policies, accountability, roles and culture. Map covers context: what the system is for, who it affects and what could go wrong. Measure covers analysis, testing and tracking of risks. Manage covers prioritizing and acting on risks, including response and recovery. The framework also describes characteristics of trustworthy AI, including that systems are valid and reliable, safe, secure and resilient, accountable and transparent, explainable and interpretable, privacy-enhanced, and fair with harmful bias managed.

Use the AI RMF to structure the program, assign ownership and report to leadership.

OWASP Top 10 for LLM Applications: the application risk list

The 2025 edition of the OWASP list covers: LLM01 Prompt Injection; LLM02 Sensitive Information Disclosure; LLM03 Supply Chain; LLM04 Data and Model Poisoning; LLM05 Improper Output Handling; LLM06 Excessive Agency; LLM07 System Prompt Leakage; LLM08 Vector and Embedding Weaknesses; LLM09 Misinformation; and LLM10 Unbounded Consumption.

Use the OWASP list as a checklist for design reviews and application security testing.

MITRE ATLAS: the adversary view

ATLAS describes how adversaries actually attack AI systems, organized into tactics and techniques, with documented case studies. It covers reconnaissance against models, gaining access, poisoning, evasion, model theft and exfiltration, among other areas.

Use ATLAS to plan red-team exercises, build threat models, and connect AI threats to your existing threat intelligence and detection engineering.

The reference architecture: six layers

The architecture groups controls into six layers. Each layer lists its purpose, core controls and the framework references most relevant to it.

Layer 1: governance and accountability

Purpose: make sure every AI system has an owner, a documented purpose, an assessed risk level and approval to operate.

Core controls:

• An inventory of AI systems, including models in use, their data sources, their tools and their owners.
• A risk-tiering method that sets the depth of review by impact: internal productivity tools need less scrutiny than systems that affect customers, money or safety.
• Defined roles across security, AI engineering, legal, privacy and the business owner.
• Acceptable-use policies for both internally built systems and third-party AI services.
• Third-party risk review for AI vendors, covering data handling, model provenance and incident notification.

Framework mapping: NIST AI RMF Govern and Map. For organizations seeking a certifiable management system, ISO/IEC 42001 provides a standard structure for AI management that can sit alongside an existing information security program.

Layer 2: data security

Purpose: protect the data used to train, tune, ground and operate AI systems, and prevent sensitive information from leaking through them.

Core controls:

• Data classification applied to training sets, fine-tuning sets, retrieval corpora and logs.
• Lineage records for every dataset: source, license, transformations and approvals.
• Scanning for secrets and personal information before data is used for training or indexing.
• Access controls on retrieval that respect the permissions of the person making the request, so a model cannot surface documents the requester could not open directly.
• Integrity controls on retrieval corpora and vector stores, including change review for new content.
• Retention and minimization rules for prompts, outputs and traces.

Framework mapping: OWASP LLM02, LLM04 and LLM08; NIST AI RMF Map and Manage; ATLAS techniques related to data poisoning and exfiltration.

Layer 3: model security and supply chain

Purpose: ensure that the models you run are the ones you intended, from sources you trust, and that they resist known attacks.

Core controls:

• An internal model registry as the only source for production deployments.
• Hash verification and signing for model artifacts, and preference for serialization formats that cannot execute code on load.
• An AI bill of materials for each system, using emerging standards such as CycloneDX machine learning support or SPDX 3.0 AI and dataset profiles.
• Pre-deployment evaluation, including adversarial testing for evasion, jailbreaks and backdoor triggers.
• Protection of model weights as sensitive intellectual property, with access logging and controls against extraction through high-volume querying.
• Secure development practices drawn from the NIST Secure Software Development Framework (SP 800-218) and its generative AI profile (SP 800-218A).

Framework mapping: OWASP LLM03 and LLM04; NIST AI RMF Measure; ATLAS tactics covering ML supply chain compromise, model access and exfiltration.

Layer 4: application security

Purpose: secure the application that wraps the model: its prompts, inputs, outputs and integrations.

Core controls:

• Trust boundaries that mark external content as data, with injection detection on inbound content.
• System prompts that contain no secrets, on the assumption that they may be disclosed.
• Output validation and encoding before any model output reaches a browser, a database, a shell or another system.
• Filters for sensitive data in outputs, aligned with your data loss prevention policy.
• Rate limits, quotas and cost controls per identity and per application.
• Grounding and citation requirements for factual answers, with clear signals when the system is uncertain.

Framework mapping: OWASP LLM01, LLM02, LLM05, LLM07, LLM09 and LLM10; NIST AI RMF Measure and Manage.

Layer 5: agent security

Purpose: constrain what agents can do, and make their actions accountable.

Core controls:

• A distinct identity for every agent, integrated with your identity provider, with short-lived, narrowly scoped credentials.
• Least-privilege tool access, with parameter limits enforced in code or policy, not in prompts.
• Separation between components that read untrusted content and components that hold privileged tools.
• Policy checks on every proposed tool call before execution.
• Human approval for consequential or irreversible actions, with the context an approver needs to decide well.
• Kill switches that can suspend an agent and revoke its credentials quickly.
• Tamper-evident records of inputs, decisions, approvals and actions.

Framework mapping: OWASP LLM06 Excessive Agency, with LLM01 and LLM05; NIST AI RMF Manage; ATLAS techniques involving abuse of AI-enabled capabilities.

Layer 6: security operations

Purpose: detect, respond to and learn from incidents involving AI systems.

Core controls:

• AI telemetry, including prompts, tool calls, policy decisions and model versions, routed to your security information and event management platform.
• Detection rules for injection attempts, unusual tool use, data exfiltration patterns and abnormal query volume.
• Incident response playbooks specific to AI: suspending agents, rolling back models, purging poisoned content from retrieval stores and notifying affected parties.
• Regular red-team exercises planned against ATLAS tactics.
• Continuous evaluation in production, with regression gates that block changes when safety or quality checks fail.

Framework mapping: NIST AI RMF Measure and Manage; MITRE ATLAS for detection and exercise planning.

Cross-cutting principles

Four principles apply across every layer.

1. Assume partial failure. No single control stops prompt injection or poisoning. Design so that when one control fails, others limit the harm.
2. Enforce outside the model. Permissions, limits and approvals must be enforced by code, policy engines or infrastructure, never only by instructions to the model.
3. Prefer evidence to assertion. Every claim about a system's safety should be backed by a test result, a log or a verifier output that a reviewer can inspect.
4. Keep people in command. The level of autonomy should match the level of assurance. Autonomy can be widened as evidence accumulates.

Deployment patterns and where controls sit

Controls apply differently depending on where the model runs.

• Frontier model through an external API. You control the application, agent and operations layers fully. For the model and part of the data layer, you depend on the provider's commitments, so contractual terms, data handling settings and vendor review matter more.
• Model hosted in your cloud tenancy. You gain control over network paths, logging and data residency. Model provenance still depends on the provider.
• Self-hosted open-weight or custom model. You own every layer, including supply-chain verification, model hardening and serving infrastructure security. This gives the most control and the most responsibility.
• Air-gapped or sovereign environment. All of the above, plus controls for importing models and updates across the boundary, and for operating without external threat intelligence feeds.

A phased roadmap

Most organizations cannot implement every control at once. We suggest three phases.

Phase 1: visibility and basics

• Build the AI system inventory and assign owners.
• Adopt a risk-tiering method and apply it to existing systems.
• Route AI telemetry to security operations.
• Apply input trust boundaries, output validation and least-privilege tools to the highest-risk systems.

Phase 2: supply chain and agents

• Stand up a model registry with verification.
• Produce AI bills of materials for production systems.
• Give agents distinct identities and scoped credentials.
• Introduce approval workflows for consequential actions.

Phase 3: assurance at scale

• Build evaluation and red-team suites mapped to OWASP and ATLAS.
• Wire regression gates into release pipelines.
• Add tamper-evident audit records and periodic independent review.
• Report program maturity to leadership using the NIST AI RMF functions.

Questions for your leadership team

1. Do we have a complete list of AI systems in use, including third-party services?
2. For each high-risk system, who is accountable, and when was it last tested?
3. Which of our agents can change data or systems, and what stops them from being misused?
4. Could we tell, within a day, which systems use a model or dataset found to be compromised?
5. Do our incident response plans cover AI-specific scenarios?

How TuQuant can help

TuQuant provides AI and model security as both products and services. Our AI governance and security practice runs maturity assessments against this architecture, threat models AI systems, conducts red-team exercises and helps teams build registries, evaluation suites and approval workflows. Our sovereign and on-premises AI services apply the same architecture where models must run entirely within your control.

Every TuQuant product is built to the principles in this paper: bounded capability, controls enforced outside the model, human command over consequential actions, and audit records a reviewer can trust. To discuss your program, contact us.

This paper is general guidance. It does not constitute legal or regulatory advice, and framework references reflect publicly available versions at the time of writing. Consult the source frameworks and your own advisors for requirements that apply to your organization.