Governing agentic AI in regulated US industries

Executive summary

Agentic AI systems, which plan and take actions through tools rather than only producing text, are moving into regulated industries. Banks are exploring agents for operations, compliance and customer service. Healthcare organizations are applying them to documentation, coding, scheduling and prior authorization. Government agencies are piloting them for case processing, records and citizen services.

Leaders in these sectors often ask what new rules apply. The more useful question is which existing rules already apply, and how. In the United States, most of the obligations that matter for agentic AI come from regimes that predate it: model risk management guidance in banking, privacy and security rules in healthcare, and authorization and security frameworks in the federal government. Each of these was written to be technology-neutral, and supervisors have repeatedly signaled that they expect them to be applied to AI.

This paper offers a governance approach that works across all three sectors. It treats each agent as three things at once: a model whose outputs must be validated, a system that must be secured, and a privileged actor whose authority must be bounded and recorded. It then summarizes, at a high level, how key US regimes bear on each of those roles.

This paper is general information, not legal advice. Regulatory requirements change, and you should confirm current obligations with your counsel, compliance function and supervisors.

What makes agents different to govern

Traditional AI governance focused on predictive models: a credit score, a fraud flag, a readmission risk. A person or a deterministic system acted on the output. Agentic systems change three things.

• They act. An agent can send a message, update a record, file a request or trigger a workflow. The output is no longer only a recommendation.
• They chain decisions. An agent may take many steps, each shaped by the results of the last. A small error early can compound.
• They are open to influence. Agents read documents, emails and web content that may contain instructions intended to redirect them. The OWASP Top 10 for LLM Applications names prompt injection and excessive agency as leading risks for this reason.

Governance must therefore cover not only whether outputs are accurate, but whether actions are authorized, appropriate and accountable.

A common governance model for agents

Role 1: the agent as a model

The reasoning at the core of an agent produces outputs that drive decisions. Those outputs need development standards, validation before use, ongoing monitoring and documentation of known limits. This is the territory of model risk management.

Role 2: the agent as a system

An agent is software that processes data, connects to other systems and runs on infrastructure. It needs the same security controls as any system handling regulated data: access control, encryption, logging, vulnerability management, incident response and change control.

Role 3: the agent as a privileged actor

An agent holds credentials and takes actions. It should have a distinct identity, least-privilege access, defined limits on what it may do without approval, and a complete record of what it did. In practice, this resembles how organizations govern employees and service accounts with elevated access.

Governing all three roles together prevents a common gap: a model that passed validation, running in a secure system, that still has far more authority than its task requires.

Autonomy tiers

A practical way to match control to risk is to define tiers of autonomy and assign each agent, or each action an agent can take, to a tier.

1. Tier 1: inform. The agent retrieves, summarizes or drafts. A person reviews and decides. No external effect.
2. Tier 2: recommend. The agent proposes a specific action with reasoning and evidence. A person approves or rejects each one.
3. Tier 3: act with review. The agent acts on routine, reversible matters within set limits. People review a sample after the fact, and exceptions are escalated.
4. Tier 4: act within strict bounds. The agent acts on its own within narrow, well-tested limits, with automated checks and immediate escalation outside those limits.

Higher tiers should require stronger evidence: longer production history at lower tiers, more extensive testing, and explicit sign-off from the accountable business owner and risk function. Actions that are irreversible, affect a person's rights or finances, or touch safety should generally remain at Tier 2 unless there is a strong, documented case otherwise.

Financial services

Model risk management: SR 11-7

The Federal Reserve and the Office of the Comptroller of the Currency issued their Supervisory Guidance on Model Risk Management in 2011, known as SR 11-7 (OCC Bulletin 2011-12). The FDIC later adopted it as well. The guidance defines model risk broadly and sets expectations in three areas: sound model development, implementation and use; model validation, including evaluation of conceptual soundness, ongoing monitoring and outcomes analysis; and governance, policies and controls, including a model inventory and clear roles.

A central idea in the guidance is effective challenge: critical analysis by objective, informed parties who can identify model limitations and drive change. Many institutions now bring AI and generative AI systems into their model risk programs. For agents, that suggests:

• Including each agent, and the models it relies on, in the model inventory.
• Documenting intended use, limits, data sources and the tools the agent can call.
• Validating behavior against ground truth for the tasks it performs, not only general benchmarks.
• Monitoring outcomes continuously, with thresholds that trigger review.
• Treating changes to prompts, tools, retrieval data or the underlying model as model changes subject to review.

Third-party risk

The federal banking agencies' 2023 interagency guidance on third-party relationships applies to AI vendors and model providers. Expect to assess a provider's data handling, security, model change practices, concentration risk and ability to support examinations.

Consumer protection and fair lending

Where agents touch credit, existing obligations such as the Equal Credit Opportunity Act and its adverse action notice requirements continue to apply. The Consumer Financial Protection Bureau has stated that creditors using complex algorithms must still provide specific and accurate reasons for adverse actions. Agents used in collections, servicing or marketing also operate within consumer protection rules on fairness and disclosure.

Securities and broker-dealers

Securities regulators have emphasized that existing rules apply regardless of the technology used. Supervision, recordkeeping and communications rules are particularly relevant when agents draft or send client communications.

Healthcare

HIPAA

The HIPAA Privacy Rule governs the use and disclosure of protected health information, including the minimum necessary standard. The HIPAA Security Rule requires administrative, physical and technical safeguards for electronic protected health information. For agents, that means:

• Limiting each agent's access to the minimum protected health information its task requires.
• Ensuring access controls, audit controls, integrity controls and transmission security apply to agent activity.
• Including AI systems in the organization's security risk analysis.
• Putting business associate agreements in place where a vendor creates, receives, maintains or transmits protected health information on your behalf, including AI service providers.
• Retaining records that allow you to reconstruct which data an agent accessed and why.

Clinical and device considerations

Some software functions, including certain AI-enabled ones, are regulated by the Food and Drug Administration as medical devices. FDA has published guidance on AI-enabled device software, including on predetermined change control plans that describe how such software may be modified after authorization. Health IT certification rules also include transparency requirements for certain decision support tools. Agents that inform clinical decisions should be assessed for these obligations early.

Clinical safety in practice

Beyond regulation, clinical settings call for conservative autonomy. Administrative tasks such as scheduling and documentation drafts may reach higher tiers. Anything that affects diagnosis, treatment or access to care should keep a qualified clinician in the decision.

Public sector

FedRAMP and federal security

The Federal Risk and Authorization Management Program provides a standardized approach to security assessment and authorization for cloud services used by federal agencies. Authorizations are granted at impact levels, commonly Low, Moderate and High, based on security controls drawn from NIST SP 800-53. Cloud-hosted AI services used by federal agencies are generally expected to fit within this process. Systems operated by agencies themselves fall under agency authorization processes under FISMA.

For agents, the implications include clear authorization boundaries that account for every model, tool and data source an agent can reach; continuous monitoring that covers agent activity; and supply-chain risk management for models and AI components.

Federal AI policy

The Office of Management and Budget has issued, and periodically revised, guidance on how federal agencies use and acquire AI, including expectations for inventories, risk management for higher-impact uses and accountable officials. Because this guidance has changed across administrations, agencies and their vendors should work from the current memoranda.

Other public sector regimes

Depending on the mission, other requirements may apply: the CJIS Security Policy for criminal justice information, Section 508 accessibility requirements for public-facing systems, records management obligations, and state-level cloud authorization programs and AI laws. Some states have enacted laws addressing high-risk AI systems, and more are under consideration.

Public trust

Government agents interact with people who usually cannot choose another provider. Transparency about when AI is involved, clear routes to a human decision-maker, and careful handling of decisions that affect benefits, rights or liberty are essential.

Cross-sector controls

Across all three sectors, a consistent set of controls supports compliance with these regimes:

• Inventory and ownership. Every agent is registered, with an accountable owner, purpose, tier, data access and tool list.
• Identity and least privilege. Every agent has its own identity and short-lived, narrowly scoped credentials.
• Policy enforcement outside the model. Limits on actions, data and recipients are enforced in code or policy engines, never only through instructions to the model.
• Human approval. Consequential actions require approval by a named, qualified person, with context to decide well.
• Verifiable evaluation. Agents are tested against ground truth before release and monitored in production, with regression gates on change.
• Tamper-evident records. Inputs, reasoning summaries, tool calls, approvals and outcomes are recorded in a way that resists alteration and supports examination.
• Incident response. Playbooks cover suspending agents, revoking access, notifying affected parties and regulators where required, and correcting downstream effects.
• Vendor oversight. Third-party AI providers are assessed for security, data handling, change management and examination support.

The NIST AI Risk Management Framework, with its Govern, Map, Measure and Manage functions, is a useful way to organize these controls across sectors, and it is referenced widely by US agencies and industry groups.

Evidence: the part that is often missing

Supervisors and auditors do not only ask whether a control exists. They ask for evidence that it operated. For agents, prepare to show:

• The validation report that supported deployment, and the results of later revalidation.
• Monitoring results over time, with any breaches of thresholds and the response.
• Records of approvals for consequential actions, including who approved and on what information.
• A complete history of changes to the agent's model, prompts, tools and data.
• Records sufficient to explain any individual action the agent took.

Designing for evidence from the start is far cheaper than reconstructing it during an examination.

A practical starting sequence

1. Inventory existing and planned agents and assign autonomy tiers.
2. Map each agent to the regimes that apply to its sector, data and actions.
3. Bring agents into existing model risk, security and privacy processes rather than creating parallel ones.
4. Implement identity, least privilege and approval workflows for anything above Tier 1.
5. Build evaluation suites and monitoring tied to the outcomes each agent affects.
6. Establish tamper-evident records and test that they answer the questions an examiner would ask.
7. Review tiers periodically, widening autonomy only where evidence supports it.

How TuQuant can help

TuQuant designs agentic systems for regulated environments, with human command over consequential actions, controls enforced outside the model and records built for review. Our AI governance and security practice helps organizations bring agents into existing model risk, security and privacy programs, and our agentic AI development practice builds agents to those standards from the start. Products such as TuQuant Samiksha for documents and audit, and TuQuant Vaani for voice intelligence, are built around the same principles.

To discuss governance for a specific program, contact us.

This paper provides general information about publicly available regulatory guidance and frameworks. It is not legal advice and does not establish compliance with any regime. Requirements vary by institution, jurisdiction and use, and change over time.