Shadow AI showed up in 20% of breaches studied last year, and those incidents cost roughly $670,000 more than conventional ones. Most security teams don't have a current inventory of the AI tools running in their environment, let alone documented risk tiers, approval gates, or monitoring for model drift.
That gap is now a security problem. Legal asks who approved the new triage agent. Auditors ask how human oversight works for high-impact actions. The EU AI Act's GPAI obligations took effect August 2, 2025, and the controls these regulations require, including risk documentation, transparency mechanisms, human oversight, and adversarial protections, are engineering and security operations work.
This guide walks through how to build an AI governance program for a lean security team: inventory and risk classification, framework mapping across NIST AI RMF, ISO/IEC 42001, and the EU AI Act, controls for agentic AI inside the SOC, and the explainability and data ownership requirements that hold the whole thing together.
Key Takeaways:
Security teams own AI governance because the work, from inventory to incident response, maps directly to security operations capabilities.
Three major frameworks (NIST AI RMF, ISO/IEC 42001, and the EU AI Act) converge on the same requirements: AI system inventory, risk-tiered oversight, human review for high-impact actions, and continuous monitoring.
Governing agentic AI in the Security Operations Center requires explicit boundaries and human-in-the-loop approval for high-impact actions, with staged rollouts tied to validated performance baselines.
If your AI tools can't show their reasoning and you don't control where your security data lives, the rest of your governance program has no foundation.
Why AI governance now falls to security teams
Security teams already have most of what AI governance requires: telemetry, detection workflows, and an investigation muscle. Proxy log analysis can surface shadow AI, while threat modeling and detection engineering support risk classification and monitoring.
Industry research confirms the trend: more than half of 300 IT and security professionals surveyed identified security teams as the primary owners of protecting AI systems.
Regulatory pressure is accelerating the timeline. The EU AI Act's GPAI obligations took effect August 2, 2025, with general application and transparency requirements arriving August 2, 2026. The controls these regulations require, including risk documentation, transparency mechanisms, human oversight, and adversarial attack protections, are engineering and security operations work.
The financial stakes reinforce the urgency: shadow AI breaches carry roughly $670,000 in added cost per incident, and that gap widens as AI adoption outpaces governance maturity.
What security teams own in AI governance
Security teams own the inventory and risk controls for AI systems, plus the detection work needed after deployment. The work starts with knowing which AI systems exist, then deciding how tightly to control them and how you will catch problems after deployment. Legal and compliance contribute policy language, but technical execution sits with your team.
The framework mapping below shows how those responsibilities line up across the main standards teams are using. The mapping gives you a practical way to translate broad governance requirements into the controls your team already recognizes.
How AI governance maps to NIST AI RMF, ISO/IEC 42001, and the EU AI Act
The major AI governance frameworks emphasize similar governance pillars, even though they do not prescribe the same core controls. All three frameworks ask for roughly the same things, whether voluntary (NIST AI RMF), certifiable (ISO/IEC 42001), or binding law (EU AI Act). ISO/IEC 42001 and the NIST AI RMF can be used together to support an EU AI Act compliance approach. AI governance depends on good processes, sound detection logic, and the logging and alerting pipelines that make both observable in production.
Here's how the requirements line up across all three:
Requirement | NIST AI RMF | ISO/IEC 42001 | EU AI Act |
AI tool inventory | Article 11 (high-risk technical documentation) | ||
Human oversight for automated actions | GOVERN 3.2 | Annex A controls | Article 14 |
Ongoing monitoring | MEASURE function | Clauses 8–10 (especially Clause 9) | Article 72 (post-market monitoring) |
Incident response for AI failures | MANAGE function | Clauses 8–10 | Article 73 |
Adversarial attack protection | MEASURE / MANAGE | Clause 8 (risk assessment and treatment) |
A step-by-step path to implementing AI governance
Implementing governance works best as a phased rollout. The five phases below build on each other, but a three-person security team doesn't need to tackle all of them at once. Start with inventory and classification, then layer in controls and monitoring as capacity allows.
1. Inventory your AI systems and surface shadow AI
Inventory comes first because you can't govern systems you haven't identified.
Minimum inventory fields: name, vendor, deployment date, risk tier, data access scope, action authority, owner, and last review date.
For shadow AI detection, use capabilities you already have for shadow IT: proxy log analysis for AI service endpoints and DLP enforcement, supported by network monitoring. Security telemetry stored in open formats in a customer-owned data lake lets you build custom queries to spot anomalous AI endpoint traffic.
2. Classify risk and define policies
Risk tiers determine how much review and control each AI system needs. Score every AI system on four dimensions:
How sensitive is the data it touches?
What actions can it take?
How deeply is it integrated with other systems?
Can you audit what it did?
Three tiers shake out:
Tier 1 (Restricted): High on data access or action authority. Touches production data or takes autonomous actions. Needs formal security review, human approval gates, and executive sign-off. Example: AI agents with production access.
Tier 2 (Managed): Medium on two or more dimensions. Needs security review at onboarding and human review before consequential recommendations. Example: AI detection builders where generated code requires review before deployment.
Tier 3 (Standard): Low across all dimensions. Self-service registration with standard vendor review. Example: AI writing assistants on public content.
3. Assign ownership across a cross-functional group
Clear ownership prevents AI governance from stalling between teams. Security teams own technical execution: Tier 1 reviews, monitoring, detection engineering, and AI-specific incident response. Legal and compliance own policy language and regulatory interpretation, including vendor DPA review.
Department heads own identifying and registering AI tools their teams use, and they serve as the first escalation point for new AI adoption. The CISO or VP of Engineering owns sign-off for Tier 1 systems and policy exceptions.
4. Embed controls into the AI lifecycle
Governance controls need to follow the system from design through operation. During design, document intended purpose and data inputs. Before deployment, run risk classification and complete the appropriate tier review. After deployment, enable audit logging. Keep model documentation current while you monitor for drift.
5. Monitor, audit, and improve
Continuous monitoring keeps governance effective after deployment. Set up automated thresholds, because a model that performs well at deployment can drift over time due to changing data patterns or adversarial inputs. Automated alerts should trigger when metrics exceed defined bounds instead of waiting for quarterly reviews. Audit cadence should track the rate of new AI tool adoption, not the calendar.
Governing agentic AI inside the SOC
Agentic AI inside the Security Operations Center (SOC) needs tighter controls because those systems can act autonomously. Agentic AI governance starts by limiting agent actions and defining where human approval remains mandatory.
Implement the controls in that order: set operating limits first, then define when approval is mandatory.
Set boundaries, confidence thresholds, and fallbacks
Every AI agent in your SOC needs explicit operating limits. Give each agent a least-privilege identity. Pair that with an instant kill-switch and rate limits for automated actions. The same guardrails that govern any privileged automation apply: agent identities need least privilege, automated actions need rate limits, and operators need an instant override path.
Configure confidence thresholds based on your environment's false positive rates. Start conservative and expand autonomy only after validating performance baselines.
Keep a human in the loop for high-impact actions
High-impact actions should require human-in-the-loop approval, especially where the consequences are material, irreversible, or outside established patterns. AI systems can't always detect or correct their own errors. When the consequences of being wrong are material, a human needs to be in the decision path.
As James Nettesheim, CISO at Block, puts it, "We still want a human in the loop overall. We're extremely bullish on adopting agentic coding and analysis."
In Panther AI, Human in the Loop Tool Approval enforces this directly: when the AI SOC analyst wants to perform a sensitive action (updating alert status, creating detection rules, modifying security data), it pauses and presents a review card. Users can accept, reject, or let the request time out, and all decisions are logged in audit logs.
Start with human-in-the-loop on every output during pilot. This captures corrections as training data and establishes performance baselines. Expand to human-on-the-loop only for low-risk playbooks where the agent has demonstrated reliable performance.
Build trust through explainability and data ownership
Analysts need reviewable AI decisions, and security teams need control over the data those decisions use. Start with decision visibility, then data control.
Require AI that shows its work
Analysts can't meaningfully approve or override decisions they can't understand. If the AI can't explain its reasoning, the human-in-the-loop model stops working. Transparency, explainability, and interpretability are related but distinct: a vendor can be transparent about what data went in without ever explaining why the model reached a given conclusion. Security teams evaluating AI tools need all three.
When evaluating security AI tools, require that every verdict comes with the specific data points, logs, and enrichments behind the decision. Detection rules and correlation logic should be human-readable.
Cresta's security team found this distinction operational: analysts trace and verify every AI-driven conclusion because visualization capabilities make relationships between data points readable. Triage time fell by at least 50%.
Detection-as-code directly addresses this problem. When detection rules are Python or YAML stored in version control, they're reviewable by both humans and AI. Panther's AI Detection Builder generates detection code from natural language descriptions, but the output is treated as a draft for review before deployment.
Govern your security data and where it lives
Your team needs control over where security telemetry lives and how AI tools use it. Your security telemetry needs the same data governance as the rest of your sensitive data. When an employee submits company data to an unapproved AI service, that can create direct regulatory exposure depending on what data was submitted.
If your security data is locked in a proprietary SIEM schema, your ability to build custom queries, run independent audits, or switch vendors is constrained by that vendor's terms. Practitioners who've migrated off proprietary SIEMs describe the same pattern: extracting historical data and re-parsing it in a new schema involves real cost and engineering time, which is exactly what proprietary storage formats are designed to discourage.
When evaluating platforms, require contractual data portability, open format storage, and written confirmation that customer data isn't used for cross-customer model training. Full control and full visibility of your security data is non-negotiable when AI tools are operating on top of it. Without it, you can't audit what the AI saw, reproduce a decision, or switch vendors when you need to.
Measure governance and stay honest about AI's limits
Governance needs both operational metrics from security and indicators specific to AI systems. Common incident-response baselines like MTTD and MTTR apply unchanged. On the governance side, track AI system risk categorization coverage and whether human oversight boundaries are clear. Also measure whether AI security safeguard reviews happen on schedule.
Be honest about what governance can't solve. Model drift and model decay are common, and governance frameworks must address adversarial threats like data poisoning and model evasion alongside continuous monitoring.
Operationalizing AI governance as your AI footprint grows
AI governance needs to evolve as your AI footprint changes. Any AI governance framework built today will likely be materially incomplete in twelve months. Recent research identifies a structural gap: existing frameworks were not designed for autonomous AI agents that can take actions and chain tasks with minimal human oversight.
Start with inventory and risk classification, then layer in lifecycle controls and continuous monitoring as your footprint expands. Teams that get governance right adopt AI faster because they spend less time arguing about whether to ship and more time validating that it works.
Analysts need reviewable decisions and a way to stop sensitive actions before they run. That's where governance stops being policy and starts being product. Explore how Panther operationalizes AI governance for security teams.
Share:
RESOURCES
