Nearly every security team has put AI inside their stack. Almost none of them trust how it's governed. GenAI plays a role in 77% of security stacks, and 95% of cybersecurity professionals say AI can improve speed and efficiency. But 72% are neutral or lack confidence in their organization's ability to execute an AI security strategy. The technology is in. The guardrails are not.
Most security programs weren't built for this. Your SOC is now responsible for governing AI used across the business and the AI embedded in your own detection and response tools, under active adversarial pressure.
This article covers what to solve first: visibility into shadow and vendor AI, accountability for autonomous actions, reviewable AI reasoning, and the data foundation any of it depends on.
Key Takeaways:
Security teams must govern AI across the entire organization and within their own security tools, under adversarial conditions.
Shadow AI and post-procurement vendor AI changes create blind spots that traditional DLP can't catch. Continuous visibility and contractual notification requirements are necessary.
Accountability and human oversight must be designed into AI workflows before expanding autonomy. Audit trails and confidence-threshold gating satisfy both compliance frameworks and operational safety.
Data quality, normalization, and residency form the foundation of reliable AI triage. Fix the data pipeline before deploying AI models.
Why AI governance is a different problem inside the SOC
Security teams must govern AI used across the organization and the AI embedded in their own tools at the same time, under active adversarial pressure.
That changes both the scope of governance and the consequences of getting it wrong. The next section explains why the Security Operations Center (SOC) has to manage internal and external AI risk at the same time, while attackers actively look for blind spots.
Security teams now govern AI twice
Your Security Operations Center (SOC) is responsible for AI use across the business and for the AI embedded in the tools your team uses to protect the organization.
Your SOC is responsible for how the rest of the business uses AI, including shadow AI discovery and acceptable-use enforcement, and for the AI embedded in the tools your team uses to protect the organization. Both responsibilities map to the same governance disciplines: AI risk management, accountability, and oversight. Both align with NIST AI RMF and ISO/IEC 42001.
No other business function carries both roles simultaneously, against an active adversary. In HR or finance, a biased AI model produces bad decisions. In the SOC, a biased model creates a systematically exploitable blind spot. Adversaries are already developing techniques to evade AI-based detection. That's a problem detection engineering teams have to plan around, not assume away.
That adversarial pressure changes how governance has to work. A suppressed or ignored alert can contribute to an undetected breach, and SOC teams may have limited visibility into the AI they rely on when models are embedded in commercial products controlled by the vendor.
Start with visibility: you can't govern AI you can't see
AI governance starts with visibility into both employee AI use and vendor AI changes after procurement.
Those two gaps look different in practice, but they create the same governance problem: you cannot control what you cannot inventory. The next two sections cover the blind spot created by shadow AI use and the separate blind spot created when vendors change AI behavior after procurement.
Shadow AI and data leaving for public models
Shadow AI creates a visibility gap because employees can move sensitive data to public models outside approved channels.
71% of employees use unapproved AI tools at work, with 51% doing so weekly. 28% do so because no approved alternative exists. Meanwhile, 63% of organizations lack AI governance policies entirely.
A primary data transfer mechanism is copy/paste through a web browser rather than file upload or API transfer, which creates a visibility gap for many traditional DLP tools. The browser is the de facto control layer, because that's where data movement actually happens. Your network-level DLP is monitoring file transfers while your engineers are pasting production configs into ChatGPT.
For lean SOC teams, aggressive blocking drives usage underground, so the better approach combines faster approval processes, browser-layer visibility, and SIEM integration for domain-level awareness of which AI services your organization is actually reaching.
Vendor AI that changes after the security review
Vendor AI can change after procurement in ways that invalidate a point-in-time security review. Traditional security reviews assume software doesn't change materially between assessments. AI systems violate that assumption architecturally. Standard procurement questionnaires miss the failure modes that matter most for AI systems: prompt injection, model manipulation, training data poisoning, and output drift.
The Microsoft 365 Copilot rollout is the canonical example. Copilot respects existing permissions, but its ability to reference enterprise files means overshared or misconfigured OneDrive and SharePoint permissions surface sensitive documents to unintended users at activation. Organizations that reviewed Microsoft 365 as a productivity suite hadn't assessed the data access model Copilot would apply to that same data.
Require vendors to provide advance notification of material AI changes. That requirement should trigger customer security review processes when their AI systems change materially. Without that mechanism, you're relying on discovering changes after they've already affected your environment.
Fix accountability before you expand autonomy
Accountability controls need to be in place before AI takes on more autonomous security actions. Only 14% of security professionals allow AI to take independent remediation actions with no human in the loop. Teams want AI speed, but the accountability structures aren't in place yet.
Decide where human approval has to remain mandatory as AI takes on more action. The next sections show how to keep those decisions enforceable and auditable.
The accountability gap when AI acts on alerts
AI actions on alerts need defined human decision points, or accountability disappears when something goes wrong.
When AI triages, escalates, or suppresses alerts without a defined human decision point, the organization bears the consequences without a clear human decision-maker. The NIST AI RMF addresses this directly: "Human roles and responsibilities in decision making and overseeing AI systems need to be clearly defined and differentiated."
Here's the failure mode: an AI triage system auto-closes a low-severity credential access alert based on historical false-positive patterns. That alert turns out to be real lateral movement. No human reviewed the suppression. When the breach surfaces weeks later, there's no human decision-maker accountable, only a model version and confidence score in a log.
Human oversight, contextual judgment, and flexibility still do the work AI models can't, especially in complex decisions.
Make human-in-the-loop an enforceable control
Human review only works when the workflow forces it. Without enforceable approval rules, "human in the loop" becomes a phrase in a policy doc rather than a control. As Stephen Gubenia, Head of Detection Engineering for Threat Response at Cisco Meraki, says, "You have to have that human in the loop early and often."
Here’s a practical decision model:
Alert enrichment and context gathering: AI autonomous, human review on demand.
Alert triage and prioritization: AI recommends, humans approve or override.
Alert suppression for novel patterns: AI flags for human review, suppression requires explicit approval.
Network containment or blocking actions: AI recommends, humans approve before execution.
Attribution (associating artifacts with specific actors): Humans only.
Automate well-understood, high-confidence cases and require human review for lower-confidence or novel ones, consistent with NIST AI RMF and other established AI oversight frameworks.
Panther implements this pattern through its Human in the Loop Tool Approval. It pauses before executing sensitive actions, shows a review card for analyst approval, and logs every decision in audit trails.
Audit trails and the frameworks you already answer to
Audit trails for AI actions should capture enough detail to support both incident review and compliance review. Your audit trail should capture every AI disposition, including suppressions and escalations, confidence scores, reasoning factors, human review actions, and the active model version.
That audit trail should be detailed enough to "help AI developers and deployers identify the potential source of system underperformance or failure following an unexpected incident."
SOC 2 doesn't contain specific AI governance requirements, but the Processing Integrity trust service criterion is your most direct hook. If an AI suppresses an alert without defined authorization logic, that failure maps to a processing integrity deficiency. ISO 42001 is more directly applicable. Its controls include requiring named accountability for AI-related decision-making and ongoing monitoring of AI system behavior.
If you're already answering to SOC 2 or ISO 42001 auditors, these hooks already exist in your compliance program.
Demand explainability and acknowledge the limits
Security teams need AI outputs they can review, and they need plans for the cases where AI still fails. Most AI tools in the SOC produce outputs analysts can't trace back to the underlying evidence. The research community is just starting to engage with that problem.
Security teams still have to answer two practical questions: whether a model can produce a useful result, and whether an analyst can understand and verify it. The next sections cover both sides by looking at reviewable reasoning first, then at the failure modes teams still need to plan around.
From black-box outputs to reviewable reasoning
AI outputs are only useful in the SOC when analysts can review and validate the reasoning behind them. Systematic research on AI in the SOC keeps finding the same gap: the field is building high-stakes decision tools while treating analyst understanding of those decisions as secondary.
A tool returning "High Risk: 87/100" with no traceable reasoning doesn't give you anything you can audit, challenge, or improve. Reviewable reasoning should give analysts enough detail to understand and validate the recommendation in context. Panther's AI SOC analyst presents transparent reasoning with evidence trails that trace conclusions back to source data. That helps analysts verify findings.
Cresta's security team found that this transparency cut triage time by at least 50%, especially for complex investigations, because analysts could trace and verify every AI-driven conclusion.
What AI in the SOC still gets wrong
Current AI models still fail in predictable ways that security teams need to plan around. Current models remain limited by high false positive rates, inconsistent generalization, and difficulty aligning outputs with evolving threat contexts. A model can perform well on benchmark data and fail systematically on production traffic without signaling that failure.
As Ken Bowles, Director of Security Operations at GreenSky, puts it: "You have to know the limitations of what it can and can't do right now."
AI detection models trained on historical data miss novel or sufficiently mutated attack patterns, and the miss is often invisible because the model doesn't flag uncertainty. In one recent pilot study, a deployed explainable-AI tool went largely unused and didn't improve analyst decision accuracy.
Treat data governance as the foundation
Reliable AI triage depends on reliable data, so data governance has to come before model rollout. AI triage is only as accurate as the data feeding it. Fix the data pipeline before deploying AI models, or you'll spend the next year explaining garbage outputs to analysts who have already stopped trusting the system.
This section focuses on two parts of that foundation: whether your telemetry is consistent enough for AI to reason over it, and where that telemetry goes once AI inference starts. The next sections cover data quality first, then the governance implications of residency and bring-your-own-model architectures.
Why unreliable data breaks AI triage
Inconsistent telemetry breaks AI triage because models cannot reliably correlate events or build accurate baselines from messy data. Without consistent field naming and time synchronization across log sources, AI models can't reliably correlate events or build accurate baselines. The same kind of field may be represented differently across log sources, which breaks correlation unless the data is normalized.
Without normalization, correlation rules and ML models can't join events across sources. Events with clock skew will be correlated incorrectly even when field names are standardized, producing false sequences that an AI triage layer has no way to detect on its own. This gets worse at cloud-native companies. As teams add new services and integrations, they also introduce more log variation.
Docker's security team faced this at scale: they needed to ingest from VPC flow logs, HA proxy, GuardDuty, and Security Hub without legacy cost constraints. After centralizing ingestion with consistent normalization, they cut false positives by 85% while tripling ingestion volume. Clean, normalized data made the AI useful instead of unreliable.
As Matthew Martin, Founder of Two Candlesticks, puts it, "Before you can really go all in on sort of AI, you got to spend time making sure that you actually understand those data sources, they're cleaned up, there's governance around them."
Data ownership, residency, and bring-your-own-model
The location where AI inference runs decides which data residency rules apply to your telemetry.
Send your security telemetry to a third-party AI inference endpoint, and that endpoint's jurisdiction now governs your data. Data sovereignty is the legal and regulatory question; data residency is the geographical one. Infrastructure choices can set compliance requirements before the legal team ever weighs in.
Bring-your-own-model architectures avoid the jurisdiction problem by keeping inference inside your environment. But they introduce new governance requirements: endpoint security between platform and model, prompt injection attack surfaces, and the risk of analysts connecting to unapproved model endpoints outside the governed pipeline.
Governing AI in the SOC without slowing it down
Teams with the strongest governance programs are the ones actually getting value from AI. The reverse is also true.
Start with what you can see. Build accountability structures before expanding autonomy. Demand reviewable reasoning and stay honest about where AI still fails. Fix your data pipeline before training models on it. None of this happens by accident. It happens because someone wrote it into a runbook.
In practice, that means human-in-the-loop approval workflows with audit-logged triage decisions, plus detection rules versioned in Git: AI that shows its work and lets your team stay in control.
See how Panther helps security teams govern AI without slowing down detection and response.
Share:
RESOURCES
