Your AI Security Team Can't Fix a Safety Problem

When an AI model hallucinates a legal clause in a client contract, that is not a security failure. When a threat actor poisons the training data feeding that same model, that is not a safety failure. Both happened last year. In both cases, the organisation involved had a team responsible for AI risk. In both cases, that team said it was the other discipline's problem. In both cases, nobody owned it.

That is what this piece is about. Not semantics. Not academic definitions. The specific, costly, and entirely avoidable consequences that follow when two distinct disciplines are treated as one, resourced as one, and governed as one.

I have worked across enough AI deployments to see this pattern repeat itself with uncomfortable reliability. And as organisations scale their AI adoption faster than their governance structures can follow, the gap between these two disciplines is widening quietly, and expensively, in the background.

What AI Safety Actually Is

AI safety is about the consequences of a system behaving correctly.

That sentence is worth sitting with for a moment, because it inverts the instinctive assumption most people bring to the word "safety." The model in question is not being attacked. It is not compromised. No one has interfered with it. It is functioning exactly as designed. The problem is that what it was designed to do — or what it has learned to do through training — does not align with what the organisation actually needs, or with the interests of the people it affects.

Hallucinations are the most visible example. An AI system generating a plausible but entirely fabricated legal precedent, a non-existent product specification, or a confident medical recommendation with no factual basis — none of those require an attacker. The model is doing its job. It is predicting the most statistically probable output given the input it received. The output happens to be wrong, and in high-stakes contexts, dangerously so. AI hallucinations cost businesses $67.4 billion globally in 2024, a figure projected to reach $112 billion in 2025. MIT researchers found that AI models are 34% more likely to use confident language specifically when generating incorrect information — which means the outputs most likely to mislead are also the ones least likely to be questioned. Tendem Tendem

But hallucination is only one category of safety failure. Model drift — where a system's behaviour shifts subtly over time as it is updated, fine-tuned, or exposed to new data — is a safety failure. Embedded bias, where a model's outputs consistently disadvantage a particular group because of skewed training data, is a safety failure. And agentic AI taking autonomous action that sits within its technical permissions but entirely outside the boundaries of sound judgement is a safety failure. No breach. No attacker. Just a system doing what it was allowed to do, in ways nobody fully anticipated.

Safety, then, is about the gap between what a system can do and what it should do.

What AI Security Actually Is

AI security is about deliberate, adversarial interference.

Where safety risks live inside the model, security risks come from outside it. Someone — a threat actor, a competitor, a malicious insider — is actively trying to manipulate, compromise, extract from, or weaponise the AI system, and they are doing so intentionally.

Data poisoning is a security attack. Between August 2024 and August 2025, researchers demonstrated successful data poisoning attacks against Microsoft 365 Copilot, ChatGPT Connectors, and multiple RAG-based enterprise AI systems, achieving data exfiltration and decision manipulation, often requiring nothing more than uploading a malicious document to a shared drive. Research from Anthropic, the UK AI Security Institute, and the Alan Turing Institute demonstrated that as few as 250 malicious documents can successfully backdoor large language models — a number that reframes data poisoning from a theoretical concern to a practical, low-effort attack vector. Trilateral Research Medium

Prompt injection is a security attack — where carefully crafted inputs attempt to override a model's instructions and redirect its behaviour toward attacker-controlled outcomes. Model inversion, where an adversary attempts to reconstruct sensitive training data from a deployed model's outputs, is a security attack. Adversarial inputs designed to cause misclassification are security attacks. Each of these requires intent. Each of them requires an adversary who has studied the system and chosen a method.

AI security, therefore, is about what happens when someone deliberately makes something go wrong with a system that would otherwise function.

The Line Between Them

One clean distinction: AI safety is about what a system does when nothing is wrong with it. AI security is about what happens when someone deliberately makes something wrong with it.

That distinction determines which team should own the problem, which frameworks apply, which testing methodologies are relevant, and which type of incident response is required when something goes wrong. Conflating them doesn't just create confusion — it creates ownership gaps, and ownership gaps have a price.

Why the Confusion Is Structural

The surface-level reason most organisations conflate these two disciplines is understandable. Both involve AI and risk. Both require specialist knowledge. Both appear in the same boardroom conversations about governance and exposure. It is natural to assume they belong to the same team.

The deeper reason is that the organisational structures most enterprises have built weren't designed for this distinction, and nobody has been forced to make it explicitly until now.

In many enterprises, security leaders assume they own anything labelled "AI security." Privacy and legal teams gravitate toward "AI safety" because that is where regulatory exposure and liability live. Product and data leaders continue building and shipping AI capabilities because the business demands it, often without a clear mandate from either group. The result is a fuzzy accountability structure and a lot of meetings that never quite resolve ownership. Knostic

Nearly 80% of organisations don't have a dedicated plan for risks associated with AI, and over 54% of those adopting AI lack structured risk frameworks. The gap isn't theoretical. It is the default state of most enterprise AI governance right now. Mooglelabs

Three Scenarios Where the Confusion Became the Problem

_Scenario One — The hallucination that became a legal liability_

An organisation deploys an AI assistant to support contract review across its legal and procurement functions. The productivity gains are real — review cycles shorten, the team handles higher volumes, stakeholders are satisfied. Several months in, a clause makes it into a vendor contract referencing a regulatory requirement that doesn't exist. The clause is executed. The error surfaces during an external audit.

The CISO's team is pulled in immediately. Security runs a full investigation. Access logs reviewed, credentials audited, system integrity checked. No breach. No compromised account. No attacker. The model had simply generated plausible-sounding contractual language that bore no relationship to actual regulation — and nobody had a validation gate in place to catch it before the document was signed.

Six weeks of security resource, diverted. The actual problem — output validation controls, mandatory human review for high-stakes legal documents, task-specific model testing — remained entirely unaddressed. Because the team that should own that problem was never in the room.

That is a safety failure. Treated as a security incident. Both went unresolved.

_Scenario Two — The poisoned model that looked like a bias audit_

An organisation uses an AI system to support early-stage recruitment screening. Over a rolling three-month period, the shortlists the system produces begin to show a pattern — certain profiles are consistently deprioritised in ways that the team cannot attribute to the defined screening criteria. An internal review flags it as a potential bias issue.

The AI ethics committee convenes. The model's outputs are audited. Bias testing is run across demographic variables. The model is retweaked and redeployed with adjusted weighting. The shortlists normalise. The case is closed.

What nobody investigated was the training data pipeline. A third-party dataset used during a recent fine-tuning exercise had been compromised. Subtle, targeted modifications had been introduced — not enough to trigger standard data validation checks, but enough to shift the model's behaviour over time in a specific direction. The attacker's objective was manipulation of hiring outcomes. The ethics team solved the symptom. The security team was never consulted. The compromised data source remained in the pipeline.

That is a security attack. Treated as a safety and ethics problem. The attacker succeeded, and the organisation still doesn't know it.

_Scenario Three — The agentic AI that nobody owned_

An organisation deploys an autonomous procurement agent. Configured to handle routine vendor payments below a certain threshold, reviewed and approved by the technology team, integrated into existing financial workflows. The productivity case is compelling.

Three months post-deployment, the agent processes a payment instruction embedded inside a PDF invoice routed through a supplier email. The instruction is malicious — a prompt injection attack, crafted by an external threat actor who had mapped the agent's accessible inputs. The agent, operating within its configured permissions and with no human review gate on transactions below the threshold, authorises and executes the payment. By the time the finance team identifies the anomaly, the funds are gone and the agent has logged the transaction as routine.

The post-incident conversation is telling. The CISO frames it as a security incident — prompt injection, a known attack vector, a security control failure. The AI governance team frames it as a safety failure — an agent with autonomous financial execution capability should never have operated without a human review gate on transactions of consequence. Both are correct. Neither team had joint governance in place. The accountability conversation runs in circles for weeks.

This scenario sits precisely at the intersection of safety and security — a security attack exploiting a safety gap. And that intersection is exactly where the absence of coordinated governance does the most damage.

What Each Discipline Actually Requires

_AI Safety requires:_

Alignment testing before any system reaches production — specifically testing whether the model's behaviour in edge cases matches organisational intent, not just whether it performs well on benchmark tasks. Output validation layers, particularly for high-stakes use cases where an incorrect output has legal, financial, or operational consequences. Bias audits run against real-world data reflective of the actual deployment context, not controlled test sets. Human oversight mechanisms calibrated to decision weight — the higher the consequence of an autonomous action, the more deliberate the oversight gate needs to be. Red teaming specifically scoped to model behaviour and output quality, separate from adversarial security testing. And continuous monitoring for model drift as systems are updated, fine-tuned, or exposed to new data over their operational lifespan.

_AI Security requires:_

Threat modelling built around AI-specific attack surfaces — data pipelines, model endpoints, retrieval systems, agent tool integrations — rather than mapped directly from traditional application security frameworks. Adversarial input testing to probe for prompt injection vulnerability and model manipulation. Data provenance controls across every stage of training and fine-tuning, including third-party data sources. Access controls on model APIs and endpoints. Active monitoring for anomalous model behaviour that could indicate an ongoing attack rather than a performance degradation. And incident response plans that account for AI-specific failure modes — including the scenario where the compromise isn't immediately visible in system logs because the attack modified the model's behaviour rather than the infrastructure around it.

_Where they overlap:_

Prompt injection sits at the intersection of both. It is a security attack — deliberate, adversarial, targeted — that produces a safety outcome: a model behaving in ways that violate organisational intent. Data poisoning follows the same pattern — a security attack that corrupts safety behaviour over time. These overlaps are not an argument for merging the disciplines. They are an argument for building governance structures that ensure both teams are in communication, understand each other's remit, and have a shared escalation path when an incident touches both.

What the Regulatory Environment Now Demands of Both

he compliance context is sharpening the distinction whether organisations have made it internally or not.

The EU AI Act — formally Regulation (EU) 2024/1689, available in full here — entered into force on 1 August 2024. Full enforcement requirements for high-risk AI systems, covering recruitment tools, credit scoring systems, and critical infrastructure, come into effect from August 2026. Penalties for safety measure failures reach up to €35 million or 7% of global annual turnover. Those penalties attach specifically to safety outcomes — biased decisions, inadequate human oversight, non-transparent outputs — not to security breaches. A separate regulatory track governs data protection and security obligations. Treating them as the same exposure does not consolidate your compliance effort. It creates gaps in both directions simultaneously.

The NIST AI Risk Management Framework is the most comprehensive voluntary guidance available for managing AI risk across both safety and security domains. Structured around four functions — Govern, Map, Measure, Manage — it provides a flexible, non-prescriptive baseline that works alongside other governance structures and is increasingly referenced in enterprise procurement, assurance, and regulatory conversations globally.

ISO 42001, published in December 2023, is the first certifiable international standard for AI management systems. Where the NIST framework is principle-driven and flexible, ISO 42001 is structured and audit-ready — it provides the formal governance scaffolding that organisations operating in regulated industries or seeking external validation of their AI governance practices are increasingly moving toward. The two are complementary rather than competing, and NIST has published a formal crosswalk mapping its framework against ISO 42001 for organisations implementing both.

For the security-specific attack surface — prompt injection, data poisoning, model inversion, adversarial inputs — two resources belong in every practitioner's reference set. The OWASP Top 10 for Large Language Model Applications maps the most critical vulnerability classes specific to LLM deployments. The MITRE ATLAS framework — Adversarial Threat Landscape for Artificial-Intelligence Systems — provides a structured threat matrix for AI-specific attack techniques, built in the same tradition as MITRE ATT&CK but scoped entirely to AI systems. Neither of these is as widely adopted in enterprise AI governance as they should be.

What Good Governance Actually Looks Like

The organisations getting this right share a common structural feature: separate ownership with a shared forum.

Two distinct mandates — one team accountable for AI safety, one accountable for AI security — with clear definitions of where each begins and ends. Both reporting into a common AI governance board that holds the full picture, sets the risk appetite, and owns the accountability when either fails. Risk tiering applied before deployment — not every AI system carries the same stakes for safety or for security, and governance should be proportionate to the actual consequence of failure rather than uniformly applied across all AI use cases.

Before any AI system reaches production, three questions should have documented answers: which team owns the safety review and signed off on alignment, output quality, and oversight design; which team owns the security review and signed off on threat modelling, adversarial testing, and data provenance; and who holds accountability if both fail simultaneously, and does that person know they hold it.

The third question is the one most organisations currently cannot answer.

Where This Leaves Us

AI safety and AI security are not competing disciplines. They are complementary ones, solving different problems, requiring different expertise, and failing in different ways when neglected. The organisations that understand this distinction will not just respond better when something goes wrong — they will know which room to send the problem to. In a field where the wrong response wastes weeks of resource and leaves the actual issue unaddressed, that clarity is not a governance nicety. It is a material operational advantage.

The how-to piece — what building this governance structure actually looks like in practice — is coming next.

For now: which of these two disciplines does your organisation currently have more mature controls for? And do you think your leadership team would give you the same answer?

Your AI Security Team Can't Fix a Safety Problem

Stay Updated

Comments (0)