How to Threat-Model an AI Workload

Threat modelling is a familiar exercise for most cloud and security engineers: map the assets, map the trust boundaries, work through how an attacker could cross them. An AI workload uses the same discipline, but the asset list and the attack surface both grow. Skipping that extra step is the most common reason AI deployments get security-reviewed too late, after they're already in production.

Start with the boundaries you already know

Before touching anything AI-specific, model the workload the way you'd model any cloud application. What identities can reach it, and with what privilege? What network paths lead in and out? Where is data stored, and who can read it — the source documents, the vector index, the logs, the model provider's endpoint? Most AI security incidents in practice trace back to a gap at this ordinary layer: an over-scoped managed identity, a storage account without proper access controls, or logs that capture full prompts and responses without any data classification applied to them. Get this layer right first; it's also the layer your existing Cloud Security and Azure Security controls already cover.

Map the AI-specific attack surface

With the conventional boundaries mapped, add the layer that's specific to generative AI:

• Prompt injection points — anywhere untrusted text reaches the model, including user input, uploaded documents and retrieved search results
• Jailbreak attempts against the model's system instructions and safety configuration
• The RAG pipeline's grounding data — can it be poisoned with false or malicious content, and can retrieval leak records the requesting user shouldn't see?
• The model endpoint itself — rate limits, cost controls and extraction or denial-of-service risk
• Any action the model or an agent can take on your behalf — sending communications, writing to a database, calling another API — because that turns a manipulated output into a real consequence, not just a bad answer on screen

For each of these, ask the standard threat-modelling question: what happens if this input is hostile, and what's the actual blast radius if it succeeds?

Use a structured framework rather than working from memory

Two references are worth building into your process rather than reinventing: the OWASP Top 10 for LLM Applications, which catalogues the common failure modes (prompt injection, insecure output handling, training data poisoning, and others), and MITRE ATLAS, which documents real adversary tactics against AI systems in a structure similar to MITRE ATT&CK. Neither replaces judgement, but both stop you from missing an entire category of risk because it didn't come up in a brainstorm.

Score what you find the way you'd score any other risk — likelihood and impact — and resist the temptation to treat every AI-specific finding as automatically critical. A jailbreak against a low-privilege chatbot with no data access is a very different risk to a jailbreak against an agent that can issue refunds.

Close the loop with detection, not just prevention

Threat modelling identifies what could go wrong; it doesn't stop everything from happening. Feed the findings into monitoring — Microsoft Defender and Microsoft Sentinel are the natural home for this, correlating identity, network and AI-layer signals so a suspicious prompt pattern gets investigated alongside the account activity around it, rather than sitting in a separate dashboard nobody watches.

This is exactly the sequence taught in week three of Cloudpilar Academy's Cloud & AI Security Engineering programme (CPS-02), building on the Zero Trust and identity foundations from week one, before moving into prompt injection defence, RAG security and Defender and Sentinel detection in the weeks that follow — and culminating in a capstone security review and incident-response exercise.