Privacy and Security in AI

Privacy and security in AI are two distinct concerns that are easy to conflate. Privacy is about the data: protecting the personal information that trains and feeds models against leakage and misuse. Security is about the system: protecting the model and its behavior against a deliberate attacker. A system can be robust yet indiscreet, or careful with data yet easy to fool.

The distinction matters because the defenses differ too, and because the rise of large language models (LLMs) and agents has widened the attack surface well beyond what classical approaches addressed.

Privacy risks

Models memorize more than they appear to. In 2017 Shokri and colleagues described membership inference attacks: with nothing but access to a model's predictions, an attacker can tell whether a specific record —a medical history, say— was part of the training data. With LLMs the problem sharpens. In 2021 Carlini and his team demonstrated training-data extraction, recovering verbatim sequences memorized by GPT-2, including names, phone numbers and email addresses. Memorization is not a curiosity; it is a leakage vector.

Security risks: from adversarial examples to agents

In 2015 Goodfellow, Shlens and Szegedy exposed adversarial examples: tiny, almost imperceptible perturbations that make a network misclassify with high confidence. Add to this data poisoning, which corrupts a model from its training set. LLMs and agents bring risks of their own, catalogued by the OWASP Top 10 for LLM Applications: prompt injection —direct or indirect, when the model obeys instructions hidden in a web page or a document— tops the 2025 list, followed by sensitive information disclosure, excessive agency (an agent with too many permissions acting through connected tools and leaking data through them), model theft and system-prompt leakage. Jailbreaks, which bypass safeguards, round out the picture.

Defenses and their limits: the case of federated learning

Differential privacy, formalized by Dwork and others in 2006, is today's reference technique with quantifiable guarantees: it adds calibrated noise to bound how much an output reveals about any single individual. Homomorphic encryption and secure multi-party computation allow computation over encrypted data. And federated learning keeps data on the user's device, training a shared model without centralizing it. Here a routinely omitted caveat is essential: keeping data local does not guarantee privacy on its own. Gradient-leakage attacks can reconstruct private samples from the shared updates, and membership inference remains possible. That is why federated learning is usually combined with differential privacy and secure aggregation; on its own it is not enough.

Governance and regulation

The institutional response is now tangible. The NIST AI Risk Management Framework (2023) offers a voluntary framework built on four functions —govern, map, measure and manage—. The EU AI Act, in force since August 2024, imposes staggered obligations: prohibited practices from February 2025 and duties for general-purpose models from August 2025. What remains unresolved is the core: there is no complete defense against prompt injection, and privacy guarantees almost always cost some utility. Security in AI is a moving balance, not a state you arrive at.