Privacy and Security in AI
Privacy and security in AI are distinct problems with distinct defenses: protecting the data that feeds models, and protecting the model against attacks such as prompt injection. We map the real risks, why federated learning does not guarantee privacy on its own, and what the EU AI Act and the NIST framework already require.
Privacy and security in AI are two different concerns that should not be confused with each other. Privacy is about the data: protecting the personal information used to train and supply AI models against leakage and misuse. Security is about the system: protecting the model and its behavior against a deliberate attacker. A system can be robust yet indiscreet, or careful with data yet easy to fool.
The distinction matters because the defenses differ too, and because the rise of large language models (LLMs) and agents has widened the attack surface well beyond what classical approaches addressed.
Explanation is not security
Explainable AI (XAI) brings together methods that make a model decision inspectable. One such method is Layer-Wise Relevance Propagation (LRP): it starts from the output and redistributes a relevance score through earlier layers until contributions reach the input features. The paper introducing LRP for nonlinear classifiers formulates that propagation as conservation of relevance between layers and visualizes the contributions. An explanation can help audit a particular prediction; it does not show that the model is secure, causal or correct outside the evaluated cases.
Privacy risks
Models memorize more than they appear to. Shokri and colleagues described in 2017 membership inference attacks: with nothing but access to a model's predictions, an attacker can tell whether a specific record —a medical history, say— was part of the training data. With LLMs the problem sharpens. Carlini and his team demonstrated training-data extraction, recovering verbatim sequences memorized by GPT-2, including names, phone numbers and email addresses. Memorization is not a curiosity; it is a leakage vector.
Security risks: from adversarial examples to agents
Goodfellow, Shlens and Szegedy showed in 2015 adversarial examples: tiny, almost imperceptible perturbations that make a network misclassify with high confidence. Add to this data poisoning, which corrupts a model from its training set. LLMs and agents bring risks of their own, catalogued by the OWASP Top 10 for LLM Applications: prompt injection —direct or indirect, when the model obeys instructions hidden in a web page or a document— tops the list, followed by sensitive information disclosure, excessive agency (an agent with too many permissions acting through connected tools and leaking data through them), model theft and system-prompt leakage. Jailbreaks, which bypass safeguards, round out the picture.
Two historical defenses show why the threat model must always be stated. Adversarial training adds deliberately generated attack examples to training; the original Goodfellow, Shlens and Szegedy study used it to reduce error against that kind of perturbation. Defensive distillation was later proposed to smooth the network’s response, but Carlini and Wagner showed that it did not withstand adapted targeted attacks. Naming a defense is not enough: state what the attacker knows, what it can change and which attack the defense was measured against.
Defenses and their limits: the case of federated learning
Differential privacy, formalized by Dwork and others in 2006, is today's reference technique with quantifiable guarantees: it adds calibrated noise to bound how much an output reveals about any single individual. Homomorphic encryption and secure multi-party computation allow computation over encrypted data. And federated learning keeps data on the user's device, training a shared model without centralizing it. Here a routinely omitted caveat is essential: keeping data local does not guarantee privacy on its own. Gradient-leakage attacks can reconstruct private samples from the shared updates, and membership inference remains possible. That is why federated learning is usually combined with differential privacy and secure aggregation; on its own it is not enough.
Two other families act on data before it is shared. k-anonymity seeks to ensure that each published combination of quasi-identifiers cannot distinguish one person from at least k−1 others; perturbation changes values —through generalization, swapping or noise, for example— to weaken the link to the original record. NIST SP 800-188 frames these as de-identification techniques and calls for measuring re-identification risk. Removing names or reaching k-anonymity does not prove anonymity: auxiliary variables and rare combinations can still re-identify people.
Governance and regulation
The institutional response is now tangible. The NIST AI Risk Management Framework (2023) offers a voluntary framework built on four functions —govern, map, measure and manage—. The EU AI Act, in force since August 2024, imposes staggered obligations: prohibited practices from February 2025 and duties for general-purpose models from August 2025. What remains unresolved is the core: there is no complete defense against prompt injection, and privacy guarantees almost always cost some utility. Security in AI is a moving balance, not a state you arrive at.
This article was produced with artificial intelligence under human editorial oversight.
