Machine Learning in Cybersecurity: From Anomaly to Response

Re-edited on July 30, 2026, this article corrects a dangerous promise: machine learning does not automatically identify unknown attacks or keep an organization “one step ahead.” It learns signals from telemetry and produces scores in a particular environment. Turning those scores into security requires asset inventories, threat hypotheses, sensors, analysts, and response.

The difference appears in an everyday anomaly: a large backup, a new administrator, or a marketing campaign may look unusual without being malicious. Conversely, an attacker using valid credentials may look normal. A model helps order evidence; it does not replace the questions of what is protected, from whom, and at what error cost.

Start with the asset and behavior, not the algorithm

A useful detection completes a sentence: “alert when this identity, device, or process performs this sequence because it may correspond to this technique and requires this action.” The MITRE ATT&CK catalog organizes observed tactics and techniques and provides vocabulary for relating telemetry to behavior. It is neither a signature list nor proof that an alert is an attack.

The inventory determines what can be seen. Authentication, DNS, process, network, email, and cloud logs provide different perspectives and blind spots. Coverage, clocks, lost events, schema changes, and permissions are documented. If a sensor does not record the behavior, no model recovers it. If an alert does not link asset, user, and sequence, the analyst loses necessary context.

The NIST Cybersecurity Framework 2.0 organizes outcomes into govern, identify, protect, detect, respond, and recover. Machine learning fits mainly as a technique inside some outcomes; it does not replace the six functions. “Prevention” can mean blocking an action, hardening a control, or reducing impact, and each meaning requires different authority and evidence.

Before training, assign the role: prioritize alerts, detect deviations, classify files, or suggest an investigation. Blocking requires much stronger evidence than ranking a queue. A false alarm can interrupt work; a false negative can permit intrusion. Thresholds reflect those costs and are reviewed by asset rather than selected to maximize an abstract score.

Supervision and anomaly detection answer different questions

Supervised learning uses examples labeled as classes. It can recognize known families when labels are reliable and the future resembles the past. But an alert closed as “benign” because analysts lacked time is not a true negative; a campaign may be discovered months later; and training on product names or dates can create shortcuts.

Anomaly detection models a reference and scores deviations. It does not need a name for every attack, but it does need a unit, window, and definition of normality. “Unknown” is not the same as “zero-day”: a system may flag a shift change or miss an attacker who imitates the baseline. The proper output is a priority with observable reasons, not a verdict.

Kitsune presented an online intrusion-detection system based on an ensemble of autoencoders and evaluated defined network scenarios. The paper makes feature extraction, training, and testing traceable. It does not establish that every neural network detects every threat; it presents one design and datasets whose transfer to another network requires testing.

Signatures and rules do not become obsolete when ML arrives. A signature can precisely block a confirmed indicator; a rule expresses policy; an anomaly finds deviations; a classifier scales triage. Defense combines layers and preserves a simple route. If the model stops responding, deterministic protections and logging must continue to work.

Datasets are historical maps, not miniature Internets

Benchmarks support comparison, but they age and reflect their laboratory. CICIDS2017 documents benign traffic and generated attacks over specific days in a test environment. UNSW-NB15 combines normal and synthetic traffic produced in a particular infrastructure. Neither represents an organization’s present network, cloud, or users by itself.

A random split can leak sessions, hosts, or nearly identical artifacts across training and testing. A harder evaluation separates by time, campaign, family, device, or organization. It preserves a future period and excludes fields created after the incident—such as an investigation label—from features. If class can be inferred from the data-generation tool, the model learns the laboratory.

For malware, EMBER released features and a labeled dataset of PE files for reproducible research. The labels and period have limits, and a static detector does not observe runtime behavior. An open reference is valuable because work can be repeated; deployment requires comparison with authorized, recent local samples.

Metrics depend on base rates. With millions of events and few attacks, high accuracy can be useless. Reports include precision, recall, false positives per unit of time, time to detection, analyst load, and technique coverage. Curves are translated into volume: daily alert counts and expected incidents missed at a threshold.

The adversary and ordinary change break assumptions

In cybersecurity, whoever generates some data is trying to evade detection. Research on adversarial examples for malware detection studied how features could be modified while preserving functionality to evade classifiers. Generic noise is not the defense: teams define what an attacker can observe and alter and test valid domain transformations.

The model itself may be targeted through extraction, poisoning, or feedback manipulation. Training data is versioned and its provenance controlled; analyst labels require review; thresholds and features are not exposed unnecessarily. An automatic loop that retrains on all its own alerts can reinforce its mistakes.

The environment also changes without an attacker: a new VPN, remote work, cloud migration, an operating-system update, or an acquisition. Teams monitor volume, distributions, alert rate, confirmations, and affected groups. Drift triggers investigation rather than automatic retraining. First determine whether the business, sensor, or threat changed.

Multi-agent, evolutionary, or reinforcement systems may be studied in simulation, but giving them authority to “adapt defenses in real time” adds risk. A reward may favor excessive blocking; a simulator may omit dependencies; agents may interact unexpectedly. Begin in recommendation mode with permitted actions, boundaries, and rollback.

From a score to an incident

Every alert needs an owner, evidence, priority, and next step. An analyst sees changed features, related events, scope, and sources and can confirm, dismiss, or escalate with a reason. That response supports measurement but should not enter training directly without controls. Model utility remains separate from investigation-process quality.

NIST SP 800-61 Revision 3, published in April 2025, integrates incident response into CSF 2.0 risk management. Detection reduces harm only if an organization has prepared communication, analysis, containment, recovery, and learning. A model does not contact the owner or restore service by itself.

A pilot runs in shadow mode first. It is measured against rules and analysts, false positives and negatives are reviewed, and sensor failure is tested. It may then automate reversible, low-impact actions such as enriching an alert or temporarily limiting a session. Isolating a critical system, deleting files, or locking an account needs policy and confirmation proportionate to risk.

The transferable skill is turning every “AI for cybersecurity” promise into a chain: asset, threat, telemetry, label or baseline, model, metric, threshold, analyst, and response. If a link is missing, there is a demonstration rather than a defense. Every link also needs an owner and a periodic test. The test must represent real operating conditions and changes over time. Machine learning can reduce load and find patterns; security comes from the human and technical system that turns uncertain signals into verifiable action.