An SVM-based AI text detector shows why context matters

An open experiment in detecting language-model-generated text is attracting attention for an unusual reason: it does not ask another large model to judge the writing. Instead, it uses classical machine-learning techniques. The AITextDetector project turns words and phrases into statistical signals and uses support vector machines, or SVMs, to distinguish human Chinese creative fiction from text produced by several LLMs.

Its author reports roughly 85% sentence-level accuracy on the project’s test set. That is an interesting result for exploring how a model’s writing habits can leave detectable traces. It is not, however, the kind of number that can establish who wrote a text in every setting. The repository itself warns that the demo was trained exclusively on Chinese creative writing and that applying it to other kinds of text will probably introduce systematic bias.

That distinction between a classifier within its domain and a general verdict is the heart of the story.

What this classifier measures

The technical idea is fairly straightforward. First, text is represented with TF-IDF, a way of counting which terms and combinations are distinctive within a collection of documents. A LinearSVC then learns a statistical boundary between examples labelled human-written and examples labelled AI-generated.

The author collected human fiction published before 2023 and created a collection of generated text from summaries of those stories. The project describes seven binary classifiers, associated with different generative models, and a voting system: a sentence is marked as suspected when at least two detectors flag it. The code and some model files are public on GitHub, which is valuable because it lets others inspect the approach instead of accepting a black box.

Choosing an SVM is not unusual. scikit-learn’s documentation includes support vector machines among supervised-learning methods for classification problems with labelled data. “Classical” does not mean obsolete. Such models can be fast, conceptually transparent and effective when the task and training data closely resemble the intended use.

That final condition is crucial. A classifier learns regularities from the examples it is given; it does not uncover a universal essence of human text or AI text.

Why a strong internal test is not enough

The data described by the author are specific: Chinese creative fiction, human texts from a pre-ChatGPT period and LLM samples generated through a particular process. The situation changes if the detector encounters journalism, coursework, email, translations, code or writing in another language. It also changes when a person carefully edits a generated draft or uses a model that was not represented in the training collection.

The project includes some checks on unseen models and on human texts from before 2022. Those are useful observations about the experiment, but they are not a replacement for independent evaluation with published corpora, pre-defined criteria and results replicated by other teams. Nor do they turn a statistical probability into proof that an individual used AI.

False positives are the most important risk. Incorrectly labelling human writing as generated can affect a student, an author or a professional without a clear way to undo the harm. That is why the selected threshold, the text domain and the way uncertainty is communicated matter as much as average accuracy.

A lesson the field already learned

OpenAI withdrew its AI-text classifier in 2023 because of its low accuracy. In the evaluation it published at the time, the system identified 26% of AI-generated texts in its challenge set as likely AI-written and incorrectly labelled 9% of human texts as AI-written. The company also warned that the tool should not be the primary basis for a decision, that it performed poorly on short passages and that it could fail outside its training data.

That withdrawal does not show that every kind of detection is useless. It does show that the problem is not solved by an automated label. Methods can help prioritise a review, investigate a coordinated campaign or analyse a tightly defined corpus. To attribute authorship, impose a penalty or judge a person’s integrity, additional evidence and a process for challenging the result are needed.

AITextDetector offers a concrete demonstration: statistical signals from some LLMs can be detectable within a well-defined corpus. The more useful lesson is not that there is now a text polygraph, but that evaluation must be as specific as the problem. Asking about language, genre, models, data and the cost of an error is more responsible than treating a percentage as a verdict.

Turning an accuracy figure into a useful question

An accuracy figure can only be interpreted alongside the composition of the test set. If half the sentences are human and half generated, 85% accuracy describes a balanced scenario. In a classroom where nearly every submission was human, even a small false-positive rate could produce more wrongful accusations than correct findings. The author’s technical account lets readers inspect the domain and procedure; it does not grant permission to transfer the percentage to any population.

The minimum table separates true positives, false positives, true negatives and false negatives. The next step is to decide which error is more costly. When studying a large collection, losing some relevant cases may be acceptable if a sample is reviewed. When sanctioning a person, a false accusation is so consequential that the detector cannot be the central evidence. The same algorithm may be reasonable for the first use and irresponsible for the second.

An out-of-domain test is also essential. It should include authors, dates, genres and models that did not appear in training, plus human editing and translation when those occur in the intended use. A drop in performance does not mean the method was fraudulent; it shows that the system was detecting local regularities. The AITextDetector repository warns about exactly that bias by limiting its demonstration to Chinese creative fiction.

A responsible output is not a red label

A useful detector should state its scope, evaluated version, threshold, uncertainty and the next step. It also needs a replication and appeal process: preserve the original text, review any available work history with consent, and let a person add context. Moving a threshold to catch more generated text will almost always change the number of human texts flagged as well.

The transferable skill is to read every detector as a classification instrument, not a lie detector. Before accepting its percentage, ask for the population, error matrix, out-of-domain test and consequence of each failure. Without those four elements, the number describes an experiment, not an attribution of authorship.

System updates deserve the same care. As generative models, user styles and text-collection practices change, the learned boundary ages. An organisation should record which detector version produced each alert and re-evaluate it before replacing the model. Without that history, two decisions carrying the same product name may come from different classifiers. Reproducibility is not achieved by publishing code alone: data, transformation, version, threshold and runtime environment must be fixed. A monitoring plan should also look for changes in the mix of texts and in the rate of contested alerts. Drift is evidence that the old evaluation no longer describes the current decision, not a reason to blame the people whose writing changed.