An SVM-based AI text detector shows why context matters

An open experiment in detecting language-model-generated text is attracting attention for an unusual reason: it does not ask another large model to judge the writing. Instead, it uses classical machine-learning techniques. The AITextDetector project turns words and phrases into statistical signals and uses support vector machines, or SVMs, to distinguish human Chinese creative fiction from text produced by several LLMs.

Its author reports roughly 85% sentence-level accuracy on the project’s test set. That is an interesting result for exploring how a model’s writing habits can leave detectable traces. It is not, however, the kind of number that can establish who wrote a text in every setting. The repository itself warns that the demo was trained exclusively on Chinese creative writing and that applying it to other kinds of text will probably introduce systematic bias.

That distinction between a classifier within its domain and a general verdict is the heart of the story.

What this classifier measures

The technical idea is fairly straightforward. First, text is represented with TF-IDF, a way of counting which terms and combinations are distinctive within a collection of documents. A LinearSVC then learns a statistical boundary between examples labelled human-written and examples labelled AI-generated.

The author collected human fiction published before 2023 and created a collection of generated text from summaries of those stories. The project describes seven binary classifiers, associated with different generative models, and a voting system: a sentence is marked as suspected when at least two detectors flag it. The code and some model files are public on GitHub, which is valuable because it lets others inspect the approach instead of accepting a black box.

Choosing an SVM is not unusual. scikit-learn’s documentation includes support vector machines among supervised-learning methods for classification problems with labelled data. “Classical” does not mean obsolete. Such models can be fast, conceptually transparent and effective when the task and training data closely resemble the intended use.

That final condition is crucial. A classifier learns regularities from the examples it is given; it does not uncover a universal essence of human text or AI text.

Why a strong internal test is not enough

The data described by the author are specific: Chinese creative fiction, human texts from a pre-ChatGPT period and LLM samples generated through a particular process. The situation changes if the detector encounters journalism, coursework, email, translations, code or writing in another language. It also changes when a person carefully edits a generated draft or uses a model that was not represented in the training collection.

The project includes some checks on unseen models and on human texts from before 2022. Those are useful observations about the experiment, but they are not a replacement for independent evaluation with published corpora, pre-defined criteria and results replicated by other teams. Nor do they turn a statistical probability into proof that an individual used AI.

False positives are the most important risk. Incorrectly labelling human writing as generated can affect a student, an author or a professional without a clear way to undo the harm. That is why the selected threshold, the text domain and the way uncertainty is communicated matter as much as average accuracy.

A lesson the field already learned

OpenAI withdrew its AI-text classifier in 2023 because of its low accuracy. In the evaluation it published at the time, the system identified 26% of AI-generated texts in its challenge set as likely AI-written and incorrectly labelled 9% of human texts as AI-written. The company also warned that the tool should not be the primary basis for a decision, that it performed poorly on short passages and that it could fail outside its training data.

That withdrawal does not show that every kind of detection is useless. It does show that the problem is not solved by an automated label. Methods can help prioritise a review, investigate a coordinated campaign or analyse a tightly defined corpus. To attribute authorship, impose a penalty or judge a person’s integrity, additional evidence and a process for challenging the result are needed.

AITextDetector offers a concrete demonstration: statistical signals from some LLMs can be detectable within a well-defined corpus. The more useful lesson is not that there is now a text polygraph, but that evaluation must be as specific as the problem. Asking about language, genre, models, data and the cost of an error is more responsible than treating a percentage as a verdict.