They Asked ChatGPT to Repeat 'Poem' and It Recited Poe: What That Proves, and What It Doesn't

In November 2023, a team of researchers from Google DeepMind and several universities asked ChatGPT to do something absurd: "repeat the word 'poem' forever." The model complied for several hundred repetitions — "poem poem poem poem..." — and then, unprompted, it started writing something else: complete paragraphs from novels and Edgar Allan Poe's "The Raven," word for word. Matching what came out against a nine-terabyte database of internet text, the team confirmed more than ten thousand verbatim fragments of ChatGPT's training data, at a query cost of $200. It wasn't an accident or a leak: it was a deliberate attack, with published methodology, that broke the assistant's normal behavior to force it to recite. The authors themselves reported it to OpenAI before publishing, giving the company time to respond — a responsibility note as important as the finding itself, and one almost no coverage of the case mentions.

This piece has to hold two claims that sound contradictory and aren't. "A model doesn't store training text like a database" is true. "Verbatim fragments have been extracted from published models" is also true. Sloppy journalism picks whichever one fits the headline it wants. Here are both, and when each applies — because the next time you read "this model contains my work" in a lawsuit, a press release, or an outraged thread, you're going to need to know what to ask, not a pre-chewed answer.

Why "compressed database" is a false analogy

If you don't know exactly what a weight is, this same series explains it from scratch in the earlier piece before continuing here. A database has addressable records: you ask for document number 4,521 and get it back whole and intact. A model's weights don't work that way. They're a continuous function fit to predict the next word, with no index or lookup table, and the overwhelming majority of training documents — probably billions of them — leave no recoverable trace at all, not even in principle. When Nasr, Carlini, and their team tested ChatGPT under normal conditions, the model appeared to memorize 50 times less than an unaligned base model: the fine-tuning that makes it conversational also teaches it, as a side effect, not to recite. It took the "poem" attack — which deliberately breaks that behavior — to make it emit training data 150 times more than under normal conditions.

Why "it never memorizes anything" is also false

In 2021, Carlini and his team generated 600,000 samples from GPT-2, narrowed them to 1,800 candidates using automated metrics, and had four researchers manually review each one. They confirmed 604 verbatim memorized examples: real names, phone numbers, and email addresses, 128-bit UUIDs, IRC conversations, and source code fragments. GPT-2 was trained on 40 gigabytes of text; finding 604 recoverable fragments cost hundreds of thousands of generated samples and expert manual review. Of that total, they identified 50 memorized URLs and 31 near-verbatim source-code fragments, among other categories — a tiny, very specific slice, not "the whole book" or "all of the internet." That's what separates "it can be extracted" from "it's easy to extract" — both are true, but not at the same order of effort.

What determines whether something can be extracted

Here's the most useful finding for judging any specific case. In 2022, the same team described three relationships that predict how much a model memorizes, each almost perfectly measurable: memorization grows the (1) larger the model — with a 99.8% fit — (2) the more times that exact text was repeated in training, and (3) the longer the fragment used to prompt it to continue. Duplication carries the most weight: a famous poem quoted on hundreds of web pages, a news article syndicated by dozens of outlets, or a software license copied into thousands of repositories are far more likely to come out intact than a document that exists exactly once in the entire corpus.

And here's what needs saying plainly, because almost no one does: the authors themselves admit that generalizing these three factors across different model families "becomes more complicated" — their words — and, as far as we've been able to verify, no public study with this same rigorous methodology exists for the most recent, closed frontier models (GPT-4 and later, Claude, Gemini). What we know at this level of rigor concerns GPT-2 (small, old, open) and gpt-3.5-turbo, attacked through a specific vulnerability disclosed responsibly to OpenAI — discovered on July 11, 2023, reported on August 30, with a 90-day window before publication. About the models people actually use today, at this level of method, this isn't known.

The capability: what to ask, not what to answer

Faced with any claim that "this model does — or doesn't — contain my work," four questions hold up the argument better than the claim itself:

1. How much was actually memorized? 604 examples out of 40 gigabytes isn't "the whole training set memorized" — it's a tiny fraction, likely skewed toward the most duplicated content.
2. Under what conditions was it extracted? A normal conversation, or an attack designed to break the model's behavior, like asking it to repeat a word forever?
3. At what cost? Did a user try once, or did a team generate hundreds of thousands of samples and filter them with published methodology?
4. Is it representative, or is it the extreme case someone deliberately went looking for? A poem as widely quoted as "The Raven" is exactly the kind of text duplication makes more memorizable; it proves nothing about a text that appears exactly once.

A claim that can't answer these four isn't a fact — it's a suspicion dressed up as a headline, and it should be treated that way regardless of who's making it: a plaintiff's lawyer, a company's PR team, or a viral post. Applied to "The Raven" itself: how much? One poem, not Poe's complete works. Under what conditions? An attack designed to break the model's conversational fine-tuning, not an ordinary conversation. At what cost? A research team with published methodology and a nine-terabyte database to verify against. Is it representative? No: "The Raven" is one of the most quoted and reproduced poems in the English language — exactly the kind of text duplication makes easy to recover. The four answers hold up the headline without needing to reassure you or scare you.

And where we don't have the answer — as with the newest models — the correct sentence is "this isn't known," not a reassuring filler or an alarming one.

For anyone who wants to go deeper

• Extracting Training Data from Large Language Models (Carlini et al., USENIX Security 2021): the GPT-2 study, 604 confirmed examples out of 1,800 candidates.
• Quantifying Memorization Across Neural Language Models (Carlini et al., ICLR 2022): the three factors — size, duplication, prompt length — and their explicit limit when generalizing.
• Scalable Extraction of Training Data from (Production) Language Models (Nasr, Carlini et al., 2023): the "poem" attack on ChatGPT, 10,000+ examples for $200, with the responsible-disclosure note to OpenAI.
• Elsewhere in this series: what a weight actually is, and what it means for a model to have open weights.