In May we estimated that about 43% of our transcribed books were “confidently new to AI training” (Did the AI Read This?). We have now measured the question twice more on today's library, with current models, controls, and the method and its stopping rules written down before any model was asked. The result: for about half of the distinct works we hold (52%, 95% interval 47–57%), none of Gemini 3.1 Pro, Claude Opus 5.5 or GPT-5.6 can name the author from the title. Counted by volumes it is about a third, because the works the models know tend to be the ones with many editions.
That is a narrower claim than May's. A model that cannot name a work's author has not met the work in any form it can retrieve. It is not proof that the text was never in its training data, and no test run from outside a closed model can give that proof. This note explains why, shows how the measurement runs, and places it among the published methods.
Two questions that sound like one
“Is this book in the training data?” hides two questions. The first is whether the model knows the work: whether a translation, an edition, a catalogue entry or a scholarly citation has reached it. The second is whether it trained on our file, the scan and transcription we hold. A model can know the Iliad well without ever having seen our 1535 printing of it.
Scroll sideways to see the whole diagram.
The second question is the one the research literature calls membership inference, and the literature is discouraging. Duan and colleagues ran the standard tests on models whose training data is public and found they barely beat random guessing. Das and colleagues showed that many published successes came from comparing two different kinds of text, and that a test which never looks at the model can do better. Our own May test agreed: given the first 80 characters of our transcription, a model continued books it “knew” only 1.17 times better than books it did not. So this note answers the first question and does not claim the second.
From “do you know it?” to “who wrote it?”
Our first re-run, on 6 October, simply asked the models whether they knew each work. Gemini 3.1 Pro said no to 45% of a random sample. A review of that run found the question too soft. Shown a work's title and author, Pro says it knows 59% of works; asked to name the author from the title alone, it manages 38%. Saying “yes, I know it” is much easier than recalling anything. The review also found that many of the “unknown” Tibetan records were volume labels and shelfmarks that nobody could recognise. So the second re-run asks for a fact that can be checked.
Scroll sideways to see the whole diagram.
We drew 500 works at random from every book with transcribed text, visible or not, one edition per work. 416 of them have a title a person could identify and an author in our catalogue to check against. Each model sees only the title, year and language; where the author's name is printed in the title, it is masked. It is asked who wrote the work, roughly when, and what it is about, and may guess if it marks the guess. The named author is matched against our catalogue mechanically, with names folded across scripts and spellings. A work counts as known if any of the three models names its author.
The controls
A model is counted only if it passes the controls. Claude Haiku did not, and is reported but excluded.
| control | Gemini 3.1 Pro | Claude Opus 5.5 | GPT-5.6 Sol | Claude Haiku |
|---|---|---|---|---|
| famous works, author named (21) | 85.7% | 90.5% | 85.7% | 66.7%, fails |
| obscure but known works (25) | 88% | 84% | 80% | 28% |
| invented titles called known (40) | 2.5% | 0% | 2.5% | 5% |
The famous works the models “miss” are catalogue quirks: our record for the Diamond Sūtra names its translator, the Qur'an is catalogued under Muhammad. The one invented title two models accepted turned out to be a near twin of a real 1619 treatise on comets, a fault in the decoy rather than a bluff. A second run of Pro agreed with the first on 98% of works.
The result
No model names the author: works, volumes, pages
Hover, tap or tab to a row for the exact figure. Bars are the share; whiskers the 95% interval.
Works (217 of 416): 52.2% unnamed by all three models, 95% interval 47.4% to 56.9%.
Numbers from run 2 of issue #6038 (7 October 2026, preregistered); intervals are 95% Wilson intervals on its counts.
The three counts answer different questions. By works it is about half. By volumes and pages it is about a third, because the famous works fill many volumes. Reading every mismatch by eye, some “wrong” answers name the right person under another spelling (Suso for Seuse, Sylvius for Dubois) or correct an error in our catalogue. Counting those as known gives 47.7% [43.1–52.6] of works.
Works whose author the model cannot name, by model
Hover, tap or tab to a row for the exact figure. Bars are the share; whiskers the 95% interval.
None of the three: 52.2% of works with no author named, 95% interval 47.4% to 56.9%.
Numbers from run 2 of issue #6038 (7 October 2026, preregistered); intervals are 95% Wilson intervals on its counts.
Works no model can name, by genre and language
Hover, tap or tab to a row for the exact figure. Bars are the share; whiskers the 95% interval.
Disputations, orations (n = 79): 75.9% unnamed by all three models, 95% interval 65.5% to 84.0%.
Numbers from run 2 of issue #6038 (7 October 2026, preregistered); intervals are 95% Wilson intervals on its counts.
The unknown works are concentrated where one would expect: university disputations, orations and dissertations (three in four unknown), and Latin and German books generally. English books are mostly known, and all 15 Chinese works in the sample are known.
One correction to our first re-run matters for anyone reading about our Tibetan and Sanskrit holdings. That run reported 63% of that shelf as unrecognised. The figure came from volume labels and monastery records, not from the great works. All 17 works by the Nālandā masters that we hold, among them the Abhidharmakośa, the Madhyamakakārikā and the Bodhicaryāvatāra, are known to every model. Of seven Tibetan commentaries on them, five are not, but seven is too few to generalise from.
Where this sits among published methods
Scroll sideways to see the whole diagram.
Most published methods ask about the text. Min-K% Prob and dataset inference need the model's token probabilities, which closed models do not expose. The name-cloze test asks a model to fill in a masked character name in a novel, and DE-COP asks it to pick a verbatim passage out of paraphrases. Both work on famous English books a model can partly reproduce. Against seventeenth-century Latin or a Tibetan manuscript they have nothing to detect. The closest relative of this note is work on long-tail knowledge such as Head-to-Tail, which found that model accuracy falls from famous to obscure subjects. What this note adds is the guard rails: invented titles, an obscure-but-known control tier, a fact that is scored rather than a feeling that is reported, and a preregistered rule for when to retract.
What it does not show
- Whether any model trained on our transcriptions. Naming an author is recall of a work, which is stricter than recognising it and says nothing about which texts were in a training set.
- Whether the models know what the works say. A model can name the author of a treatise and know nothing of its argument.
- Our catalogue is the answer key. Where it is wrong, a model can be marked wrong for being right; we read every mismatch by eye, which is how the 47.7% figure arose.
The next run tests the first two points from outside the models: whether passages of our transcriptions appear in the public web crawls that models are trained on, whether a model can answer questions about what a text says, and whether a machine-readable text of the work already exists elsewhere. The data, prompts, preregistrations and scripts are in issue #6038.
