The Suda is a tenth-century Byzantine encyclopedia with about 31,000 entries: part dictionary, part gossip column, part salvage operation for a thousand ancient books that no longer exist. It was long considered untranslatable by any one person. The Suda On Line proved it could be done by two hundred people: starting in 1998, volunteer classicists translated it entry by entry, finishing in 2014: sixteen years, one of the first great crowdsourced scholarship projects on the web.
We hold Bekker’s 1854 edition of the Suda (1,158 dense two-column pages, alpha to omega), and our pipeline translated it the way it translates everything: as ordinary throughput. Which raises the question every page of this library should be asked. How good is it?
For almost every book we hold, that question has no external answer: there is no prior translation to check against, which is usually the reason the book is here at all. The Suda is the great exception: 31,000 entries of scholar-vetted English, sitting right there. So we spent a day building the comparison, and it told us things we could not have learned any other way. Some of them are not about us.
Two books, one text, different orders
Step one was mirroring the Suda On Line: all 31,342 entries, with their translations, translator names, vetting status, and (a gift we had not expected) the Greek text of Adler’s 1928–38 critical edition embedded in every entry. Step two was cutting our OCR of Bekker into entries and matching the two lists up.
That sounds mechanical. It is actually philology. Scholars cite the Suda by Adler number (“delta 872”), but Adler numbers postdate our edition by seventy-five years; nothing on Bekker’s pages carries one. Worse, the two editions do not even agree on the order of the alphabet: the Suda’s native arrangement is antistoichic (letters grouped by Byzantine pronunciation, so that αι- words file after alpha as their own block), and Adler kept that arrangement while Bekker quietly re-alphabetized it. Our first alignment pass scored exactly 0% on three letter-groups before we understood why. And when several people share a name (the Suda has seven consecutive entries for men called Didymus), Adler’s numbering does not follow Bekker’s printed order, so the only way to match them is by what the entries actually say.
The finished aligner matches on headwords, insists the sequence stay monotonic, uses the scholarly reference itself to find entry boundaries our OCR ran together, and disambiguates homonyms by text similarity against Adler’s Greek. It aligned 27,149 entries, 86.6% of the Suda, each one now carrying its Adler number, its scan page, our Greek, our English, and the scholars’ English.
The grading
Then we graded a stratified sample of 49 entries the slow way: an independent AI judge per entry (deliberately from a different model family than the one that made the translations, for reasons that will become obvious), with both Greek texts on the desk as arbiters. Every discrepancy had to be traced to its source: our translation, the scholars’ translation, or the two editions genuinely disagreeing.
The first pass judged 49 entries chosen to be interesting: long, quotation-heavy, the kind where translation goes wrong. That is the right way to find failure modes and the wrong way to count them, so we later drew 150 more entries uniformly at random from all 27,149 and judged those too. The corpus figures below come from the random draw; the error taxonomy comes from both.
- 77% of entries fully faithful, 21% with minor issues, 2% with major errors (95% intervals: 70–83%, 15–28%, 0.7–5.7%). The faults are dominated by flattened nuance and small mistranslations (a Greek second-person rendered as third, an “in part” dropped), not fabrication. The three major failures are all the same shape: a garbled word read as a different real word and rendered fluently. In one, the OCR’s corrupt εξι was taken for the numeral six and became “six books of erotic works.”
- Recitation (translating from memory instead of from the page) did not appear at all in the 150 random entries (95% interval 0–2.5%, so at most a few hundred entries in the whole book). It is real but rare: three confirmed cases have surfaced across every pass, including one where our OCR reads μειζον, “greater,” and our English asserts what Adler’s text says instead. Readers of our earlier note on models reciting canonical texts will know why we hunt this failure specifically: it is the one that cannot be caught by reading the English alone.
- The judges also documented the opposite, anti-recitation: our translation preserving Bekker’s readings against the standard text. Where Bekker prints Aristarchus and Adler prints Archilochus, our English says Aristarchus. Where Bekker’s entry ends two sentences before Adler’s, our translation stops exactly where our page stops. That is what translating-from-source looks like when you can check.
Two hundred judged entries is a sample, not a census, and the intervals above say so. But the shape of the result matches what we found reading pages by hand: reliable for reading and finding, with a known failure mode (confident fluency over corrupt input) concentrated exactly where the OCR met the hardest Greek. The interesting-entry pass and the random pass disagreed by twenty points on how often we are perfectly faithful, which is worth stating plainly: how you choose what to check determines what you find, and only one of those two designs can be quoted as a rate.
One more check belongs here, because a reader should have thought of it. Our judges were shown the scholars’ translation while grading ours, and the Suda On Line is in every model’s training data, so perhaps they were scoring agreement with SOL and calling it fidelity. We tested it: half the random sample was judged with SOL’s translation in the packet, half with it removed, assigned at random. The verdicts came out identical, seventeen flagged entries in each arm. The judges were reading the Greek. (One difference did show up: the blind judges catalogued half again as many individual faults: seeing the answer makes a grader less curious about the details, without changing the grade.)
The graders, graded
Three findings came out of the comparison that are not about our translation at all.
First: the editions disagree far more than anyone tells you. 42 of our 49 sampled entries contained genuine differences between Bekker’s 1854 text and Adler’s critical text: different quotations included, different case readings, different entry boundaries. Anyone who graded an AI translation of Bekker against a translation of Adler without checking the Greek would count every one of those as an AI error. Most published evaluations of historical-text translation do exactly that. The aligned corpus is, as a side effect, the raw material for a systematic collation of the two editions, something that does not currently exist.
Second: the reference makes mistakes too. In 5 of 49 entries the judges found the scholarly translation erring against its own Greek: small things, a word construed against its attested sense, a misparsed genitive, the ordinary residue of any twenty-five-year project. (We first counted six. On re-examination with SOL’s own annotations in view, one claim dissolved: the error in that entry was ours, and we have corrected this paragraph accordingly. Every surviving claim was re-adjudicated the same way, with instructions to extend SOL maximal charity.) We say this with gratitude, not glee: the Suda On Line is the only reason this measurement exists, and we will report what we found upstream, with the evidence, the way their own contributors would. When a machine translation and a human translation can each be checked against the Greek, each becomes an error-detector for the other. That reciprocity, not replacement, is the actual relationship between AI translation and scholarship that this experiment demonstrates.
Third, and to us most instructive: a judge that cannot do the task does not say so; it says “faithful.” Before scaling the grading, we tested whether a cheap model from the same family as our translator could stand in for the expensive judges. Given the identical entries, the identical rubric, and errors we knew were present because they were already catalogued, it called 47 of 49 entries clean, missing 90% of the faults, with zero false alarms. Agreement with the gold labels: κ = 0.107, barely above chance. A bigger model of the same family with four times the thinking budget did no better. We initially read this as the obvious morality tale (a model going easy on its own family’s homework), and an earlier version of this paragraph said exactly that.
Then we ran the control an honest reviewer would demand, and the morality tale fell apart. We had fresh translations of the same entries made by the judge’s rival family, and had both families grade the identical artifacts: on focused Greek-plus-translation pairs, the cheap judge found errors at the same rate as its expensive rival, including, decisively, 6 of 7 known error-entries in its own family’s work once we re-presented them in the focused format. The blindness was never loyalty. It was task collapse: our original test asked the judge to first locate one entry’s translation inside a full translated page and then grade it, and the cheap model cannot do that composite task, so it silently returned “faithful,” forty-seven times, instead of once saying “I can’t.” The practical rule survives in sharper form: validate a cheap judge on the exact task format you will deploy it on, because its incapacity will arrive dressed as approval. And the methodological rule above it: when a result flatters your suspicion of someone else’s model, that is precisely the result to attack with a control.
The census
The same cheap model had gone two-for-two on the coarse checks in the small test, so we gave it what looked like the job it could hold: a categorical sweep of all 27,149 aligned entries, twenty-four at a time. Is the translation present? Is the alignment right? Does the English contradict the Greek in the direction of received knowledge? It returned 97.4% of entries clean, with 446 flagged translation-not-found, 342 for alignment, and 66 for possible recitation, for $1.11.
Then we did to the sweep what we had just done to the judge, and it failed too. The 49 gold-labelled entries were inside those 27,149, so the sweep could be scored against them: it missed all three of the known events (the real recitation case, the real misalignment) and added one false alarm. We then seeded twelve entries with a deliberate contradiction, a name in the English that the Greek does not contain, and ran them through both call formats. The twenty-four-at-a-time format caught four of twelve. One entry per call caught five. Neither is a measuring instrument.
So the numbers above are a screen, not a census. We then sent all sixty-six recitation candidates to cross-family judges, one entry each, and the screen came out worse than its sensitivity alone suggested: two of the sixty-six were real. One is lovely: our Greek reads δóρυ, spear, and our English says a small Persian sword, which is what an akinakes actually was. In the other, our page reads “died aged 58,” the reading Adler prints, while our English gives 69, the age every reference book assigns Aeschylus. Both contradict our own scan in the direction of received knowledge. That is the fingerprint.
The other sixty-four failed in one consistent way, and it is the more interesting half. The screen had flagged any disagreement between our English and the scholars’, without regard to direction, and most of those disagreements turned out to be our translation faithfully following a corrupted line of our own scan, and so departing from Adler. That is the mirror image of recitation: evidence the translator was reading the page rather than remembering the book. A detector built to catch a machine trusting its memory had mostly caught the machine trusting its eyes. It also, while failing, handed us something useful: a list of real scanning errors, spotted in passing by judges who were busy rejecting the flag attached to them.
Which leaves the number itself. We went and got it the slow way: 150 entries drawn at random, judged one at a time by judges that can do the task. None of them showed recitation: an interval of 0 to 2.5%, so at most a few hundred entries across the whole book, and probably far fewer. Three confirmed cases have surfaced in total, from every pass combined. That is the honest shape of it: the failure is real, we can show you it happening, and it is rare. A cheap sweep turned out to be a fair way to find candidates and a bad way to count them, and the two it did find were worth the dollar.
Everything in this note that touched a paid API cost $1.18, the sweep included, and the experiments that demolished two of our own conclusions cost about four cents of that. Against the sixteen years of volunteer labour all of it is measured with, the asymmetry is the fact of our era; it is also, we think, the reason to be slow about believing cheap numbers. The scholarship was the expensive part and it is the part that held.
What happens with it
The 750 flagged entries become a repair worklist before most readers ever hit them. The alignment becomes Adler-number citation for our Suda: cite “Suda δ 872” and land on the leaf. And the aligned corpus (two Greek edition texts, our OCR, our translation, the scholars’ translation, per-entry quality labels) is being prepared for release as an open dataset, because parallel corpora for ancient Greek with vetted ground truth essentially do not exist at this scale, and we would like the next pipeline to be graded harder than ours was. The Suda On Line’s translations are CC BY-NC-SA, and the dataset will honor that: a full aligned version under their terms, and an our-columns-only version under CC BY-SA that anyone can rejoin.
One book in our library came with its own answer key. We used it, and the grade was: good, honestly flawed, measurably improving, and now we know which 2.6% of the entries to fix first. The other 34,000 books have no answer key. The method (align, arbitrate against the source, never let a model grade its relatives) is how we intend to grade ourselves anyway.
Corrections
This note is a running record of an investigation, and the investigation kept overturning it. Every change to a factual claim is logged here rather than made quietly; the evidence for each is in the public issue linked below.
- 12 August 2026: errors found in the reference: 6 → 5. The six claims were re-adjudicated with the Suda On Line’s own annotations restored and instructions to extend it maximal charity. One dissolved: SOL’s reading faithfully rendered the transmitted Greek and our translation had normalised it. Five survived.
- 12 August 2026: “a model cannot grade its own family’s homework” withdrawn. A control (the same cheap judge, its rival’s translations, and its own family’s work re-presented as focused pairs) showed the blindness was task collapse on our packet format, not loyalty. The section now reports the corrected finding.
- 12 August 2026: the 27,149-entry census demoted to a screen. Scored against the gold labels it missed all three known events, and on seeded contradictions it caught four of twelve. Its clean-rate and recitation percentages were removed as statistics and are now described as candidates at measured low sensitivity.
- 12 August 2026: all 66 recitation candidates verified; 2 were real. Cross-family judges cleared 64 of them, most because our translation had faithfully followed a corrupted scan rather than a remembered text. The section now reports the verified outcome and states plainly that the corpus-wide rate remains unmeasured.
- 13 August 2026: corpus rates replaced with a random sample; the anchoring objection tested and answered. The published fidelity figures came from 49 deliberately interesting entries. We drew 150 at random and judged those: 77% fully faithful rather than 57%, with major errors at 2% where the first pass had found none, and no recitation at all (interval 0–2.5%, superseding the earlier 0.4–11%). Half that sample was judged blind to the scholars’ translation as a control; the verdicts were identical, so the judges were reading the Greek and not the answer key.
Read the Suda: sourcelibrary.org/book/suidae-lexicon-suidas. The Suda On Line lives at cs.uky.edu/~raphael/sol, twenty-five years of open scholarship without which none of this measurement would be possible. Methods, alignment code, and the full evaluation trail are public in issue #3884.
