Two weeks ago we wrote that the Internet Archive's own OCR text is good enough to take, free, wherever it agrees with our reading on nineteen words in twenty. That test compares two machines with each other. This note compares both of them with a third party: 122 English pages that volunteers have proofread letter by letter against the same scans. On the same page, our reader is better on 57, the Archive's on 5, and 43 are a tie. The result that matters to anyone quoting from Archive text is smaller and worse, and it is preliminary. In this first measurement the Archive silently misreads about 1 percent of the numbers printed on a page, a 5180 that comes out as 6180, with nothing in the text to show that anything went wrong. If you are quoting a date, a page reference or a sum from Archive OCR, open the image.
This partly corrects The Reading We Did Not Have to Pay For, which reported 66,322 pages accepted from the Archive on the strength of agreement with our own reading. Agreement between two readers is not accuracy, and that note said so. What it could not say, because nothing independent of both readers had been measured, is how large the gap was or where it fell. Now we know one place it falls: in the digits.
What was measured
The reference pages are the first English pages in our OCR benchmark, built the way How We Measure OCR Quality describes: one interior page per book, sealed before anyone looked at an engine's output, with an independent transcription of that leaf as the truth. For 87 pages the truth is a proofread Page: transcription on English Wikisource made from the same Internet Archive scan we hold. For 35 more it is Project Gutenberg's HTML text of the same edition, cut at its printed page breaks. Every page's leaf identity was checked against the image before scoring, because the Archive's leaf numbering and our page numbering do not always agree; 25 of the first 134 images turned out to be a neighbouring leaf, 14 were recovered by searching the neighbours, and the rest were excluded with the reason recorded.
Two readings of each page were scored against the truth. The first is the Archive's own text file for that leaf, produced by ABBYY FineReader or Tesseract depending on the year the Archive ran it. The second is a fresh reading by our production engine, Gemini Flash-Lite, with the production prompt, one page at a time, and one retry when the model refused. The score is character error rate on letters and digits after trimming an engine's unmatched running heads. The 122 books fall into four cells by date of edition and by whether the page is dense with numbers, which here means the reference carries at least six numbers of two to four digits.
The question behind the design was practical. We had proposed routing some English Archive books past the paid reader entirely, and a spot check of four family-history books that the earlier test had accepted found years misread, 1836 as 1886. Dates in a county history or a genealogy are the whole point of the page, so a misread digit there is not a typo but a false fact. We wanted to know how often it happens, measured against something independent of both readers.
What the numbers say
The table gives, for each cell, the pooled character error rate of each reading, the count of pages on which one reading beat the other by more than a fifth of a percentage point, and the number misreads described in the next section. The two cells before 1880 hold fewer than thirty books each and are exploratory; the two cells from 1880 to 1930 are large enough to be directional but not to decide anything alone.
| Cell (books) | Archive error rate | Our reader | Same page: ours better / Archive better / tie | Silent number misreads, Archive / ours | Our reader refused |
|---|---|---|---|---|---|
| Before 1880, prose (25) | 6.6% | 1.5% | 15 / 1 / 7 | 1 of 10 / 0 of 10 | 2 |
| Before 1880, number-dense (22) | 6.4% | 4.0% | 11 / 1 / 7 | 2 of 265 / 0 of 194 | 3 |
| 1880–1930, prose (41) | 0.62% | 0.34% | 16 / 1 / 19 | 0 of 12 / 0 of 12 | 5 |
| 1880–1930, number-dense (34) | 3.5% | 4.9% | 15 / 2 / 10 | 6 of 599 / 0 of 534 | 7 |
| All (122) | 3.95% [2.3, 6.1] | 2.62% [1.2, 4.6] | 57 / 5 / 43 | 9 of 886 (1.0% [0.3, 2.0]) / 0 of 750 | 17 (14%) |
Error rates are pooled over the cell's pages; brackets are 95 percent bootstrap intervals. The paired columns count only the 105 pages both readers produced. A tie is a difference under 0.2 percentage points. Refusals are counted after the retry.
Character error rate against the proofread text, by cell
Lower is better. Bars are the pooled rate; whiskers the 95% interval. Hover, tap or tab to a row for the figures.
All (122 books): Internet Archive 3.95% (2.3 to 6.1), our reader 2.62% (1.2 to 4.5).
Numbers from the run's report (run en-ocr-ref-5124-2026-09, 122 English pages scored against Wikisource and Project Gutenberg transcriptions); intervals are 95 percent bootstrap intervals. The table below gives the same figures.
Same page, both readers: which came closer to the proofread text
Each bar is the pages both readers produced in that cell. Hover, tap or tab to a row for the counts.
All, 105 pages: our reader closer on 57, a tie on 43, the Archive closer on 5.
Numbers from the run's report (run en-ocr-ref-5124-2026-09, 122 English pages scored against Wikisource and Project Gutenberg transcriptions); intervals are 95 percent bootstrap intervals. The table below gives the same figures.
Our reader is better on the same page in every cell, and the sign test on the paired pages gives p below 0.001. In the one cell where the Archive's pooled error rate is lower, 1880 to 1930 number-dense, the difference is one page: a two-column index in Lane's Arabian Society in the Middle Ages (1883) that our reader laid out as a row-wise table, reading across both columns instead of down each. That is an error of order, not of reading, and 39 of the 63 numbers our reader is scored as dropping in that cell are on that one page.
The Archive's worst page is not a misreading either. It is a page of Thomas Taylor's 1823 Select Works of Porphyry that Tesseract read as Greek: the English text comes out as ΟΝ ΤῊ ΟΑΥ̓Ε… and scores an 87 percent error rate. Anyone can see that page is broken. The problem this note is about is the pages nobody can see are broken.
PrintedAnd again, that the gate, which is pervious by men, should be said to be turned towards the north wind…
The Archive's text fileΑπᾶ αραίη, {παῦ {Π6 ραίβ, Ψ ΒΊΟΝ 15. ρθυνίοβ ὈΥ͂ ΤΠΕΏ,…
Porphyry, Select Works, tr. Thomas Taylor (1823), p. 173
Silent number misreads
A conventional OCR engine that cannot read a character usually leaves a trace. It writes 1s79 for 1879, ii for 11, loi for 101. Those are visibly garbled: a reader, or a program, can see that something is wrong and go to the image. The dangerous case is the one where the engine reads a printed number as a different, well-formed number. We call that a silent misread, and we counted it strictly: the page must print the reference's number, which we checked by opening the image for every candidate, and the engine must have written a different number one glyph away.
This finding is preliminary, and the numbers should be read with their intervals. Across the 122 pages the Archive's text does this to 9 of the 886 numbers printed, or 1.0 percent with a 95 percent interval from 0.3 to 2.0, on six different pages. (Corrected on 18 October 2026: an earlier version counted 13. Re-reading the Archive's text files showed that four of them, such as “Prop. 1 6” for 16, are the right digits split by a space, and they are no longer counted as misreads.) On the 105 pages both readers produced, the count is 8 against 0. Eight events is a small base. Our reader produced no silent misread that survived the image check on any of its 750 numbers, and zero observed in 750 does not show that its rate is zero; it bounds the rate below about 0.4 percent at 95 percent confidence. The honest statement is that the Archive's rate is somewhere between one in 340 and one in 50, and ours is below one in 250 and may be far lower. A dedicated numbers test is underway, in which every disagreement between the two engines on a printed number is adjudicated on the page image, together with an estimate of the errors both engines could share; it is tracked as issue #5224.
The page at the top of this note is page 300 of Edward Sachau's 1879 translation of al-Bīrūnī's The Chronology of Ancient Nations. It prints “others as 5180 years” in the text and “5180 years” again as a displayed line. The Archive's text file has 6180 in both places. We read the page by eye to be sure and checked the Archive's file directly. A reader searching the Archive's text for the interval between Adam and Alexander would not find it, and a reader who copied the figure would copy the wrong one, with no sign that it was wrong.
PrintedFor some of them count the interval between Adam and Alexander as 5069 years, others as 5180 years.
The Archive's text fileFor some of them count the interval between Adam and Alexander as 5069 years, others as 6180 years.
al-Bīrūnī, The Chronology of Ancient Nations, tr. Sachau (1879), p. 300
On a page of the 1883 White translation of Spinoza's Ethic Demonstrated in Geometrical Order, the printed cross-references “Prop. 16, pt. 2” and “Prop. 11, pt. 2” come out as “Prop. 1 6” and “Prop, 1 1”: the digits are right but split by a space, so a search for Prop. 16 misses them and our scorer read them as two numbers. In James Mooney's 1896 The Ghost-Dance Religion, a source citation printed “(G. D., 36; War, 8.)” becomes 30 on the page. The remaining cases in the run are of that second kind: a page reference in an index, a footnote, a year, each changed by one digit into another plausible number. Most of them are in the 1880 to 1930 number-dense cell, which is the cell a county history or a genealogy would fall in.
Printed…joined Big Foot or went on to Pine Ridge. (G. D., 36; War, 8.)
The Archive's text file…Joined Big Foot or went on to Pine Ridge. (G. D., 30'; War, 8.)
Mooney, The Ghost-Dance Religion (1896), p. 860
The reference itself is not perfect. On four numbers where the engines disagreed with Wikisource, the image sided with the engines: the volunteers had made the error. Those four are excluded from the counts above. The reference error rate is small but it is not zero, and a measurement that trusted the reference blindly would have charged the engines with four misreads they did not make.
Where our reader is worse
Three things count against our own engine in this run, and they should be said as plainly as the number misreads.
It refuses. On 34 of the 122 pages the model declined to transcribe on the first attempt, and on 17 it still declined after a retry, which is 14 percent of pages with an interval from 9 to 21. These are recitation refusals: the model recognises a published text and will not reproduce it. The Archive's reader never refuses anything. A page our reader will not read is a page the Archive has read, and on the evidence here its reading of a prose page from 1880 to 1930 is good.
It drops numbers. Where the Archive loses 3.7 percent of the numbers printed, our reader loses 8.4 percent. On the paired pages that is 63 numbers against 28, and 39 of the 63 are the one index page described above; but the pooled figure is what it is.
It files footnotes as notes. The production prompt asks the model to mark its own interpretive remarks inside a <note> tag, and on these pages the model put the printed footnotes there too. Any surface of ours that hides notes hides those footnotes. That is a defect in our text, not the Archive's, and it is being tracked separately.
What this changes in the earlier note
The earlier note took a book's Archive text when the median agreement with our own reading across the sampled pages was 0.85 or better, and argued that where two independent readers match on nineteen words in twenty, the free text is at least as reliable as the text we would have paid for. That argument is about words. A silent digit misread changes one word on a page, and a book that agrees at 0.93 has room for many of them. Nothing in the agreement score sees a 5180 that became 6180, because the score does not know that one word on the page matters more than the others.
The 66,322 accepted pages were not re-measured here; this run drew its own sample, one page per book, from a different pool. So this note does not say how many of those pages carry a misread digit. It says that the Archive's reader makes this kind of error at about one number in a hundred on pages of this period and type, that the error leaves no trace in the text, and that the acceptance test was never designed to catch it. Those pages still carry the label that says where their text came from. A reader who finds one can reach us from the page, and the label is what will let us replace them in bulk when a better reading arrives.
Proposed routing, not yet applied
The run was made to answer a routing question: for which English Archive books may the Archive's own text replace a paid reading? The proposal below follows from the numbers above. It has not been applied. It waits on a human decision, as every engine decision in the library does.
- Our reader stays the default for English. The Archive lane earns no number-dense cohort at all, because a silent misread rate on printed numbers of around 1 percent, against none found in 750, is disqualifying for exactly the pages where numbers are the content.
- One candidate cohort, at the directional grade only: pages from 1880 to 1930 whose Archive text contains no number of two to four digits and is at least 90 percent Latin-script letters. There the Archive's error rate is 0.6 percent against our 0.3, and there is no number to get wrong. The script test is not optional; the Porphyry page above is why.
- When our reader refuses a page after the retry, write the Archive's text for that page rather than nothing.
- Prose before 1880 is not earned: 6.6 percent against 1.5.
What this cannot say
The pool is books that Wikisource volunteers chose to proofread or Distributed Proofreaders chose to transcribe. Those are, by selection, legible and well known. A random page from our English Archive holdings is likely to be harder than these, and 4,888 English Archive books we hold had no independent text at all by these matchers; that list is now the transcription queue.
The run measured our reader one page at a time, not in the batch mode that production uses for most pages, and it made one pass with no replication. The two cells before 1880 hold 25 and 22 books, under the threshold at which we treat a cell as more than exploratory. It covers no French and no English printed before 1600; both remain open. The leaf checks were made by a model and cross-checked against the engines, and 13 of them are queued for a human to confirm. And the reference error is not zero, as the four Wikisource cases show.
None of that touches the direction of the character-error result: on every kind of page measured, our reader is closer to the printed text than the Archive's. The number finding is weaker. It rests on eight paired events, and it shows that the Archive's silent misreads exist at a rate worth acting on, not that ours are absent. The caveats limit the size of the numbers; the sign holds for characters and is preliminary for digits.
Method and data
Page numbers in the captions are the printed folios; each picture opens that page in our reader. Method and code: scripts/eval/en-ocr-reference-5124.mjs in the public repository; the sealed registry of 216 drawn rows with every exclusion reason, the 122 reference transcriptions, both engines' outputs and scores, and the full report with breakdowns by Archive OCR engine, scanner and OCR year are committed beside it under run en-ocr-ref-5124-2026-09. The run cost under fifty cents.
