Digitization Primer

I decided to check out Juliet Sutherland's "A Mass Digitization Primer" article from about a year ago. Sutherland is the head of the Distributed Proofreaders project, a site that allows volunteers to proofread pages from public domain books.

Sutherland provides a quick breakdown of mass digitization for books:

• Page Pictures
• Raw OCR
• Corrected OCR
• Semantic Coding

Each category is followed with strengths and weaknesses. Page pictures are naturally good for capturing the physical reality of a book, indicating paper type, stains and foxing, margins, notations, and so on. As a previous reading mentioned, digitization can force ambiguity out of the picture, a key example being the dashes used by Emily Dickinson. Any OCR (and any print version for that matter) will necessarily force these dashes, each unique and of varying force, length, and angle, into either a figure, en, or em dash. For that matter it's worthwhile to note Dickinson's marks are only "ambiguous" to a modern reader accustomed to printed type; the actual marks Dickinson made do not themselves refer to a typist's set of dashes. To me this is a really incredible application of digitization: finally moving manuscript writings closer to their original context and appearance (in a letter, in a margin).

I'm surprised that Sutherland doesn't mention digitization of other parts of the book like the spine, headband, jacket, binding, endpapers and so on that would fall into this section. There must be some issues regarding the best way to proceed digitizing these, but I suppose the interested audience is not substantial for this article.

Shortcoming to page pictures are fairly obvious; there's nothing new discussed here, but Sutherland does point the wide range in quality page pictures encompass, Google marks the lower end of that spectrum.

Raw OCR is typically included as a next step after actual photographing of a page. Sutherland states that "commercial OCR of modem [sic] printed materials is now virtually perfect." She uses that typo to point out some of the shortcomings of OCR with older texts with their odd fonts, partial punctuation spacing, deteriorated paper and so on.

Corrected OCR fixes these problems by ensuring fidelity to the printed text. Humans can do this, but the labor is expensive. It's even expensive when it's cheap: double-key entry from workers in foreign countries is still costly for the amount text and books corrected.

Alternatives exist, and as Meg blogged a few weeks ago Recaptcha is one such solution. This service uses the ambiguous words an OCR device encounters as human-verification criteria for various sites. As users type the words they see in the image Recaptcha provides, they also provide a +1 for their reading of that word. Ultimately, this resolves the OCR's ambiguity. Recaptcha has important shortcomings though. It can't present words with old orthography or spacing, it will not present words the OCR device did not tag as ambiguous or words which it simply skipped on account of illegibility, and word context is stripped.

Sutherland's own Distributed Proofreaders is another solution. As expected accuracy is high, output is relatively low.

Semantic coding is another topic we've covered in class, particularly in regard to XML's and JSON's appropriateness for such marking. The semantic encoding mentioned here is fairly pedestrian as such things go: chapter headings or is Washington a city, state or person?, etc., but Sutherland leaves space for more interesting semantic markup in the future.

Sutherland concludes that digitization should be seen as a multistep process. I agree 100%, and I think it's clear Google similarly conceives of digitization as iterative, with it's "Getting there!" disposition toward accurate data and metadata (compared to OCA's metadata which is on the whole much much better). The slippery slope to this approach is that every step is a little less accomplished than it could be; raw OCR quality is contingent on page picture quality, corrected OCR needs somewhat accurate raw OCR for context. With that in mind I wonder if it would not be better to more closely consider first steps rather than applying so much functionality and completeness to later steps.