Digitizing Books: A Multi-Step Process

This week I looked at an article suggested by Prof. Winget, "A Mass Digitization Primer" by Juliet Sutherland, from the Summer 2008 issue of Library Trends. We've been discussing digitization projects throughout the class, and this article is a good place to look for a simple technical discussion of how these projects proceed.

Sutherland discusses the four levels of digitization, starting with page images. Page images have the advantage of replicating the actual experience of reading the book quite closely, as the layout is the same and images appear along with the text. They save future wear on the book by providing an easily browseable copy. However, they can be hard to view and require lots of scrolling on the screen. Sutherland mentions Google as the largest producer of scanned book images and states "Google's purpose is not to archive printed material, but rather to make it accessible via their expertise in search." This is an interesting perspective and makes it sound as if Google was merely looking for a large body of content to set their indexing and searching tools loose on, once they had conquered the web. Considered in this light their lack of interest in accuracy is not surprising.

Scanned images can be improved upon with OCR (optical character recognition) which produces the actual text, but here we run into problems based upon poor image quality. Intelligent OCR mechanisms that disambiguate words such as "modern" vs. "modem" based on the vocabulary contemporary to the text can reduce some of these problems.

Corrected OCR requires a lot of work, and there are a few different solutions. Double-keying, which is the practice of having two people type in the identical text to reduce errors, is costly and time-consuming. Some more intelligent solutions are Recaptcha and Distributed Proofreaders. The former relies on using CAPTCHAs (those scrambled text boxes required to log into sites to prevent automated spam) and human intelligence to read distorted words. The key trick here is Recaptcha always asks for two words: one it knows, and one it doesn't know, and can therefore judge whether a user made the correct judgement or not. This is a clever solution that utlizes lots of distributed bits of time that a user barely notices to add up, like SETI@Home. The second solution, Distributed Proofreaders, requires a little more active participation from users willing to proof scanned texts.

The final step in digitization which will take us beyond print books is semantically tagging books, such as identifying named entities ("Washington", "Europe"). This will likely require a combination of automation and intelligent intervention. Projects such as the Biodiversity Heritage Library are combining all of these steps to produce scanned books online which are useful and reliable. We can see that the distributed approach common to many online projects helps produce a higher-quality product. Also, the issues of trust and reliability come up with many of these projects: are we more likely to trust the Biodiversity Heritage Library because of the prestigious institutions involved? Or are we more likely to trust something that has been professionally proofread? At the very least we can know that most of the errors we might encounter are technical, and not the fault of a human.