From Babel to Knowledge

In "From Babel to Knowledge: Data Mining Large Digital Collections" Daniel J. Cohen answers the question from last week's discussion, what do you do with a million books? Specifically, Cohen explores ways in which algorithms developed by computer scientists might be applied to humanities-themed collections to extract useful information from the chaos caused by an overabundance of sources. (Cohen artistically begins his article with a reference to a Borges short story involving a "library of babel.")

In the article, Cohen discusses two major cs techniques he has used to harvest historical information. (Cohen is squarely in the history-is-a-humanity camp.) The first method involves an API he designed to seek out and gather course syllabi on the web. His API uses a "dictionary of notions" to identify keywords and sequences of keywords common to course syllabi, and can be narrowed by keywords to look for specific topics. Cohen reports about a ninety percent success rate for his Syllabus Finder. (Interestingly, the concept of a "dictionary of notions" originated with one Hans Luhn who built the groundwork for the idea while creating an inverted index for cocktail recipes by ingredient in what may have been the most every-day practical application of information science to date.)

The second tool discussed by Cohen is H-Bot, a powerful search tool designed to answer specific historical fact-based questions (thereby freeing historians for what Cohen describes as "higher levels of history.) H-Bot, in addition to being designed to understand complex questions, is able to gather information on two levels: a quick mode which mines data from (more or less) trusted encyclopedias and dictionaries and a "pure" mode which extracts info from pages across the entire web. The program than evaluates the most recurring terms and determines which terms makes sense syntactically. Cohen illustrates the process with the example of the query "When did Charles Lindbergh fly to Paris?" The article did not mention how H-Bot deals with more complicated questions, but I took the liberty of experimenting with it a bit. H-bot sincerely apologized to me for not being able to tell me what caused the civil war, told me that Robert Anton Wilson shot Kennedy, and that the Beatles landed in New York on February 7th, 1964. (H-Bot can be found here: H-Bot.) Cohen did admit that answering questions was a lot more complicated than finding syllabi.

Beyond answering the question on how to deal with a million books (or attempting to, at any rate), Cohen's article directly touched on our last class discussion through its conclusions. Cohen's first conclusion is that more non-profit organizations need to come out with open API's. (Currently, H-Bot gives you the choice of using Google or Yahoo.) This sentiment seems to echo a lot of the mistrust leveled at Google. Second, Cohen asserts that free tools, no matter how imperfect, are more beneficial than pay services, which also casts misgivings onto Google Books. In his final conclusion, however, Cohen alleges that quantity is more important than quality in digital collections and objects, as the more individual sources that can be mined, than the greater chance an API has of providing the most relevant information. In this regard, Cohen seems to bolster at least partially Google's position.

Overall, I found Cohen's article to be a fairly interesting exploration of the juntion of information science and the humanities.