Last year at the ARL symposium called Managing Digital Assets, I smiled inwardly to think of the grumbling likely to be kicked off by observations such as this by Donald Waters of the Mellon Foundation:

…what unites our interest in digitization and open access in a digital world is that the material becomes “processable,” or subject to computational processing. That is, the growth in the market of readers is not among groups of humans, but of machines, which are programmed to index, manipulate, mine, aggregate, decompose, and build up scholarly and other forms of content by algorithm. It is this machine “processability” that makes digitized objects and open access materials most valuable to scholars.

Protest, fume, rail against the subjection of your most exquisitely developed thought to the dumb imperatives of ones and zeros — Waters is absolutely right. You want influence? Or, more to the point, you want to avoid obliteration in the vast digital swamp? You’d better know how to demarcate, classify, and optimize your work for machine crunching — or find someone who does. And pray that the stewards of such crunching, the information managers you never thought about, have your best interests in mind.

All this occurred to me while reading a new D-Lib piece by Daniel Cohen, director of research projects at the very creative Center for History and New Media at George Mason University. Cohen also spoke at that ARL session, and at the time he sold me on Firefox scholar. His new article, “From Babel to Knowledge: Data Mining Large Digital Collections”, offers two nice examples of humantist-friendly manipulation of machine “processability.”

First: Syllabus Finder. Where was this godsend when I was inefficiently wandering around the chaff of the web, trying to crib ideas for my own syllabi? It’s a very sensible, very needed genre-based search tool. First, it defines “document classification” through a very simple dictionary of keywords endemic to syllabi (”assignment,” “office hours,” etc.). This classification is fed into Google through its API service, along with the search query, for optimized searches. The results can then be further refined through more automated analysis or combined with other search results.

I gave it a spin, using canonical writers from the Romantic era as search terms. To my happy surprise, good old Ashes Sparks & Hypertext, a six year old syllabus for a seminar I taught back in the day at UC Berkeley, kept showing up — and at or near the top of results. #1 for Coleridge, #2 for Byron, #1 for Wordsworth, #2 for Blake, #4 for Hemans. Yeah, baby! But we drop down to #14 for Keats, alas, and as for Shelley, he just kept coming up as a “fatal error,” an “Uncaught SoapFault exception.” So Syllabus Finder is a little buggy — but, dare we say it, a little poetic too. Maybe we’re just overly pleased by taking the silver for Byron:

I don’t know what to make of the way this tool seems to like the Ashes Sparks syllabus — certainly I indulged in no optimization — no thought about how the thing would be retrieved. The only distinguishing feature of that document, really, is that it’s been online steadily for six years. It’s just one of those Google-blessed mysteries. Perhaps cannier post-processing could promote syllabi more deserving of prominence. But Syllabus Finder works pretty well–I’d recommend it to a fledgling (and not-so-fledgling) instructor. As Cohen puts it, it does a surprisingly good job at achieving its modest goal – on most topics for every ten documents it retrieves, about nine are syllabi – and it has thus far found and catalogued over 600,000 syllabi, synthesizing a collection of course materials considerably larger than any created or maintained by a professional organization, educational institution, or library, or by any other effort on the web to aggregate syllabi.

A second and more complex treat today from the George Mason wizards: H-Bot. This is an automated historical fact finder that can field natural language queries. (Or at least ones that begin with ‘what’ or ‘when’ or ‘who’; it’s not ready to handle where, which, how, or why). The algorithm here is “question answering” — which involves the identification of relevant documents, some natural language processing (to interpret queries), and statistical/linguistic analysis of retrieved documents. (In addition to the D-Lib article, there’s more on H-bot here)

Playing with H-Bot is fun. When did Hitler die? The answer in an eyeblink, as the Germans say: April 30, 1945. When did Gandhi die? Here’s a quirk:

Well sure, but that wasn’t the Gandhi I meant. Interestingly, here’s what happens when I ask the same question but tell H-Bot not to “check trusted websites first”:

Here’s a case when the unfiltered swamp actually answered my question — or read my mind — better than “trusted websites.” Quantity over quality? Very sensibly, H-Bot demurs when I ask “Is God dead?” or “When did God die?” (”I’m sorry. I cannot provide any answer on that.”) But ask it “Who is God?” and H-Bot serves up a perky little answer:

Simple-minded? Sure. But viable. Arguments will rage, hairs will split, blood will spill, but our dumb machines have given us an efficient pulse of information in the midst of the cacophony, delivered by strategic sifting of great gobs of data.

Which brings us to a final point that Cohen makes about machine data-mining: “Quantity may make up for a lack of quality.” Even the most ardent humanist can’t deny: when it comes to information, we’ve got a whole lot of quantity these days. It’s how we draw from such quantity that counts.