Government Data

Government data is hugely useful in lots of applications. Three examples are social science research, transparency in government, and an individual's decision about where to live. It appears government data is making its way to the web. Data.gov hosts datasets from many different federal agencies, such as the Environmental Protection Agency, the Bureau of Transportation Statistics, the Centers for Disease Control and Prevention, and the Department of Education. You can get the raw data - XML, CSV or text - or you can use (more or less) people-friendly tools from their tool catalog. The tool catalog is a collection of widgets and extraction tools that are supposed to make the data on the site useful. For example, the MyEnvironment Widget allows a user to enter her zip code and get all kinds of related environmental data on a single page, complete with pie charts, maps, and graphs. I discovered a landfill in my neighborhood that I had no idea about. Crap. Related to data.gov is DataMasher, which we looked at in class earlier in the semester. It takes government data and allows registered users to create "mashups" of that data. For example, there are mashups of Hate Crimes vs Population, State Education Spending vs SAT scores, and Federal Spending per Congressional Representative. There's definitely a push out there to go beyond making the data available, but also making it useful to developers and end users.

And it's not just the federal government that is putting its data online. San Francisco recently launched datasf.org, which has datasets related to elections, environment, geography, health, housing, public safety, public works, and transportation. The site also features an App Showcase, which at this point mostly contains interactive maps and phone apps related to crime and the BART trains. I think it's a good idea to put the applications in one place; it makes it plainly obvious how important open data is. It occurs to me that if I were an app builder in Austin, I'd start pushing Austin to open up its data. Economic incentives might make the open data/transparent government pill a little easier to swallow.

The W3C, the definitive authority on all things web, published a working draft of Publishing Open Government Data in early September of 2009 that provides "straightforward steps to publishing government data." They recommend making the data available in well-known machine-readable formats such as XML, RDF, and CSV, while also making the data available in human-readable formats by putting it in HTML and using "permanent" and findable URIs. They provide guidance on documenting the data, preserving the data, not creating interfaces that prohibit access by other applications, and choosing what data to publish. The last point is an interesting one. How do you choose what data to publish? One of the people behind datasf.org, Jay Nath, writes about the 80-20 rule of government data. He discovered by analyzing downloads on the Washington DC Data Catalog that 80% of dataset downloads came from 20% of the datasets, mostly crime- and 311-related data. Nath and his group prioritized the data they made available on datasf.org based on these kinds of download statistics. One has to wonder, though, if the most unpopular datasets will ever make it online. If the only motivation for opening up government data is economic, then we might not ever see true transparency.