Monkeying Around with Twitter Data

This week I was looking for articles about access to data and found a new interesting development. An Austin-based company called InfoChimps.org, that offers data sets for download, last week announced they were selling Twitter datasets. For quite a bit of money.

InfoChimps' mission "is to increase the world's access to structured data". The company appears to offer much data for free and prefers to be a platform where people may "post data under an open license". The data is available for browsing, and if the site doesn't actually have the data, it will point the user to where they can get the data for free. This is excellent for data sharing and access to large data sets. The site's homepage lists "Interesting Datasets" for perusal. I clicked on the first one, which was College Enrollment of Recent High School Completers 1960-2005. There is an intro paragraph to the data, where it's from, and an example of it. This set was prefaced with the caveat that the "files have data mixed with notes and references, multiple tables per sheet, and, worst of all, the table headers are not easily matched to their rows and columns." Kind of funny, yet good information to know!

So, Infochimps offers free datasets - fabulous. But their recent announcement to sell Twitter data met with some skeptical questions. The Read Write Web blog discusses this development in depth, describing the data, which isn't the full tweets, but hashtags, RTs, @ messages, and other associated info. This is apparently really useful and great information to have, and the developers at InfoChimps are hoping that people create interesting apps with this data. InfoChimps mined this data themselves, by hitting Twitter Developer API 20,000 times per hour (I almost know what this means). That's a LOT of data. Marshall Kirkpatrick, the author at RWW, questions the complete legality of selling this data, and worries Twitter is going to come a-knockin'. Many commenters on the blog entry also thought so. InfoChimps (last week) swore they were on solid legal ground, but a new post from yesterday on their blog revealed that Twitter had asked them to remove the datasets. While InfoChimps swears their data had nothing personally revealing, privacy concerns came from commenters and apparently Twitter, who claims they just want to prevent any 'malicious use' of the data.

I gleaned from the blog entries related to this issue that Twitter isn't very forthcoming with their data, which is making them a new Bad Guy of social media and data sharing. It's interesting that people are upset that Twitter won't share, but probably don't care as much if a university won't share some research datasets? Shouldn't data be data? Who cares if one is more sexy than another? Access is still important. InfoChimps sounds a little naive to think that selling Twitter datasets for $9000 wouldn't cause a stir, but maybe that's what they actually intended! At least it brought some mild attention to the issues at play here.