Bad Data Handbook

Chapter 12. When Databases Attack: A Guide for When to Stick to Files

Tim McNamara

My Masters dissertation still feels likes a personal defeat. At least four months of the nine-month project were sabotaged because I didn’t understand the implications of the technology choices that I was making. This chapter will be a bit of a postmortem of the project and a walk-through of a strategy I should have used: storing plain-text data on-disk, instead of in a database.

History

But first, a little more about my story. In 2010, I was undertaking a Masters in Public Policy at Victoria University of Wellington. I was focusing my efforts on the arguments surrounding open data within the science sector. Specifically, I wanted to know: do the arguments that academics, officials, politicians, and the public align? I had the sense that open data and open government meant quite different things to different people, and I wanted to quantify that.

Getting access to information about what officials thought about open data was fairly easy. There was a major review of New Zealand’s publicly funded research bodies being undertaken at the time. Policy advice between departments is available under the Official Information 1982, which is New Zealand’s freedom of information statute. Information from politicians are even easier to find, as they talk all the time. All press releases are syndicated via scoop.co.nz. But what about the views of the public at large?

The political blogosphere is very active. ...