The Data Science Handbook

Chapter 14Databases

Databases play an important role in data science, but data scientists coming from backgrounds other than programming are often woefully ignorant of them. That was certainly my own experience! In fact, I didn't even really appreciate what the role of a “database” was and why you would use one as opposed to just files of data organized in a directory structure.

A database is ultimately just a framework for storing and efficiently accessing data. A prototypical database is a burly server, which holds more data than would fit into a normal computer, stores it in a way that it can be quickly accessed (this usually involves a ton of under-the-hood optimizations that the database user is blissfully ignorant of), and is standing at the ready to field requests from other computers to access or modify the data. The main advantage of a database relative to raw files is performance, especially if you are running a time-sensitive service (such as a web page). Databases also handle other overhead, such as keeping multiple copies of data synced up and moving data between different storage media.

On the smaller end, many pieces of single-computer software will run a database under the hood, using it as an efficient way to store one program's data. On the larger end, many databases span hundreds of different servers, with complicated protocols for syncing with each other, and users access them over the Internet or a local network. The idea of a single database in this case ...