The Data Science Handbook, 2nd Edition

14Databases

Databases play an important role in data science, but data scientists coming from backgrounds other than programming are often woefully ignorant of them. That was certainly my own experience! In fact, I didn’t even really appreciate what the role of a “database” was and why you would use one as opposed to just files of data organized in a directory structure.

A database is ultimately just a framework for storing and efficiently accessing data. A prototypical database is a burly server, which holds more data than would fit into a normal computer, stores it in a way that it can be quickly accessed (this usually involves a ton of under‐the‐hood optimizations that the database user is blissfully ignorant of), and is standing at the ready to field requests from other computers to access or modify the data. The main advantage of a database relative to raw files is performance, especially if you are running a time‐sensitive service (such as a web page). If you somehow know exactly the computation you want to do, and when you want it done, then a batch process might be better; but if you want flexibility with low latency, then databases are the answer. Databases also handle other overhead, such as keeping multiple copies of data synced up and moving data between different storage media.

On the smaller end, many pieces of single‐computer software will run a database under the hood, using it as an efficient way to store one program’s data. On the larger end, some databases ...