The Hitchhiker's Guide to Python

Chapter 11. Data Persistence

We mentioned ZIP compression and pickling already in “Data Serialization”, so there isn’t much left to cover besides databases in this chapter.

This chapter is mostly about Python libraries that interface with relational databases. These are the kinds of database we normally think about—they contain structured data stored in tables and are accessed using SQL.¹

Structured Files

We already mentioned tools for JSON, XML, and ZIP files in Chapter 9, and pickling and XDR when talking about serialization. We recommend PyYAML (get it via pip install pyyaml) to parse YAML. Python also has tools in its Standard Library for CSV, *.netrc used by some FTP clients, *.plist files used in OS X, and a dialect of the Windows INI format via configparser.²

Also, there’s a persistent key-value store available via the shelve module in Python’s Standard Library. Its backend is the best available variant of the database manager (dbm—a key-value database) on your computer:³

>>> import shelve
>>>
>>> with shelve.open('my_shelf') as s:
...     s['d'] = {'key': 'value'}
...
>>> s = shelve.open('my_shelf', 'r')
>>> s['d']
{'key': 'value'}

You can check which database backend you’re using like this:

>>> import dbm
>>> dbm.whichdb('my_shelf')
'dbm.gnu'

And you can get the GNU implementation of dbm here for Windows, or check your package manager (brew, apt, yum) first, then try the dbm source code.

Database Libraries

The Python Database API (DB-API2) defines a standard ...