High Performance Python

Chapter 11. Using Less RAM

Questions You’ll Be Able to Answer After This Chapter

• Why should I use less RAM?
• Why are numpy and array better for storing lots of numbers?
• How can lots of text be efficiently stored in RAM?
• How could I count (approximately!) to 1e77 using just 1 byte?
• What are Bloom filters and why might I need them?

We rarely think about how much RAM we’re using until we run out of it. If you run out while scaling your code, it can become a sudden blocker. Fitting more into a machine’s RAM means fewer machines to manage, and it gives you a route to planning capacity for larger projects. Knowing why RAM gets eaten up and considering more efficient ways to use this scarce resource will help you deal with scaling issues.

Another route to saving RAM is to use containers that utilize features in your data for compression. In this chapter, we’ll look at tries (ordered tree data structures) and a DAWG that can compress a 1.1 GB set of strings down to just 254 MB with little change in performance. A third approach is to trade storage for accuracy. For this we’ll look at approximate counting and approximate set membership, which use dramatically less RAM than their exact counterparts.

A consideration with RAM usage is the notion that “data has mass.” The more there is of it, the slower it moves around. If you can be parsimonious in your use of RAM your data will probably get consumed faster, as it’ll move around buses faster and more of it will fit into constrained caches. ...