Chapter 11. Using Less RAM
We rarely think about how much RAM we’re using until we run out of it. If you run out while scaling your code, it can become a sudden blocker. Fitting more into a machine’s RAM means fewer machines to manage, and it gives you a route to planning capacity for larger projects. Knowing why RAM gets eaten up and considering more efficient ways to use this scarce resource will help you deal with scaling issues. We’ll use the Memory Profiler and IPython Memory Usage tools to measure the actual RAM usage, along with some tools that introspect objects to try to guess how much RAM they’re using.
Another route to saving RAM is to use containers that utilize features in your data for compression. In this chapter, we’ll look at a trie (ordered tree data structures) and a directed acyclic word graph (DAWG) that can compress a 1.2 GB set of strings down to just 30 MB with little change in performance. A third approach is to trade storage for accuracy. For this we’ll look at approximate counting and approximate set membership, which use dramatically less RAM than their exact counterparts.
A consideration with RAM usage is the notion that “data has ...
Get High Performance Python, 2nd Edition now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
