MongoDB and Python

Fast Accounting Pattern

Many of the applications people are building today are realtime with very large data sets. That is to say, users expect changes they make to be reflected within the application instantly. For example if a user wins a new high score in a multiplayer game, they expect the high score table in the game to be updated immediately. However, it may not be a single high score table which must be updated. Perhaps you are also ranking by high score this week, or this month, or even this year. Furthermore, as the application developer you may wish to keep a detailed log of each change—including when it occured, what the client IP address was, the software version of the client, etc.—per user for analytics purposes.

This pattern isn’t limited to high scores. Similar high performance accounting requirements exist for in-app social activity feeds, billing systems which charge per byte, and so on. Not only do these counts need to be fast to read from the database, they needs to be quick to write. Additionally, with potentially millions of users, the data set can grow very large, very quickly.

You might be tempted to keep only a detailed log, with one document per change. Totals for the various time periods can then be calculated by an aggregate query across the collection. This may work well initially, with only hundreds or thousands of documents to be aggregated to compute the result. However when the number of these documents grows into the millions or even billions—which ...