Intel Threading Building Blocks

Caches

The speed of processors has grown to be much faster than main memory. Making all of memory nearly as fast as a processor would simply prove too expensive for most computers. Instead, designers make small amounts of memory, known as caches, operate nearly as fast as the processor. The main memory can then be slower and more affordable. The hardware knows how to move information in and out of caches as needed, thereby adding to the number of places where data is shuffled on its journey between memory and the processor cores. Caches are critical in helping overcome the mismatch between memory speed and processor speed.

Virtually all computers use caches only for a temporary copy of data that should eventually reside in memory. Therefore, the function of a memory subsystem is to move data needed as input by each processing core to caches near that processor core, and to move data produced by the processing cores out to main memory. As data is read from memory into the caches, some data needs to be evicted from the cache. Cache designers work to make the data evicted be approximately the data least likely to be used again.

Once a processor accesses data, it is best to exhaust the program’s use of it while it is still in the cache. Continued usage will hold it in the cache, whereas prolonged inactivity will likely lead to its eviction and future usage will need to do a more expensive (slow) access to get the data. Furthermore, every time an additional thread runs on a processor core, ...