To make sense of why processing large files requires a new approach, I will briefly discuss computer memory and databases here. Feel free to skip ahead if you are familiar with how memory and databases work.
In computer hardware, memory is the medium that stores data, programs, and files. The hardware for computer memory is split into primary storage, which generally takes the form of RAM (random access memory), and secondary storage, which generally takes the form of a hard drive. Primary storage is used for storing the machine code and the data of active programs, while secondary storage is used for storing all data and files not currently in use. This division in memory usage reflects a few differences in ...
