Clojure: High Performance JVM Programming

Processing data with reducers

We will now study a simple example that depicts the use of reducers in efficiently processing large collections. For this example, we will use the iota library (https://github.com/thebusby/iota) to handle large memory-mapped files. The usage of the iota library with large files is encouraged as an efficient alternative to using concrete collections. For example, loading the records in a 1 GB TSV file as strings into a Clojure vector would consume over 10 GB of memory due to the inefficient storage of Java strings. The iota library avoids this by efficiently indexing and caching the contents of a large file, and this is done with much lower amount of memory overhead when compared to using concrete collections.

Note ...