O'Reilly logo

Clojure: High Performance JVM Programming by Akhil Wali, Shantanu Kumar, Eduardo Díaz

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Processing data with reducers

We will now study a simple example that depicts the use of reducers in efficiently processing large collections. For this example, we will use the iota library (https://github.com/thebusby/iota) to handle large memory-mapped files. The usage of the iota library with large files is encouraged as an efficient alternative to using concrete collections. For example, loading the records in a 1 GB TSV file as strings into a Clojure vector would consume over 10 GB of memory due to the inefficient storage of Java strings. The iota library avoids this by efficiently indexing and caching the contents of a large file, and this is done with much lower amount of memory overhead when compared to using concrete collections.

Note ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required