Bioinformatics with Python Cookbook - Second Edition

Computing sequencing statistics using Spark

If you need to use parallel computing, then Spark is one alternative to Dask. Its abstraction level is slightly higher. This gives you less granular control over the computation, but is more declarative to code. Spark is also somewhat language agnostic (it is actually Java/Scala-based). Here, we will compute some very basic statistics over the Parquet dataset that we generated in the previous recipe.