Hands-On Deep Learning with Apache Spark

Loading multiple CSVs for RNN data pipelines

Before wrapping up this chapter, here are a few notes about how we can load multiple CSV files, each containing one sequence, for RNN training and testing data. We are assuming to have a dataset made of multiple CSV files stored in a cluster (it could be HDFS or an object storage such as Amazon S3 or Minio), where each file represents a sequence, each row of one file contains the values for one time step only, the number of rows could be different across files, and the header row could be present or missing in all files.

With reference to CSV files saved in an S3-based object storage (refer to Chapter 3, Extract, Transform, Load, Data Ingestion from S3, for more details), the Spark context has ...