Data ingestion through DataVec and transformation through Spark

Data can come from many sources and in many types, for example:

  • Log files
  • Text documents
  • Tabular data
  • Images
  • Videos

When working with neural nets, the end goal is to convert each data type into a collection of numerical values in a multidimensional array. Data could also need to be pre-processed before it can be used to train or test a net. Therefore, an ETL process is needed in most cases, which is a sometimes underestimated challenge that data scientists have to face when doing ML or DL. That's when the DL4J DataVec library comes to the rescue. After data is transformed through this library API, it comes into a format (vectors) understandable by neural networks, so DataVec ...

Get Hands-On Deep Learning with Apache Spark now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.