Training Data for Machine Learning

Chapter 4. Data Engineering

Introduction

In earlier chapters, you were introduced to abstract concepts. Now, we’ll move forward from that technical introduction to discuss implementation details and more subjective choices. I’ll show you how we work with the art of training data in practice as we walk through scaling to larger projects and optimizing performance.

Data ingestion is the first and one of the most important steps. And the first step to ingestion is setting up and using a training data system of record (SoR). An example of an SoR is a training data database.

Why is data ingestion hard? Many reasons. For example, training data is a relatively new concept, there are a variety of formatting and communication challenges. The volume, variety, and velocity of data vary, and there is a lack of well-established norms, leading to many ways to do it.

Also, there are many concepts, like using a training data database, and who wants to access what when; that may not be obvious, even to experienced engineers. Ingestion decisions ultimately determine query, access, and export considerations.

This chapter is organized into:

• Who wants to use the data and when they want to use it

• Why the data formats and communication methods matter; think “game of telephone”

• An introduction to a training data database as your system of record

• The technical basics of getting started

• Storage, media-specific needs, and versioning

• Commercial concerns of formatting and mapping data

• Data access, security, ...