Scaling Machine Learning with Spark

Chapter 5. Feature Engineering

In machine learning, feature engineering is the process of using domain knowledge to select and transform the most relevant variables in the data to reach the goal of the machine learning process. Domain knowledge here refers to an understanding of the data and where it originated. In data science, it’s less about the tools and more about the data and problem itself, encompassing the general background in a specific field or vertical. For example, in finance it might involve familiarity with financial terms and the various possible applications of the data, such as loan scoring. Depending on the experience of the team working on the project, it might be necessary to consult a financial expert to create a representative feature to solve the problem at hand. Similarly, in healthcare tech you might work with a medical doctor to design the features, and knowledge of anatomy, biological systems, and medical conditions may be required.

The goal of feature engineering is for the final features to serve as proxies to the information the data contains about the world or the specific context where the problem takes place. Domain experience is what enables you to make those links, often intuitively. Using domain knowledge can help you simplify the challenges inherent in machine learning, improving your chances of reaching your business goal. Indeed, to be successful, you must have a solid understanding of the problem and the data you are working with.

Here’s ...