Foundations for Architecting Data Solutions

Chapter 1. Key Data Project Types and Considerations

The basis for any successful data project is a clear understanding of what you’re tasked to build and then understanding the major items that you need to consider in order to design a solid solution. We categorize data projects into three types that from our experience will typify many data projects. This categorization then allows us to explore the primary items we need to consider before starting on implementation. Not every project will fall neatly into one of these categories, and some projects might straddle these categories, but we feel that these project types will provide a useful framework for understanding your data use cases.

In this chapter, we begin by describing these major project types, followed by a description of the main items to consider, in general, for implementing solutions. We then take a deeper dive into these considerations for each project type.

Major Data Project Types

Let’s begin by describing the three project types that we use to categorize data projects:

Data pipelines and data staging

We can think of these as Extract, Transform, and Load (ETL)–type projects; in other words, these are projects that involve the collection, staging, storage, modeling, and so on of datasets. These are essentially projects that provide the basis for performing subsequent analysis and processing of data.

Data processing and analysis

These are projects that end in providing some kind of actionable value. This might ...