Data Curious

Chapter 4. Building Your Analytics Platform

Whether you aim to build a simple dashboard highlighting basic company KPIs or a real-time predictive model to recommend products to customers, you will want to work backwards from the output to define the data architecture, design, tools, and people you will need to leverage to achieve this. This chapter will start with an overview of the current landscape of tools, data needs, and associated costs. It will end with some best practices, including Agile project management and building with quality and stakeholder trust in mind.

Technological Options

The AI hype has gone through several cycles of boom and bust. However, since 2010, we have seen an aggressive and steady push to inject more analytics and data savviness into nearly all industries and companies. To remain relevant and competitive, leaders have been pressured to learn and incorporate data and data systems into business decisions and product development pipelines. While some companies have lagged behind due to a lack of infrastructure or data expertise, or resistance to change, most companies have begun to leverage data to inform business decisions, and others have fully leaned in to create data-driven products. The industry, company, product, and leadership all impact the stage of data adoption and maturity they are and in what ways data is being leveraged.

In Chapter 1, we introduced the concept of data-informed decision-making. Applying this concept to companies, this implies a company that is leveraging data inputs to help support or guide the decision-making process but ultimately using human judgment to land on final decisions or strategy. This can be seen through the use of data to inform inventory management, pricing strategies, marketing, and product improvement. This behavior is common among more traditional companies (e.g., finance, healthcare, entertainment, and retail) that have historically relied heavily on human experience to guide decision-making processes. As data integration has become increasingly important to gain or keep a competitive edge, these companies have worked to lean in to incorporate more data-led decision-making. Data-informed decisions rely heavily on historical views, descriptive statistics, and dashboards for KPI monitoring, but predictive analytics can be leveraged and aid in decision-making as well.

When applicable, a company may develop data-driven decision-making through products that leverage data as the primary input to decision-making. Examples of this exist in nearly every application or technology we interact with nowadays, from personalized “for you” experiences from Netflix, Spotify, and YouTube to recommended routes and predicted drive time from Google Maps, Waze, and Apple Maps, and self-driving capabilities from Tesla. The list seems endless to encapsulate our reliance on products and services that are built on the back of big data and automated ML systems that drive real-time predictions. These companies have done a great job of building systems that continually position data at the center of their product development processes. However, even in what might seem like a hands-off system of inputs and outputs, there is always a need for human oversight. As seen with products like ChatGPT and Facebook, there are safety and quality assurance angles that will always require a level of human monitoring.

A company can have both data-driven and data-informed decision-making aspects. A company like Netflix is data-driven when forming personalized experiences on their app and is data-informed when making decisions within their finance and HR departments. Since some companies embody both decision-making types, this expands the collection of data products they use and the sophistication of the tools, storage solutions, and processes they will need to develop.

It all starts with an idea! A product, a service, or a gap in the existing marketplace is identified. Once leaders commit to what they will build, they then move on to how. How will it get built? How will success be measured? At the start of any company’s formation, whether it was 100 years ago or a year ago, a myriad of key business decisions have been made to get the company up and running. As leaders then rush to build and prove relevance, the time to be thoughtful about analytics platform infrastructure is shortened, typically focused on short-term needs, and relies on existing expertise. Organizations will typically work to build a proof of concept on fragile infrastructure until value is shown. This leaves a lot of room for improvement, and typically companies evolve their perspectives and toolsets as they go through the different maturity stages that determine how easily they can pivot and adapt to changes in the technology landscape.

No matter what stage in this process you are currently in, whether it’s rushing through the early establishment decision-making process or thoughtfully thinking through new technology systems to help support the next stage and evolution of your company, this chapter will provide you with the right tools and considerations to start off on the right foot. This will involve making decisions around analytic tools, data storage, and data process and pipelines.

Analytic Tools

Your approach to data products will impact what your tech stack looks like (i.e., the tools you’ve decided to rely on for the data outputs you will be producing). Based on the data products you aim to establish within your organization (basic analytics, dashboards, forward-looking views, etc.), you will be faced with many tool options that have different data requirements and needs. The specific tools you choose will depend on the type of data you have, the questions you want to answer, and your technical skills and resources. Let’s take a look at the major tool types, while highlighting some options you can consider for each:

Spreadsheet software

The first and most common software that drives nearly all businesses in some fashion are spreadsheets. This software is powerful and can be used to store, analyze, visualize, and even predict data. Excel (Microsoft) and Google Sheets (G-suite) tend to dominate this space, and their usage numbers only continue to grow. The latest statistic on usage put Excel reliance by companies at about 63 percent.¹ When it comes to quick and accessible data analysis, nothing tops spreadsheets.

Typically, spreadsheets are used as an early solution for individuals and smaller teams. In my (Sarah’s) experience, I’ve even seen some larger companies relying on spreadsheets as the main data storage system as well! This becomes limiting once the data becomes too large and more sophisticated analytics are being requested. Still, much of the operational and ad hoc reporting covered in Chapter 2 can be handled with spreadsheet software.

Low-code/no-code data analytics

When your data volume or complexity grows, running repetitive data preparation, cleaning, and reporting tasks can become very time consuming, and you might want to turn to other software options. Low-code or no-code data analytics platforms like KNIME, RapidMiner, and Alteryx all offer drag-and-drop interfaces to help build data preparation workflows, analytics, and dashboarding. Each tool will provide a different set of features, and you’ll want to explore each to land on the one that works best for your needs.

Shifting your data prep, analysis, and visualization over to one of these tools will help you “work smarter, not harder” by streamlining your processes and empowering non-technical users to perform more sophisticated analysis. In addition to helping automate processes, added features of these tools include the ability to collaborate with your peers, version control, and data quality monitoring. Spreadsheets can be prone to human error and lack the necessary controls for data governance. These tools help circumvent those concerns.

BI software

The next group of tools are business intelligence tools like Tableau, Looker, Power BI, Qlik, or Incorta that can be used to visualize and analyze data in near real time, create interactive dashboards, and generate reports for decision-making. Features across these tools will vary, but many of them incorporate low-code or no-code interfaces, with drag-and-drop and pre-built components, making them accessible to users without extensive programming knowledge. Additionally, this software provides rich collaboration, sharing, and distribution options.

One major drawback to BI tools like this is that the ETL process required to get data ready to visualize will typically involve data pipelines built outside of these tools. Some tools, like Tableau, have developed prep tools like Tableau Prep that allow power users to own their pipeline workflows in a similar and familiar tool. However, when prep tools like this are not available, data engineering resources may be needed to help build the workflows and provide data tables for your use cases.

Statistical/machine learning (ML) software

For large and complex data sets or if you need more flexibility and customization beyond what traditional BI tools may provide, advanced statistical software like Python, R, or SAS can be leveraged. Libraries for visualization like Matplotlib, seaborn, and ggplot can be used to produce publication-quality graphs and charts, and libraries for machine learning like scikit-learn or Carat can be used to develop predictive models, classify data, or automate decision-making processes.

Python and R have grown in popularity due to the availability of powerful libraries, the myriad of software established to help manage, test, create, develop, and deploy ML solutions, and their open source nature. Open source refers to a software’s open and generally free nature, where the source code is made publicly available and generally anyone is free to use it without paying for a license. In addition to being free to use, the tools are improved through crowdsourcing, which means anybody can contribute to adding new features, fixing bugs, and optimizing performance.

Adopting statistical and machine learning software has a few barriers to entry, including learning a programming language, understanding statistical concepts, and familiarity with setting up development environments. With open source software, the barrier might be a little lower due to all the resources made available by the community, but still take time and effort to learn. Luckily, AutoML tools abstract away much of the complexity of machine learning, making it easier for non-experts to build and deploy models.

Depending on your needs, you may find yourself using one or many of the tools just presented. Most organizations, regardless of size, will have some reliance on spreadsheets and BI tools as a way of sharing data across the company. If you end up leveraging statistical or ML software, you’ll likely find yourself piping the output into a spreadsheet or BI tool for wider consumption by the company. Although reading from a code-heavy Jupyter Notebook may work for some folks, it won’t work for a wider audience.

Especially in large companies with established processes and numerous people working on similar goals, there may be an over-reliance on the limited features of spreadsheets in data-informed parts of the business. However, we are slowly starting to see more technical skills trickling into those areas, which could improve process and output. This is largely in part to the democratization of data and tools within organizations and a general emphasis on data literacy. This process involves simplifying and broadening the means to access and analyze data.

As you look to rationalize your existing analytical stack, it’s important to continue to watch the landscape as it continues to change and evolve. We are seeing this with continued integrations of technology tools, such as Google Sheets integrating with BigQuery and ChatGPT and books teaching Python for Excel.

Data Storage and Management

As you continue to focus on your data product and the inputs needed for the desired output, you will need to ask how much data you will need and where to store it.

Live snapshots are needed for real-time reporting and monitoring of week over week (WoW) and day over day (DoD) changes and help answer questions like these: How has revenue changed WoW or DoD? What was our biggest customer yesterday? Which product drove the highest revenue last week? Real-time reporting is great for things like assessing business health or giving a quick view of KPIs in an easy to read and understand format (typically a well-formatted spreadsheet or dashboard). This allows the business to react appropriately to any unexpected behavior in KPIs (good or bad).

The word live gives you the impression that the data is updated by the second (or even more frequently), but more realistically, there are lags between when the data was created, where it was stored, and the data pipelines built to get it to you. Depending on the setup of your existing ETL process and the complexity of the pipeline, this may be a delay of an hour or a day, or even longer. Either way, you might consider the latest view possible to be “live.”

Historical tables extend time horizons (or time frames) back to a year and beyond, allowing us to compare longer-term trends—things like evaluating year over year (YoY) and seasonal trends or driving predictive modeling. Changes in metrics today could be explained by a “normal” fluctuation, but you’d need the historical data to suss that out. Depending on how familiar you are with the metrics, KPIs, and the culture of your team or company, you will likely see a lot of reactionary behavior.

An example of this is advertiser spending during Ramadan, shown in Figure 4-1. In the Middle East and North Africa (MENA), advertisers will spend up on food delivery services in preparation for Eid. This will mean elevated levels of spending leading up to and during the month of Ramadan. However, the end of Eid signifies the end of Ramadan spending, and you will typically see a large revenue drop-off (similar to the cliff you see after New Year’s in Western and Chinese cultures). Understanding relevant regional behavior and having historical data to set and confirm expectations, and even predict the current year’s drop-off, can be helpful ways of guiding the business and can help it move from a reactive “fire drill” mindset to a more proactive “how do we offset the dip” one.

Thus, we come back to the question of “how much data to store and where?” The answer will depend on the needs and the cost. Data storage options and costs have evolved over the years, from floppy disks to cloud storage that allows for more data to be stored more reliably and more cost-effectively. This has enabled the volume and variety of data we can store to broaden and incorporate more use cases and business needs, and this sometimes leads to a “let’s track this for now and see if we need it later” mentality. As a result, it’s become common to have more data than is necessary to accommodate the needs of the business. However, even if that may be a sound approach, it’s important to understand the needs of all of the stakeholders before making a final decision on what data should live where and, at what storage level, to ensure that none of the “must-haves” is missed. Understanding which metrics are needed, by which teams, and in what historical time horizon, is important to make the right storage decisions. Teams predicting future revenue will need multiple years of revenue to identify and understand trends, seasonal patterns, holiday dependence, etc. Without easy access to historical data, the finance and revenue strategy teams are limited in their forecasting abilities.

Figure 4-1. Short versus longer time horizons

There are three major types of storage that you may consider when balancing the trade-off between cost and accessibility: hot, warm, and cold (see Figure 4-2). After doing some due diligence and gathering requirements, you can then make choices around which data to store and at what level.

Figure 4-2. Storage pyramid (adapted from an image by Anne Herreria)

Hot storage

To manage the demands of near-real-time reporting and live snapshots of data, you will need that data living in what’s called hot storage. This type of storage option is meant to store data that is accessed frequently, and it offers the highest level of performance. This is also typically more expensive than other storage options.

Warm storage

For data that is needed less frequently, warm storage is the next possible option that can handle data that needs to be readily available. This type of storage is less expensive than hot storage, but still offers relatively fast access times.

Cold storage

Cold storage is designed for data that is rarely accessed but needs to be stored for long periods of time. This type of storage is the least expensive, but offers the slowest access times. This could be data that is archived for compliance reasons.

Data Processes and Pipelines

To get the output you desire, you’ll need to ensure that the inputs are available how you need them, when you need them, and at the rate at which you need them.

If you’re working through an ad hoc (or one-off) analysis, you are typically OK working with snapshots of data provided to you by the source owner. If you don’t have ready access to certain types of data or data sets, you can typically request and work off of one-off data pulls that can help satisfy the need.

Additionally, ad hoc data pulls can be very helpful in building out proof of concept. When you are working to establish value from a new analysis, approach, or process, it’s likely that the data inputs you require don’t exist in a scaled format yet. Leveraging data snapshots is a great way to take a first pass, get feedback from stakeholders, iterate, and perfect the output. Building a proof of concept is an important part of the cycle of building data products at scale and can be used to position arguments for why to invest in a third-party vendor or in building internal pipelines for your use case. This will move your one-off task into a continuous need, and you’ll be presented with a few options of how to proceed from there.

If your team has dedicated BI or engineering support, you can work closely with them to get an end-to-end process to support the use case. Alternatively, you can leverage one of the many self-service tools mentioned in “Analytic Tools” to help automate the workflows yourself. In the absence of IT support or investment in self-serve data analytics tools, your team members may find other ways to build their pipelines, such as leveraging open source tools like Python, R, cron jobs, etc. In these situations, you may find scrappy team members building their own pipelines. Generally, building in isolation can have its pros and cons. On one hand, this provides the team a way to build out new processes quickly and without dependence on other teams. However, this can be problematic when the skills on the team are limited and can’t be extended to support production-style code and structure, and will need consistent oversight to ensure stability. This means the process will continue to live in a sub-optimal state.

Those companies that have optimally set up their teams with the tools or team structure to support the right data needs will have one additional layer of process that will help them run effectively: data governance. Data governance encompasses all the practices, processes, policies, and guidelines across the company that ensure data quality, integrity, and availability.

Once you’ve proved value from your data product, you will need to invest in time and resources to build out scaled and production-level backend data pipelines to ensure you are able to support the data product you’ve created on a continuous basis. You’ll need to establish things like the dimensions and measures you need, the level of granularity required, and the frequency at which data is to be refreshed.

How to Choose an Analytical Architecture

Landing on your analytical architecture can be challenging as you manage budget constraints, the skill makeup of your team, differing leadership styles, and the reality of how long it takes to make or create change within a team or organization. In this section, we will dive into the considerations factoring into these decisions.

Assessing Total Cost of Ownership

They say “nothing in life is free,” and this applies to your data product (or suite of data products) as well! To bring your vision to life, you’ll need a combination of people, tools, and processes. Each component has an associated cost that you’ll need to take into consideration as a part of the total cost of ownership (TCO). The TCO represents the cost associated with the product throughout its lifetime, from deployment to deprecation, and everything in between. In a world with many open source tools to leverage, making use of them can help with some of the cost, but there are still many other factors to consider:

People

The first consideration is people. Do you have the right skills on the team to accomplish and build what you’ve set out to produce? You’ll need to keep in mind analytics skills, modeling skills, engineering abilities, creativity, product management, etc. This cost will include salaries, for data scientists, data engineers, business analysts, and any other team members involved. In some cases, you will need to hire a consultancy or bring on partners to help evaluate and determine the best execution plan. Additional costs include employee benefits, recruitment costs, the cost of training team members on new tools and technologies, and ongoing training to keep their skills up to date.

Tools

The second consideration involves data, tools, and technology. Let’s talk about the data first—this includes the cost of acquiring and storing data, as well as any fees associated with using third-party data sources. Getting refined with exactly what data you need, and in what time horizons, can help bring down your total cost here. This includes third-party data and the needs and costs associated with enriching your first-party data. Third-party data and licenses can be very expensive, so think carefully about what subset of the data is necessary to meet the needs of your outlined use cases. Can you combine use cases across the company? Can you get ahead of all the needs and create a centralized source for this? As time goes on, the volume of data will continue to increase, so you’ll want to continue to re-evaluate the scope of the needs and adjust data storage options accordingly. What can be moved from hot to warm or warm to cold storage?

Next, you’ll need to think about the tools or technology needed, which can include expenses for licenses for analytical software tools, data governance tools, and any hardware or cloud services. These tool costs will vary and may need to be calculated based on the number of users (e.g., Tableau) or total usage (e.g., Amazon Redshift). Do you have the right tools in place to accomplish what you’ve set out to do? Again, many companies will “run” with a default choice that you’ll need to evaluate as a part of this process. Does it work for your needs? Does your team need a different tool? If so, what’s the cost of the tool, and how does it compare to the existing contract?

Sometimes you might decide that an existing tool is no longer sufficient, and migration to another tool is needed. In general, migrations can be incredibly lengthy processes, as teams take time to process the changes, go through training to understand the new tool, build in time to migrate, and then work to rebuild processes and reports into the new tool. It usually follows the cycle of grief, as your engineers and analysts go from denial to anger to bargaining, and finally to acceptance. Additionally, it can involve the cost of changing contracts, which in and of itself can land you with much higher costs than the original contract. Existing contracts are often years old and can have the advantage of legacy pricing, making them very hard to beat.

If there is no migration process or you’ve completed the migration, then comes the cost of maintaining and updating the technology and tools you are using, as well as any ongoing support costs.

Processes

The last consideration is the cost of processes, as there are implied costs to your resources when bad processes are being followed. Taking the time to evaluate how processes may be broken and can be improved can likely increase the value of the output of any limited resources you may have. A process improvement can include training your team on how to write more cost-efficient code and developing code review processes. Another example could be encouraging more close collaboration across teams and developing communication and knowledge sharing forums to improve output. It could also mean establishing roles and responsibilities around how teams engage with one another on projects or data workflows. The list is endless, but I hope this gives you an idea of how you can use process improvements to lower your overall costs.

Speed of Business Change

At this point, you’ve successfully evaluated the tools, the cost, and the general scope of needs for your data product, and you may start to ask…what comes next? Well, the unfortunate truth is that the journey only gets harder from here, so get prepared! It’s time to put the technology aside and see how well you can lean on your relationships and connect with and incentivize people to change. The first step will require buy-in from leadership teams and executives, which will be critical to any successful data project. You’ll want to be ready to answer questions like: why the change? And why now?

Your answers should consider the lens of your audience. Are they leaders who are less familiar with data technologies and may be less able to understand the implications of the changes being made or may be more resistant to change in general? Conversely, are they more data-savvy leaders who may be more likely to embrace new technologies and approaches but may also have higher expectations for the benefits that these changes will deliver? Either way, you’ll want to prepare the right pitch for your product, your proposal, and yourself. Remember that big investments are not just given to good ideas, they’re given to great leaders, so don’t forget about the importance of your role in bringing this to life.

You’ll likely end up in weeks, months, and sometimes even years of debate, as it takes time to gain consensus among leaders, approve budgets, allocate resources, and move into the execution phase. The time it will take to move through this phase will in large part depend on the scope of change requested and the data culture of the organization or team. Are you asking for additional head count? Ingestion of new data sources? A platform shift? Migration to the cloud? Major rebuilds of data pipelines and data stores? Not all leaders are fully leaned in to the data opportunity and will need more time and repetition to get them fully on board, and even those who are fully leaned in may not have the budget at the time of your request to get things approved. Be patient with this part of the process, but also don’t get discouraged and give up!

Now, let’s assume you’ve made it to the other side of the approval process and want to move into execution—let’s talk brass tacks on what it will take to follow through till the end of what you’ve promised. Many factors will impact how quickly you can move, including the size of your organization, the complexity of the existing data infrastructure, the scope of the changes being made, and the level of technical expertise of the leaders and employees involved. There are some major factors to keep in mind:

Size of the team

The first consideration that will impact the speed of change is the size of the company (or team). In smaller teams, changing data architecture may be less complex but still require significant effort and present unique challenges. Smaller organizations may have less technical expertise in-house and face resource constraints, such as limited budgets or a lack of dedicated IT staff. This can make it difficult to plan and execute changes effectively. Your challenge here will be finding a means to be effective with limited resources. Sometimes this can mean leaning on consultants where possible; other times you’ll need to move more slowly, and another option you might consider is deprioritizing team members’ projects to focus on the migration.

In larger businesses with more complex data infrastructures, there may be multiple legacy systems that need to be integrated or migrated, and the process of mapping data flows between these systems can be time consuming and prone to error. Efforts to migrate will involve a high level of coordination, likely across many people and multiple teams. As you work through this process, you may find a need to hire new personnel with specialized technical skills to manage the new architecture, which will add to the time requirements and complexity of the process. At times, transitioning the team will feel like redirecting a cruise ship through turbulent waters, and you have to be ready to steer.

Scope and complexity

There are various levels of change that can be introduced into a team, which include (but are not limited to) storage migration, application migration, business process migration, and data migration. Whether one or more of these changes are being proposed, you’ll want to think through the complexity of the existing infrastructure. Have you carefully analyzed and understood the existing data systems, including their structure, content, and dependencies? Typically, the longer the reliance on a set of tools or infrastructure, the higher the complexity and the more time you will need to spend carefully planning to manage potential points of failure or data losses. After the migration, you’ll need to set aside time for validation and potential decommissioning of old processes or tools. The scope of your changes should be well thought out and outlined, with a timeline including any changes and expectations you have of the teams you’ll be anticipating to move from one system to another.

Business leadership

Above and beyond the initial sign-off for you to start building, you will need leadership support all the way through the migration and adoption phases. You will likely run into challenges and roadblocks along the way and will need leadership on your side to help unblock them, so be sure to ask for this up front. Alignment and communication from leadership will help make your life easier as you start enforcing migration deadlines.

Overall, the challenges involved in migrating and changing analytical architecture can be significant and require careful planning, execution, and ongoing management to ensure a successful transition. Don’t forget that underlying all this change are humans, and research shows that habit formation and breaking can take anywhere between 18 and 254 days.² So remember to keep your empathy muscle strong and stay in close communication with all teams that will be impacted by the changes to ensure things move smoothly and successfully. Overcommunication can help bring people along on this journey with you by providing context around purpose-setting (why are we doing this) and timelines (when are we doing this).

How Will It Be Used?

Before the implementation phase starts, you’ll have to make a decision about how you will approach the migration: big bang or trickle. Big bang migrations involve moving all the data from an existing system to a new system in a single, large-scale operation. At small scales, this is a clean and effective approach. However, for larger, more complex systems, this could be risky, involving significant downtime and data loss or corruption if things go wrong. In theory, this approach should cost less than a trickle approach if all things go smoothly and risks are mitigated correctly.

Alternatively, you can move more slowly and take the trickle approach. This would involve moving data or processes in small, incremental stages, over a longer period of time. This approach can be less disruptive and less risky, as it provides more time to test and validate the new system before switching over. The main downside to a trickle approach is the challenge of maintaining two systems for an extended period of time to ensure proper transitions. This requires more people-power and will end up costing much more to maintain.

Internal Team versus External Support

The last consideration, but inherently also a part of each component, is people. Who will you be leaning on to bring your vision to life? Will it be the existing team members, or will you be looking to hire contract workers? You can help yourself arrive at the answer to this question by analyzing the trade-off between cost and time.

If you have time constraints, need to move quickly, and lack the required skills on the team, you’ll want to consider bringing in temporary contractors or consultants to help move things along. The trade-off here is the cost, as consultants are expensive. Alternatively, you can work to up-level your existing team through investment in learning and development training. This process will take much longer and will involve a different type of cost—the cost of potential mistakes that can be made in the process of learning. The people aspect of each project is nuanced, and we will discuss this in more detail in Chapter 5.

Deployment

The deployment phase will be where you start building and your vision will come to life as tangible output for your end users. This can be the most exciting part, since the impact of your work can start to be really felt at this stage. If you’re ready, there are a few things to keep in mind; let’s walk through them together.

Agile Development

The gold standard for project management used to be the waterfall method, which involved extensive effort to gather up-front requirements (aka specs), months of building in isolation, little to no modifications to those original specs, and the release to stakeholders only once the product was fully built. This meant that no feedback was provided until the product was fully completed, leaving little room for flexibility and iteration. At this point, stakeholders might have gotten smarter about what they wanted and often had extensive feedback on the product once it was delivered. Fortunately, just as technology is changing, methodologies and processes around it are changing as well. This is where Agile comes in.

This may not be the first time you hear about Agile methodology, but what is it exactly, and how will it help you? Well, Agile focuses on a flexible, iterative, and customer-centric approach. Before starting any project, you will go through a similar gathering of requirements from stakeholders, but the difference here is that you will use MoSCoW prioritization, which allows features and requests to get prioritized through a series of questioning that will help you land on the Must-haves, Should-haves, Could-haves, and Won’t-haves. Through this process, you can start to simplify the expected output into a version made of only the must-haves, also known as minimum viable product (MVP). Starting with an MVP will help shorten the time it takes to build, get feedback, and iterate. Adapting an Agile approach to product development can be critical to setting up your output for success.

Let’s say you work at a social media company and have been asked to build a benchmarking tool that helps identify industries that are accelerating their advertising spending. You have been asked to develop an output that joins multiple data sources together, both first and third party, and that provides an internal view as well as a full market view. To develop an MVP, you might suggest narrowing down to just first-party data and developing good internal benchmarks. This could satisfy the needs of stakeholders until you are able to layer in third-party data. Joining in third-party data would involve mapping the data sets, performing quality assurance to ensure accuracy, and validating impact based on the KPIs intended for the analysis. In most cases where third-party data is being used in a descriptive way, there is a clear path to add value, so this can be very well worth the time investment.

Plan Deployment

The first step of any good deployment plan is understanding all the downstream impacts resulting from the changes you are making. You will have done most of this during the scoping phase, when you reached out to the relevant teams, but it’s always good to maintain a strong communication flow with your end users, as requirements may change over time. Once this is done, be sure to share the timeline, milestones, contingency plans, and communication protocols with them so they know what to expect and when. Make sure all teams (IT, data engineering, business intelligence, data science, analysis, or other teams) are looped in and understand their role in the process. To that end, ensure the communication protocols and expectations are clear. Communicating early and often and using multiple formats (email, Slack, one-on-ones) will help mitigate issues down the line, so spend time ensuring you are sending clear, concise, and frequent updates to all relevant team members.

Once you’ve moved into deployment, you’ll want to verify that everything is working as intended by testing, monitoring, and evaluating changes in a controlled environment. Thus, deployment will typically have two stages that you’ll want to accommodate for: testing and production. Let’s cover what happens in both:

Testing

The testing phase is an important part of deployment, as it will help minimize the risk of errors or disruptions. Start with an MVP and take your time in this phase, ensuring your end users are involved. This will alert you early on as to whether you are satisfying their needs or not and allow them time to give feedback. During the testing phase, you’ll have time to evaluate and course-correct, which will be hard to do once you’ve productionalized and rolled out the changes. You will need to be ready for anything, such as changes in scope, incompatibility issues, data loss, etc., and you may realize that you need to pivot to accommodate unforeseen issues. Make adjustments as necessary, and continue to gather feedback from team members to identify areas for improvement. One of my (Sarah’s) favorite quotes about following plans is from the fictional character Leonard Snart from the TV show Arrow:

“Doesn’t matter. There are only four rules you need to remember: make the plan, execute the plan, expect the plan to go off the rails, throw away the plan. Follow my lead and you’ll be fine.”

Production

As you move into the release phase, where you deploy the changes to a production environment, you’ll want to ensure you update any related documentation and communicate the changes to end users. Take the lead to set up training, user guides, and other resources to help team members adapt to the changes.

To mitigate risk when moving through the development phase, you will want to lean on Agile risk management policies to help continually monitor and address issues. This involves a collaborative approach among the development team, project managers, and stakeholders to identify and prioritize risks and develop strategies to mitigate them.

Use Analysis Techniques to Monitor Deployment

After deployment is complete, you’ll want to closely monitor, and aim to improve, your data pipelines and products’ health, quality, and performance. This is known as data observability, and it’s not a new concept. Traditionally, it was limited to monitoring individual components of a fairly simple data stack, and data quality checks were often performed manually. From a reporting standpoint, this might look like this: end users such as subject matter experts seeing breaks in reporting (a day of missing data, an outlier, or a weird data point), alerting business intelligence teams on the issue, and suggesting a break in the data pipeline or QA flow. This leveraged a bottom-up approach to data observability that relied on SMEs to drive quality checks on reporting. However, modern data observability practices include automated data quality checks, anomaly detection, and real-time monitoring of the data pipelines that live within the engineering and BI teams. This has led to a general shift towards a more top-down, proactive, and automated approach to monitoring, observing, and actioning improvements to meet SLA requirements and keep data quality high. Let’s cover some of the important aspects of building strong data observability practices within your organization.

First, you’ll want to define the metrics to monitor (number of sales, daily revenue, number of clicks, etc.) and establish acceptable ranges for these metrics such that when data points lie outside of “normal” ranges, you are able to develop automated alerts and notifications to observe or fix them. This is most often referred to as outlier detection and is an important component of the data observability framework. Making this more automated helps engineering and BI teams notice and resolve issues before they land in, or break, reporting.

Second, you’ll want to monitor the health of data pipelines to ensure that data flows smoothly from its source to its destination. With many systems coming together to form data sets and reporting, there are many points of failure that can lead to delays or gaps in the data. This could be due to human error, hardware failures, software malfunctions, or numerous other reasons. For example, the data may be lost if an application crashes while writing data to a database. Thus, setting up mitigation plans to handle data latency and data loss issues will be critical for sustaining data integrity and meeting your agreed-on SLA requirements.

Third, you’ll want to regularly review the alerts, notifications, and reports generated by data observation practices to identify trends and insights. When running high-level analytics on the reporting you’ve built to monitor your data systems, you will be able to uncover things like: Are there any patterns for when data fails? On weekends? On Monday mornings when everyone is competing for resources? What do usage metrics look like, and how often are end users leveraging reporting built for them a month ago or a year ago? Can we deprecate reports that are no longer being leveraged? And the list goes on…

Finally, you’ll want to monitor the usage of any established reporting or data sets. This is the best way to know if what you’ve built is going to good use! Depending on the tools used, you may have some built-in analytics that will monitor usage.

Having strong data observability practices in place can help your team get ahead of data quality issues, establishing more transparency and allocating responsibility to them to ensure that reporting is accurate and reliable. Coming back to our theme of stakeholder trust, building strong data observability practices can help strengthen the trust between your team and your end users. Over time, this has only a positive ripple effect.³

Develop Through Use

Despite appearing as the last section of this chapter, developing through business use cases is where all good data products should start. Let’s be honest—not all fun data projects are impactful, and not all impactful business problems are fun, but there is a large middle ground where much great work can be done. So, grounding your work on what can move the needle for your business is where your work should start.

Create Key Reports/Dashboards

The data savviness of your organization will help guide what you tackle first, how you aim to build trust, and how you’ll show immediate value. Any newly established team, product, or output takes time to gain traction and achieve adoption. It is in this spirit that you should aim to develop products that have clear benefits. Take an audit of what currently exists, and look for areas to add value. If dashboards and key reports already exist, where are there areas of opportunity for either automation or optimization?

Automation

Automation refers to the use of technology to perform tasks or move through processes that would otherwise be handled manually by individuals on the team. You can easily identify where your team is spending a lot of time on repetitive processes that can use some form of automation. Investment in automated solutions for frequent, repetitive tasks will help free up their bandwidth to do more with their time and help improve the accuracy of their output. If stakeholders are relying on output from these processes, this also has downstream implications that improve on your SLA delivery.

Optimization

Optimization refers to the process of finding the best solution to a problem or task. It’s very common for teams to adopt methodologies that stay somewhat consistent over time, allowing for very little creativity or improvements to be made. Humans are hard to change, and established processes are hard to alter. However, the introduction of new and different approaches not only can be helpful but can also improve the accuracy and quality of product output. We encourage you to look and question the modes of operation to find areas of opportunity to optimize for better processes and solutions. Get others involved and hear people’s thoughts and ideas about the existing workflow. Are they itching to change it but just don’t have time or the skills to do so?

Building Trust

At the heart of any data product(s) you create is ensuring you have the trust of stakeholders and executives in the company. Trust takes a long time to build, and you may be set up for success (with past data teams leaving a strong positive impression), or you may need to work to rebuild that trust. Personally, there have been teams I’ve (Sarah) joined where trust was so low that it was incredibly hard to garner adoption on the data output I was creating. It took many months, my persistence despite the headwinds, and the “output speaking for itself” to finally see the tides turn around. Eventually, stakeholders were relying so heavily on the output of the models I had created that it had become a crucial and necessary part of the decision-making process. This is only to say, don’t give up if you feel push back initially! Understand that this is a long process that involves trust in people and trust in output. Thus, I would encourage you to start simple—take a small process and improve it slightly, and then move on to bigger and more complex projects. Also, involve your stakeholders early to give them a voice in the process and an idea of the changes they will be expecting. Seeing themselves as thought partners and contributors to the output can help build trust faster, and in turn, they’ll champion the results.

Quality versus Quantity

At the heart of any successful data product are high levels of adoption and trust. In the early days of Facebook, Mark Zuckerberg established the motto “move fast and break things.” Although this was a great way to experiment, build fast, and attract attention for his product, it was not a sustainable framework to keep for the long term. In 2022, at the time of the rebrand to Meta, and due to some negative press around the callousness implied by moving quickly without regard, they shifted to “move fast with stable infrastructure.” This example highlights the importance of moving quickly enough to prove value but also cautions against moving so quickly that many mistakes are made and trust is broken.⁴

Especially in the beginning, you’ll need to consider the trade-off between building for speed and building for quality. On one extreme, you can move very quickly and risk losing trust with stakeholders if output is wrong or misleading. On the other extreme, you can move so slowly, requiring lengthy timelines to get things completed and shipped, that you can leave stakeholders wondering when the promised output will be delivered. The key is to land somewhere in the middle of this spectrum, moving quickly enough to deliver value but slowly enough to validate and ensure accurate output. A great place to start is with any low-hanging fruit that can deliver high value and can be achieved with little effort. As you watch your team or stakeholders receive, consume, and process information, watch the flow and output. How robust are the analytics, dashboarding, and real-time analytics? Can you start integrating predictive modeling into the stack? Find creative ways to provide automation or optimization to their workflows in an impactful and straightforward manner.

When delivering data products, it’s important to remember that you are the owner of that product for the lifetime of its existence and use. The one exception here will be when there is an established process for handing over your work to engineering teams who will work to stabilize the output through productionalization. Once your product is out in the wild, your stakeholders may broaden; and in the absence of engineering support, you’ll want to be prepared to handle questions around the output and fix the backend processes if anything breaks. Thus, you will need to monitor your output to ensure stakeholders continue to get, and see, the right output. Often, depending on who has built the backend pipelines and if it was built on fragile infrastructure, you will need to monitor this closely.

Consider this an ongoing iterative loop. Similar to Brian McKnight’s lyrics, “if ever I believe my work is done, then I’ll start back at one,” you’ll want to continue to look for opportunities to improve on infrastructure, data enrichment, tools, and processes. You may find yourself involved in multiple negotiations for more resources or better tools, and this will all start from working through the needs of your stakeholders.

¹ StackCommerce, “63 Per Cent of Companies Consider Excel a Vital Accounting Tool,” Financial Post, April 29, 2021, https://oreil.ly/1a37R.

² Suzy Davenport, “How Long Does It Take to Break a Habit and What Is the Best Way to Do It?” Medical News Today, October 11, 2022, https://oreil.ly/5C4aC.

³ For more on data observability, see Fundamentals of Data Observability or What Is Data Observability?, both authored by Andy Petrella and published by O’Reilly.

⁴ Emily Stewart, “Mark Zuckerberg’s New Values for Meta Show He Still Hasn’t Truly Let Go of ‘Move Fast and Break Things,’” Business Insider, February 2, 2022, https://oreil.ly/8hQBc.