Skip to Content
View all events

Applying Data Centric AI Principles To Production Systems

Published by O'Reilly Media, Inc.

Intermediate to advanced content levelIntermediate to advanced

From experimental pipelines to production ML systems

This live event utilizes Jupyter Notebook technology

In this course, you’ll:

  • Understand statistical properties and distributions of data
  • Learn data-centric AI (DCAI) principles for building and operating ML models
  • Identify how to operationalize DCAI in various data environments and stages in the machine learning pipeline

Data-centric AI is a growing trend in the ML and MLOps community for good reason: issues in your dataset are a common cause for AI system failures and poor performance. But despite its relevance to production data systems, DCAI is most often discussed as a paradigm shift from focusing on model architectures and optimization to dataset iteration and quality in the experimental and model building stages of machine learning. As a result, there is less information for data scientists and ML engineers looking to translate DCAI principles to working solutions for production systems at massive scale, streaming data, and real-world data drift. Visibility into ML data in these settings requires approaches that are purpose-built for datasets and their distributions and dependencies.

Join experts Bernease Herman and Alessya Visnjic to learn the basics of data-centric machine learning practice: detecting and debugging input data and label data quality issues, improving ML model performance via dataset and data slice analysis, and identifying data problems postdeployment. You'll use open source data science libraries in Python to debug an example ML model and system, tying these practices to the underlying DCAI principles discussed by the broader community.

What you’ll learn and how you can apply it

  • Detect input data and label data quality issues in experimentation or production
  • Improve ML model performance via dataset and data slice analysis
  • Automatically identify data problems postdeployment in production systems at scale

This live event is for you because...

  • You’re an ML engineer or data scientist.
  • You want to build robust and responsible ML applications.
  • You work with ML applications that are running in production.

Prerequisites

  • Familiarity with Python programming language
  • Basic understanding of machine learning, statistics, and machine learning models in production

Recommended preparation:

Recommended follow-up:

Schedule

The time frames are only estimates and may vary according to how the class is progressing.

Introduction (20 minutes)

  • Presentation: Course overview and learning objectives
  • Hands-on exercise: Setup and configuration
  • Group discussion: Your experience with experimental ML (building models, toy datasets, Kaggle) and production ML systems (at scale, live systems, MLOps)
  • Q&A

Overview of ML datasets and their properties (25 minutes)

  • Presentation: Common statistical distributions and properties; generating a synthetic dataset using statistical information; the anatomy of an ML dataset and what transformations it undergoes through the model lifecycle
  • Jupyter notebook: Statistical properties and distributions of data
  • Q&A
  • Break

Determining dataset quality (50 minutes)

  • Presentation: Exploratory data analysis techniques; dataset and label sources (input data generating processes, dataset collection); causes of dynamic data quality issues (errors, dataset drift, label data quality issues, and target leakage)
  • Jupyter notebook: EDA and identifying common data quality issues on an example dataset
  • Q&A
  • Break

Principles of data-centric AI (50 minutes)

  • Group discussion: What is data-centric AI (DCAI)? Does that differ from what data scientists and ML engineers have always done?
  • Presentation and discussions: Discourse about DCAI in the ML community; DCAI principles and practices for experimental ML; DCAI principles and practices for production ML; DCAI principles and practices for enterprise scale systems
  • Q&A
  • Break

Improving ML models via dataset analysis at scale and streaming (50 minutes)

  • Presentation: Basics of sampling and data profiling; advanced approaches to sampling and data profiling; performance metrics and observability
  • Jupyter notebook: Data structures and algorithms for capturing data at scale and streaming use cases
  • Group discussion: Data retention and performance for massive scale data
  • Q&A
  • Break

Extending data-centric practices to nonstructured data (30 minutes)

  • Presentation: Image and text dataset quality, statistical approach
  • Hands-on exercise: Debug a data issue in a CV model
  • Group discussion: Unstructured data practices at massive scale
  • Q&A

Wrap-up (15 minutes)

  • Presentation: Recap; more directions for data-centric AI
  • Q&A

Your Instructors

Alessya Visnjic

Alessya Visnjic is the CEO of WhyLabs, the AI Observability company building tools that power robust and responsible AI deployment. Prior to WhyLabs, Alessya was a CTO-in-residence at the Allen Institute for AI, where she evaluated commercial potential for the latest AI research. Earlier, Alessya spent 9 years at Amazon leading ML initiatives, including forecasting and data science platforms. Alessya is also the founder of Rsqrd AI, a global community of 1,000+ AI practitioners who are committed to making enterprise AI technology responsible.

linkedinXlinksearch

Bernease Herman

Bernease Herman is a data scientist at WhyLabs, the AI observability company, and a research scientist at the University of Washington eScience Institute. At WhyLabs, she’s building model and data monitoring solutions using approximate statistics techniques. Earlier in her career, Bernease built ML-driven solutions for inventory planning at Amazon and conducted quantitative research at Morgan Stanley. Bernease also serves on the faculty of the University of Washington master’s program in data science and as chair of the Rigorous Evaluation for AI Systems (REAIS) workshop series. She has published work in top machine learning conferences and workshops such as NeurIPS, ICLR, and FAccT. She’s a PhD student at the University of Washington, where her research focuses on evaluation metrics and interpretable ML with specialty in synthetic data and societal implications, and she holds a bachelor’s degree in mathematics and statistics from the University of Michigan.

linkedinXlinksearch

Skill covered

Machine Learning