Skip to Content
View all events

Combating Data Drift and Model Decay

Published by O'Reilly Media, Inc.

Intermediate content levelIntermediate

Monitor ML models in production

What you’ll learn and how you can apply it

  • Identify the types of drift that affect AI systems
  • Implement effective monitoring frameworks to detect and diagnose AI system issues in production
  • Apply proactive strategies for mitigating and managing drift, including retraining, alerting, and model governance
  • Use metrics, dashboards, and observability tools to maintain AI reliability and business impact
  • Translate technical monitoring insights into actionable recommendations for cross-functional teams

Course description

AI systems don’t fail all at once; they degrade silently. As data shifts, user behavior evolves, and real-world conditions change, models that once performed well can quickly become unreliable in production. In this hands-on course, software engineer Chisom Nwokwu shows you how to guard against system breakdown.

You’ll learn how to monitor AI systems effectively, define the right metrics, and detect drift before it impacts users or business outcomes. You’ll explore real-world failure cases, understand the differences between model and business performance, and gain practical experience diagnosing issues using statistical techniques and observability frameworks. You’ll also learn how to respond to drift, design retraining strategies, implement alerting systems, and build end-to-end monitoring pipelines that scale across teams and use cases.

You’ll come away with a clear mental model, a practical monitoring framework, and a production-ready checklist to ensure your AI systems remain reliable, trustworthy, and aligned with real-world conditions.

This live event is for you because...

  • You’re a machine learning engineer or data scientist who builds or deploys models and wants to confidently monitor, maintain, and improve performance in production.
  • You’re an SRE or DevOps engineer who supports AI-powered systems and is looking to apply reliability and observability practices to machine learning.
  • You’re a technical lead or engineering manager who oversees AI-driven products and needs to understand model performance, manage risk, and make informed decisions.

Prerequisites

  • A foundational understanding of machine learning concepts (e.g., models, training, evaluation)
  • Basic familiarity with Python for data analysis

Recommended preparation:

  • Refresh key machine learning concepts, including supervised learning and evaluation metrics
  • Review a short article or case study on AI system failure in production to build context for real-world challenges

Recommended follow-up:

Schedule

The time frames are only estimates and may vary according to how the class is progressing.

Introduction to AI systems, and why monitoring matters (30 minutes)

  • Presentation: Fundamentals of AI systems; deterministic versus probabilistic systems, and why they differ in reliability; why AI systems fail in production; introduction to drift; monitoring challenges with LLMs; case study of AI recommendation system failure due to changing user behavior
  • Group discussion: What’s the biggest risk if your AI system silently degrades in production?
  • Q&A

Metrics, observability, and what to monitor (25 minutes)

  • Presentation: What does good performance mean in production?; model metrics versus business metrics; fundamentals of observability; data quality monitoring; tracking model performance over time; monitoring inputs versus outputs versus outcomes
  • Demonstration: What to look for on a simple dashboard showing model performance and data distribution over time
  • Group discussion: Which metrics would matter most for your current or imagined AI use case?
  • Q&A
  • Break

Detecting and diagnosing drift (20 minutes)

  • Presentation: Types of drift; when drift is actually a problem; statistical techniques (PSI, distribution comparison, thresholding); tracing drift back to pipelines using root cause analysis
  • Demonstration: Detecting drift in a dataset using Python
  • Group discussion: What would you do if you detected drift but couldn’t identify the cause?
  • Q&A

Mitigation strategies and incident response (20 minutes)

  • Presentation: What to do after detecting drift (a decision framework); scheduled versus triggered retraining strategies and when each is appropriate; alerting and escalation—what deserves a page, what deserves a log, and what can wait; human-in-the-loop systems and fallback strategies; governance basics (model versioning, audit logs, and who owns the decision); case study of a fraud detection model adapting to new attack patterns and how the team detected, diagnosed, and responded
  • Group discussion: Should every drift event trigger retraining?; What would your decision criteria be?
  • Q&A
  • Break

Building an end-to-end monitoring pipeline (20 minutes)

  • Presentation: Architecture of a production-ready monitoring system; key components and how they connect; integrating data pipelines, models, and monitoring tools; the tooling landscape and how to evaluate build versus buy for your team; designing for scale; what to consider when monitoring involves multiple models and teams
  • Hands-on exercise: Using a guided framework, sketch a monitoring architecture for an AI use case of your choice, covering data inputs, model outputs, alert thresholds, and escalation paths
  • Q&A

Checklist and actionable takeaways (10 minutes)

  • Presentation: AI monitoring checklist—what every production system must have before go-live; the three most common mistakes teams make and how to avoid them; a concrete starting plan regardless of your current stack
  • Q&A

Your Instructor

  • Chisom Nwokwu

    Chisom Nwokwu is a Software Engineer working at the intersection of product, data, and applied AI. Passionate about Big Data and Artificial Intelligence, she leverages big data and cloud technologies to design and develop scalable data platforms for organizations, with experience at Microsoft and Bank of America. She is currently contributing to the development of next-generation systems powering large-scale AI model training.

    Chisom is also a multi-published author, most recently of Data Engineering for Beginners, a complete roadmap for aspiring data engineers published by Wiley Global. In addition to her professional achievements and multiple awards, she is an accomplished public speaker and digital creator, playing a pivotal role in the global tech scene.

    linkedinXlinksearch

Skills covered

  • Site Reliability Engineering (SRE)
  • Security Engineering
  • Artificial Intelligence (AI)
  • Machine Learning