book

Ten Things to Know About ModelOps

by Thomas Hill, Mark Palmer, Larry Derany

June 2022

Beginner to intermediate

41 pages

59m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Overview of ModelOps Data-Driven OrganizationsExtracting Information from Data via ModelsThe Model Building/Authoring Life CycleWhat Are ModelOps Tools?
1. ModelOps Is the Enterprise “Operating System” for AI/ML AlgorithmsMany Types of ModelsMany Different People/Roles Build Models2. There Are Many Algorithms and Many Model Formats: Deliver Them AllModel FormatsSupport Your Model Builders, Don’t Constrain Them3. First, Do No Harm: Anticipate Risks Rather Than React to Damage DoneOffending CustomersRisky Predictions for ManufacturingConsidering Wrong PredictionsAnticipating Risk4. Aim to Build Your Organization’s IP Secret Sauce Through Repeatable Processing StepsManaging and Accumulating IP Through ModelsDisorganized Analytics Across Silos5. ModelOps Is About Maintaining and Expanding Your IPModel Decay (Drift): Changing Populations, Data, and RelationshipsFlexible Monitoring of End-to-End Scoring Pipelines and Agile DeploymentThe Bane of AI/ML at Scale: Big CodeGoverned Inputs/Outputs; Small Reusable Steps; and Efficient, Targeted Monitoring That Enables Automation6. The World Is Changing Fast: Think Agile Model Deployment and Streaming DataBatch and Real-Time Data—Use CasesResponding to a Dynamically Changing World7. Plan for Success and Monitor for Value (Not Just Accuracy)Every Project Starts at the End—or It Is Likely to Fail!Planning for How Model Scoring Is to Be Consumed—and Valuable8. The Future of AI Will Be Regulated—ModelOps Helps Ensure ComplianceRequirements for Analytics in Medical Device and Pharma Manufacturing Offer Guidance for What’s to ComeModelOps Capabilities, Beyond Efficient Deployment and Scoring9. Expect Quality and Process Best Practices from the ModelOps Operating System10. ModelOps Is a Required Tool for Managing AI/ML Model Deployment: Start Small, but Start Now

Overview

The past few years have seen significant developments in data science, AI, machine learning, and advanced analytics. But the wider adoption of these technologies has also brought greater cost, risk, regulation, and demands on organizational processes, tasks, and teams. This report explains how ModelOps can provide both technical and operational solutions to these problems.

Thomas Hill, Mark Palmer, and Larry Derany summarize important considerations, caveats, choices, and best practices to help you be successful with operationalizing AI/ML and analytics in general. Whether your organization is already working with teams on AI and ML, or just getting started, this report presents ten important dimensions of analytic practice and ModelOps that are not widely discussed, or perhaps even known.

In part, this report examines:

Why ModelOps is the enterprise "operating system" for AI/ML algorithms
How to build your organization's IP secret sauce through repeatable processing steps
How to anticipate risks rather than react to damage done
How ModelOps can help you deliver the many algorithms and model formats available
How to plan for success and monitor for value, not just accuracy
Why AI will be soon be regulated and how ModelOps helps ensure compliance