book

High Performance Python, 3rd Edition

by Micha Gorelick, Ian Ozsvald

April 2025

Intermediate to advanced

524 pages

14h 43m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Includes

Quizzes

Sandbox

Foreword
Preface
Who This Book Is ForWho This Book Is Not ForWhat You’ll LearnPython 3LicenseHow to Make an AttributionUsing Code ExamplesErrata and FeedbackConventions Used in This BookO’Reilly Online LearningHow to Contact UsAcknowledgments
1. Understanding Performant Python
The Fundamental Computer SystemComputing UnitsMemory UnitsCommunications LayersIdealized Computing Versus the Python Virtual MachineIdealized ComputingPython’s Virtual MachineSo Why Use Python?How to Be a Highly Performant ProgrammerGood Working PracticesOptimizing for the Team Rather than the Code BlockThe Remote Performant ProgrammerSome Thoughts on Good Notebook PracticeYour WorkThe Future of PythonWhere Did the GIL Go?Does Python Have a JIT?Wrap-Up
2. Profiling to Find Bottlenecks
Profiling EfficientlyIntroducing the Julia SetCalculating the Full Julia SetSimple Approaches to Timing—print and a DecoratorSimple Timing Using the Unix time CommandUsing the cProfile ModuleVisualizing cProfile Output with SnakeVizUsing line_profiler for Line-by-Line MeasurementsUsing memory_profiler to Diagnose Memory UsageCombining CPU and Memory Profiling with ScaleneIntrospecting an Existing Process with PySpyVizTracer for an Interactive Time-Based Call StackBytecode: Under the HoodUsing the dis Module to Examine CPython BytecodeDigging into Bytecode Specialization with SpecialistDifferent Approaches, Different ComplexityUnit Testing During Optimization to Maintain CorrectnessNo-op @profile DecoratorStrategies to Profile Your Code SuccessfullyWrap-Up
3. Lists and Tuples
A More Efficient SearchLists Versus TuplesLists as Dynamic ArraysTuples as Static ArraysWrap-Up
4. Dictionaries and Sets
How Do Dictionaries and Sets Work?Inserting and RetrievingDeletionResizingHash Functions and EntropyWrap-Up
5. Iterators and Generators
Iterators for Infinite SeriesLazy Generator EvaluationWrap-Up
6. Matrix and Vector Computation
Introduction to the ProblemAren’t Python Lists Good Enough?Problems with Allocating Too MuchMemory FragmentationUnderstanding perfMaking Decisions with perf’s OutputEnter numpyApplying numpy to the Diffusion ProblemMemory Allocations and In-Place OperationsSelective Optimizations: Finding What Needs to Be Fixednumexpr: Making In-Place Operations Faster and EasierGraphics Processing Units (GPUs)Dynamic Graphs: PyTorchGPU Speed and Numerical PrecisionGPU-Specific OperationsBasic GPU ProfilingPerformance Considerations of GPUsWhen to Use GPUsDeep Learning Performance ConsiderationsA Cautionary Tale: Verify “Optimizations” (scipy)Lessons from Matrix OptimizationsWrap-Up
7. Pandas, Dask, and Polars
PandasPandas’s Internal ModelArrow and NumPyApplying a Function to Many Rows of DataNumba to Compile NumPy for PandasBuilding from Partial Results Rather than ConcatenatingThere’s More Than One (and Possibly a Faster) Way to Do a JobAdvice for Effective Pandas DevelopmentDask for Distributed Data Structures and DataFramesDiagnosticsParallel Pandas with DaskParallelized apply with Swifter on DaskPolars for Fast DataFramesWrap-Up
8. Compiling to C
What Sort of Speed Gains Are Possible?JIT Versus AOT CompilersWhy Does Type Information Help the Code Run Faster?Using a C CompilerReviewing the Julia Set ExampleCythonCompiling a Pure Python Version Using CythonpyximportCython Annotations to Analyze a Block of CodeAdding Some Type AnnotationsCython and numpyParallelizing the Solution with OpenMP on One MachineNumbaPyPyGarbage Collection DifferencesRunning PyPy and Installing ModulesA Summary of Speed ImprovementsWhen to Use Each TechnologyForeign Function Interfacesctypescffif2pyCPython Extensions: CCPython Extensions: RustWrap-Up

9. Asynchronous I/O
Introduction to Asynchronous ProgrammingHow Does async/await Work?Serial Web CrawlerAsynchronous Web CrawlerShared CPU–I/O WorkloadSerial CPU WorkloadBatched CPU WorkloadFully Asynchronous CPU WorkloadWrap-Up
10. The multiprocessing Module
An Overview of the multiprocessing ModuleEstimating Pi Using the Monte Carlo MethodEstimating Pi Using Processes and ThreadsUsing Python ObjectsReplacing multiprocessing with JoblibRandom Numbers in Parallel SystemsUsing numpyFinding Prime NumbersQueues of WorkAsynchronously Adding Jobs to the QueueVerifying Primes Using Interprocess CommunicationSerial SolutionNaive Pool SolutionA Less Naive Pool SolutionUsing manager.Value as a FlagUsing Redis as a FlagUsing RawValue as a FlagUsing mmap as a FlagUsing mmap as a Flag ReduxSharing numpy Data with multiprocessingSynchronizing File and Variable AccessFile LockingLocking a ValueWrap-Up
11. Clusters and Job Queues
Benefits of ClusteringDrawbacks of Clustering$462 Million Wall Street Loss Through Poor Cluster Upgrade StrategySkype’s 24-Hour Global OutageCommon Cluster DesignsHow to Start a Clustered SolutionWays to Avoid Pain When Using ClustersTwo Clustering SolutionsUsing IPython Parallel to Support ResearchMessage Brokering for Cluster EfficiencyOther Clustering Tools to Look AtDockerDocker’s PerformanceAdvantages of DockerWrap-Up
12. Using Less RAM
Objects for Primitives Are ExpensiveThe array Module Stores Many Primitive Objects CheaplyUsing Less RAM in NumPy with NumExprUnderstanding the RAM Used in a CollectionBytes Versus UnicodeEfficiently Storing Lots of Text in RAMTrying These Approaches on 11 Million TokensModeling More Text with scikit-learn’s FeatureHasherIntroducing DictVectorizer and FeatureHasherComparing DictVectorizer and FeatureHasher on a Real ProblemSciPy’s Sparse MatricesTips for Using Less RAMProbabilistic Data StructuresVery Approximate Counting with a 1-Byte Morris CounterK-Minimum ValuesBloom FiltersLogLog CounterReal-World ExampleWrap-Up
13. Lessons from the Field
Developing a High Performance Machine Learning AlgorithmHigh Performance Computing in JournalismLessons from the Field of Cyber ReinsurancePython in Quant FinanceMaintain Flexibility to Achieve High PerformanceStreamlining Feature Engineering Pipelines with Feature-engine (2020)Highly Performant Data Science Teams (2020)Numba (2020)Optimizing Versus Thinking (2020)Making Deep Learning Fly with RadimRehurek.com (2014)Large-Scale Social Media Analysis at Smesh (2014)
Index
About the Authors

Content preview from High Performance Python, 3rd Edition

Chapter 13. Lessons from the Field

In this chapter we have collected stories from successful companies that use Python in high-data-volume and speed-critical situations. The stories are written by key people in each organization who have many years of experience; they share not just their technology choices but also some of their hard-won wisdom. We have five great new stories for you from other experts in our domain. We’ve also kept some of the “Lessons from the Field” from the first and second editions of this book; their titles are marked “(2014)” and “(2020)” respectively.

Developing a High Performance Machine Learning Algorithm

Martin Goodson linkedin.com/in/martingoodson

We had overpromised to our customers and our machine learning models were making a stream of errors ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781098165956Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

High Performance Python, 3rd Edition

by Micha Gorelick, Ian Ozsvald

Chapter 13. Lessons from the Field

Developing a High Performance Machine Learning Algorithm

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.