book

High Performance Python, 2nd Edition

by Micha Gorelick, Ian Ozsvald

April 2020

Intermediate to advanced

468 pages

12h 52m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Foreword
Preface
Who This Book Is ForWho This Book Is Not ForWhat You’ll LearnPython 3Changes from Python 2.7LicenseHow to Make an AttributionErrata and FeedbackConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
1. Understanding Performant Python
The Fundamental Computer SystemComputing UnitsMemory UnitsCommunications LayersPutting the Fundamental Elements TogetherIdealized Computing Versus the Python Virtual MachineSo Why Use Python?How to Be a Highly Performant ProgrammerGood Working PracticesSome Thoughts on Good Notebook PracticeGetting the Joy Back into Your Work
2. Profiling to Find Bottlenecks
Profiling EfficientlyIntroducing the Julia SetCalculating the Full Julia SetSimple Approaches to Timing—print and a DecoratorSimple Timing Using the Unix time CommandUsing the cProfile ModuleVisualizing cProfile Output with SnakeVizUsing line_profiler for Line-by-Line MeasurementsUsing memory_profiler to Diagnose Memory UsageIntrospecting an Existing Process with PySpyBytecode: Under the HoodUsing the dis Module to Examine CPython BytecodeDifferent Approaches, Different ComplexityUnit Testing During Optimization to Maintain CorrectnessNo-op @profile DecoratorStrategies to Profile Your Code SuccessfullyWrap-Up
3. Lists and Tuples
A More Efficient SearchLists Versus TuplesLists as Dynamic ArraysTuples as Static ArraysWrap-Up
4. Dictionaries and Sets
How Do Dictionaries and Sets Work?Inserting and RetrievingDeletionResizingHash Functions and EntropyDictionaries and NamespacesWrap-Up
5. Iterators and Generators
Iterators for Infinite SeriesLazy Generator EvaluationWrap-Up
6. Matrix and Vector Computation
Introduction to the ProblemAren’t Python Lists Good Enough?Problems with Allocating Too MuchMemory FragmentationUnderstanding perfMaking Decisions with perf’s OutputEnter numpyApplying numpy to the Diffusion ProblemMemory Allocations and In-Place OperationsSelective Optimizations: Finding What Needs to Be Fixednumexpr: Making In-Place Operations Faster and EasierA Cautionary Tale: Verify “Optimizations” (scipy)Lessons from Matrix OptimizationsPandasPandas’s Internal ModelApplying a Function to Many Rows of DataBuilding DataFrames and Series from Partial Results Rather than ConcatenatingThere’s More Than One (and Possibly a Faster) Way to Do a JobAdvice for Effective Pandas DevelopmentWrap-Up
7. Compiling to C
What Sort of Speed Gains Are Possible?JIT Versus AOT CompilersWhy Does Type Information Help the Code Run Faster?Using a C CompilerReviewing the Julia Set ExampleCythonCompiling a Pure Python Version Using CythonpyximportCython Annotations to Analyze a Block of CodeAdding Some Type AnnotationsCython and numpyParallelizing the Solution with OpenMP on One MachineNumbaNumba to Compile NumPy for PandasPyPyGarbage Collection DifferencesRunning PyPy and Installing ModulesA Summary of Speed ImprovementsWhen to Use Each TechnologyOther Upcoming ProjectsGraphics Processing Units (GPUs)Dynamic Graphs: PyTorchBasic GPU ProfilingPerformance Considerations of GPUsWhen to Use GPUsForeign Function Interfacesctypescffif2pyCPython ModuleWrap-Up
8. Asynchronous I/O
Introduction to Asynchronous ProgrammingHow Does async/await Work?Serial CrawlerGeventtornadoaiohttpShared CPU–I/O WorkloadSerialBatched ResultsFull AsyncWrap-Up

9. The multiprocessing Module
An Overview of the multiprocessing ModuleEstimating Pi Using the Monte Carlo MethodEstimating Pi Using Processes and ThreadsUsing Python ObjectsReplacing multiprocessing with JoblibRandom Numbers in Parallel SystemsUsing numpyFinding Prime NumbersQueues of WorkVerifying Primes Using Interprocess CommunicationSerial SolutionNaive Pool SolutionA Less Naive Pool SolutionUsing Manager.Value as a FlagUsing Redis as a FlagUsing RawValue as a FlagUsing mmap as a FlagUsing mmap as a Flag ReduxSharing numpy Data with multiprocessingSynchronizing File and Variable AccessFile LockingLocking a ValueWrap-Up
10. Clusters and Job Queues
Benefits of ClusteringDrawbacks of Clustering$462 Million Wall Street Loss Through Poor Cluster Upgrade StrategySkype’s 24-Hour Global OutageCommon Cluster DesignsHow to Start a Clustered SolutionWays to Avoid Pain When Using ClustersTwo Clustering SolutionsUsing IPython Parallel to Support ResearchParallel Pandas with DaskNSQ for Robust Production ClusteringQueuesPub/subDistributed Prime CalculationOther Clustering Tools to Look AtDockerDocker’s PerformanceAdvantages of DockerWrap-Up
11. Using Less RAM
Objects for Primitives Are ExpensiveThe array Module Stores Many Primitive Objects CheaplyUsing Less RAM in NumPy with NumExprUnderstanding the RAM Used in a CollectionBytes Versus UnicodeEfficiently Storing Lots of Text in RAMTrying These Approaches on 11 Million TokensModeling More Text with Scikit-Learn’s FeatureHasherIntroducing DictVectorizer and FeatureHasherComparing DictVectorizer and FeatureHasher on a Real ProblemSciPy’s Sparse MatricesTips for Using Less RAMProbabilistic Data StructuresVery Approximate Counting with a 1-Byte Morris CounterK-Minimum ValuesBloom FiltersLogLog CounterReal-World Example
12. Lessons from the Field
Streamlining Feature Engineering Pipelines with Feature-engineFeature Engineering for Machine LearningThe Hard Task of Deploying Feature Engineering PipelinesLeveraging the Power of Open Source Python LibrariesFeature-engine Smooths Building and Deployment of Feature Engineering PipelinesHelping with the Adoption of a New Open Source PackageDeveloping, Maintaining, and Encouraging Contribution to Open Source LibrariesHighly Performant Data Science TeamsHow Long Will It Take?Discovery and PlanningManaging Expectations and DeliveryNumbaA Simple ExampleBest Practices and RecommendationsGetting HelpOptimizing Versus ThinkingAdaptive Lab’s Social Media Analytics (2014)Python at Adaptive LabSoMA’s DesignOur Development MethodologyMaintaining SoMAAdvice for Fellow EngineersMaking Deep Learning Fly with RadimRehurek.com (2014)The Sweet SpotLessons in OptimizingConclusionLarge-Scale Productionized Machine Learning at Lyst.com (2014)Cluster DesignCode Evolution in a Fast-Moving Start-UpBuilding the Recommendation EngineReporting and MonitoringSome AdviceLarge-Scale Social Media Analysis at Smesh (2014)Python’s Role at SmeshThe PlatformHigh Performance Real-Time String MatchingReporting, Monitoring, Debugging, and DeploymentPyPy for Successful Web and Data Processing Systems (2014)PrerequisitesThe DatabaseThe Web ApplicationOCR and TranslationTask Distribution and WorkersConclusionTask Queues at Lanyrd.com (2014)Python’s Role at LanyrdMaking the Task Queue PerformantReporting, Monitoring, Debugging, and DeploymentAdvice to a Fellow Developer
Index
About the Authors

Content preview from High Performance Python, 2nd Edition

Chapter 12. Lessons from the Field

In this chapter we have collected stories from successful companies that use Python in high-data-volume and speed-critical situations. The stories are written by key people in each organization who have many years of experience; they share not just their technology choices but also some of their hard-won wisdom. We have four great new stories for you from other experts in our domain. We’ve also kept the “Lessons from the Field” from the first edition of this book; their titles are marked “(2014).”

Streamlining Feature Engineering Pipelines with Feature-engine

Soledad Galli (trainindata.com)

Train in Data is an education project led by experienced data scientists and AI software engineers. We help professionals improve coding and data science skills and adopt machine learning best practices. We ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781492055013Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

High Performance Python, 2nd Edition

by Micha Gorelick, Ian Ozsvald

Chapter 12. Lessons from the Field

Streamlining Feature Engineering Pipelines with Feature-engine

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.