book

High Performance Python

by Micha Gorelick, Ian Ozsvald

September 2014

Intermediate to advanced

368 pages

10h 14m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Preface
Who This Book Is ForWho This Book Is Not ForWhat You’ll LearnPython 2.7Moving to Python 3LicenseHow to Make an AttributionErrata and FeedbackConventions Used in This BookUsing Code ExamplesSafari® Books OnlineHow to Contact UsAcknowledgments
1. Understanding Performant Python
The Fundamental Computer SystemComputing UnitsMemory UnitsCommunications LayersPutting the Fundamental Elements TogetherIdealized Computing Versus the Python Virtual MachineIdealized computingPython’s virtual machineSo Why Use Python?
2. Profiling to Find Bottlenecks
Profiling EfficientlyIntroducing the Julia SetCalculating the Full Julia SetSimple Approaches to Timing—print and a DecoratorSimple Timing Using the Unix time CommandUsing the cProfile ModuleUsing runsnakerun to Visualize cProfile OutputUsing line_profiler for Line-by-Line MeasurementsUsing memory_profiler to Diagnose Memory UsageInspecting Objects on the Heap with heapyUsing dowser for Live Graphing of Instantiated VariablesUsing the dis Module to Examine CPython BytecodeDifferent Approaches, Different ComplexityUnit Testing During Optimization to Maintain CorrectnessNo-op @profile DecoratorStrategies to Profile Your Code SuccessfullyWrap-Up
3. Lists and Tuples
A More Efficient SearchLists Versus TuplesLists as Dynamic ArraysTuples As Static ArraysWrap-Up
4. Dictionaries and Sets
How Do Dictionaries and Sets Work?Inserting and RetrievingDeletionResizingHash Functions and EntropyDictionaries and NamespacesWrap-Up
5. Iterators and Generators
Iterators for Infinite SeriesLazy Generator EvaluationWrap-Up
6. Matrix and Vector Computation
Introduction to the ProblemAren’t Python Lists Good Enough?Problems with Allocating Too MuchMemory FragmentationUnderstanding perfMaking Decisions with perf’s OutputEnter numpyApplying numpy to the Diffusion ProblemMemory Allocations and In-Place OperationsSelective Optimizations: Finding What Needs to Be Fixednumexpr: Making In-Place Operations Faster and EasierA Cautionary Tale: Verify “Optimizations” (scipy)Wrap-Up
7. Compiling to C
What Sort of Speed Gains Are Possible?JIT Versus AOT CompilersWhy Does Type Information Help the Code Run Faster?Using a C CompilerReviewing the Julia Set ExampleCythonCompiling a Pure-Python Version Using CythonCython Annotations to Analyze a Block of CodeAdding Some Type AnnotationsShed SkinBuilding an Extension ModuleThe Cost of the Memory CopiesCython and numpyParallelizing the Solution with OpenMP on One MachineNumbaPythranPyPyGarbage Collection DifferencesRunning PyPy and Installing ModulesWhen to Use Each TechnologyOther Upcoming ProjectsA Note on Graphics Processing Units (GPUs)A Wish for a Future Compiler ProjectForeign Function Interfacesctypescffif2pyCPython ModuleWrap-Up
8. Concurrency
Introduction to Asynchronous ProgrammingSerial CrawlergeventtornadoAsyncIODatabase ExampleWrap-Up
9. The multiprocessing Module
An Overview of the Multiprocessing ModuleEstimating Pi Using the Monte Carlo MethodEstimating Pi Using Processes and ThreadsUsing Python ObjectsRandom Numbers in Parallel SystemsUsing numpyFinding Prime NumbersQueues of WorkAsynchronously adding jobs to the QueueVerifying Primes Using Interprocess CommunicationSerial SolutionNaive Pool SolutionA Less Naive Pool SolutionUsing Manager.Value as a FlagUsing Redis as a FlagUsing RawValue as a FlagUsing mmap as a FlagUsing mmap as a Flag ReduxSharing numpy Data with multiprocessingSynchronizing File and Variable AccessFile LockingLocking a ValueWrap-Up

10. Clusters and Job Queues
Benefits of ClusteringDrawbacks of Clustering$462 Million Wall Street Loss Through Poor Cluster Upgrade StrategySkype’s 24-Hour Global OutageCommon Cluster DesignsHow to Start a Clustered SolutionWays to Avoid Pain When Using ClustersThree Clustering SolutionsUsing the Parallel Python Module for Simple Local ClustersUsing IPython Parallel to Support ResearchNSQ for Robust Production ClusteringQueuesPub/subDistributed Prime CalculationOther Clustering Tools to Look AtWrap-Up
11. Using Less RAM
Objects for Primitives Are ExpensiveThe Array Module Stores Many Primitive Objects CheaplyUnderstanding the RAM Used in a CollectionBytes Versus UnicodeEfficiently Storing Lots of Text in RAMTrying These Approaches on 8 Million TokenslistsetMore efficient tree structuresDirected acyclic word graph (DAWG)Marisa trieDatrieHAT trieUsing tries (and DAWGs) in production systemsTips for Using Less RAMProbabilistic Data StructuresVery Approximate Counting with a 1-byte Morris CounterK-Minimum ValuesBloom FiltersLogLog CounterReal-World Example
12. Lessons from the Field
Adaptive Lab’s Social Media Analytics (SoMA)Python at Adaptive LabSoMA’s DesignOur Development MethodologyMaintaining SoMAAdvice for Fellow EngineersMaking Deep Learning Fly with RadimRehurek.comThe Sweet SpotLessons in OptimizingWrap-UpLarge-Scale Productionized Machine Learning at Lyst.comPython’s Place at LystCluster DesignCode Evolution in a Fast-Moving Start-UpBuilding the Recommendation EngineReporting and MonitoringSome AdviceLarge-Scale Social Media Analysis at SmeshPython’s Role at SmeshThe PlatformHigh Performance Real-Time String MatchingReporting, Monitoring, Debugging, and DeploymentPyPy for Successful Web and Data Processing SystemsPrerequisitesThe DatabaseThe Web ApplicationOCR and TranslationTask Distribution and WorkersConclusionTask Queues at Lanyrd.comPython’s Role at LanyrdMaking the Task Queue PerformantReporting, Monitoring, Debugging, and DeploymentAdvice to a Fellow Developer
Index
Colophon
Copyright

Content preview from High Performance Python

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781449361747Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

High Performance Python

by Micha Gorelick, Ian Ozsvald

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.