book

Fast Python

Name: Fast Python
Author: Tiago Antao
ISBN: 9781617297939

by Tiago Antao

June 2023

Intermediate to advanced

304 pages

8h 53m

English

Manning Publications

Audiobook available

Read now

Unlock full access

inside front cover
Fast Python
Copyright
contents
front matter
prefaceacknowledgmentsabout this bookWho should read this book?How this book is organized: A road mapAbout the codeliveBook discussion forumHardware and softwareabout the authorabout the cover illustration
Part 1. Foundational Approaches
1 An urgent need for efficiency in data processing
1.1 How bad is the data deluge?1.2 Modern computing architectures and high-performance computing1.2.1 Changes inside the computer1.2.2 Changes in the network1.2.3 The cloud1.3 Working with Python’s limitations1.3.1 The Global Interpreter Lock1.4 A summary of the solutionsSummary
2 Extracting maximum performance from built-in features
2.1 Profiling applications with both IO and computing workloads2.1.1 Downloading data and computing minimum temperatures2.1.2 Python’s built-in profiling module2.1.3 Using local caches to reduce network usage2.2 Profiling code to detect performance bottlenecks2.2.1 Visualizing profiling information2.2.2 Line profiling2.2.3 The takeaway: Profiling code2.3 Optimizing basic data structures for speed: Lists, sets, and dictionaries2.3.1 Performance of list searches2.3.2 Searching using sets2.3.3 List, set, and dictionary complexity in Python2.4 Finding excessive memory allocation2.4.1 Navigating the minefield of Python memory estimation2.4.2 The memory footprint of some alternative representations2.4.3 Using arrays as a compact representation alternative to lists2.4.4 Systematizing what we have learned: Estimating memory usage of Python objects2.4.5 The takeaway: Estimating memory usage of Python objects2.5 Using laziness and generators for big-data pipelining2.5.1 Using generators instead of standard functionsSummary
3 Concurrency, parallelism, and asynchronous processing
3.1 Writing the scaffold of an asynchronous server3.1.1 Implementing the scaffold for communicating with clients3.1.2 Programming with coroutines3.1.3 Sending complex data from a simple synchronous client3.1.4 Alternative approaches to interprocess communication3.1.5 The takeaway: Asynchronous programming3.2 Implementing a basic MapReduce engine3.2.1 Understanding MapReduce frameworks3.2.2 Developing a very simple test scenario3.2.3 A first attempt at implementing a MapReduce framework3.3 Implementing a concurrent version of a MapReduce engine3.3.1 Using concurrent.futures to implement a threaded server3.3.2 Asynchronous execution with futures3.3.3 The GIL and multithreading3.4 Using multiprocessing to implement MapReduce3.4.1 A solution based on concurrent.futures3.4.2 A solution based on the multiprocessing module3.4.3 Monitoring the progress of the multiprocessing solution3.4.4 Transferring data in chunks3.5 Tying it all together: An asynchronous multithreaded and multiprocessing MapReduce server3.5.1 Architecting a complete high-performance solution3.5.2 Creating a robust version of the serverSummary
4 High-performance NumPy
4.1 Understanding NumPy from a performance perspective4.1.1 Copies vs. views of existing arrays4.1.2 Understanding NumPy’s view machinery4.1.3 Making use of views for efficiency4.2 Using array programming4.2.1 The takeaway4.2.2 Broadcasting in NumPy4.2.3 Applying array programming4.2.4 Developing a vectorized mentality4.3 Tuning NumPy’s internal architecture for performance4.3.1 An overview of NumPy dependencies4.3.2 How to tune NumPy in your Python distribution4.3.3 Threads in NumPySummary

Part 2. Hardware
5 Re-implementing critical code with Cython
5.1 Overview of techniques for efficient code re-implementation5.2 A whirlwind tour of Cython5.2.1 A naive implementation in Cython5.2.2 Using Cython annotations to increase performance5.2.3 Why annotations are fundamental to performance5.2.4 Adding typing to function returns5.3 Profiling Cython code5.3.1 Using Python’s built-in profiling infrastructure5.3.2 Using line_profiler5.4 Optimizing array access with Cython memoryviews5.4.1 The takeaway5.4.2 Cleaning up all internal interactions with Python5.5 Writing NumPy generalized universal functions in Cython5.5.1 The takeaway5.6 Advanced array access in Cython5.6.1 Bypassing the GIL’s limitation on running multiple threads at a time5.6.2 Basic performance analysis5.6.3 A spacewar example using Quadlife5.7 Parallelism with CythonSummary
6 Memory hierarchy, storage, and networking
6.1 How modern hardware architectures affect Python performance6.1.1 The counterintuitive effect of modern architectures on performance6.1.2 How CPU caching affects algorithm efficiency6.1.3 Modern persistent storage6.2 Efficient data storage with Blosc6.2.1 Compress data; save time6.2.2 Read speeds (and memory buffers)6.2.3 The effect of different compression algorithms on storage performance6.2.4 Using insights about data representation to increase compression6.3 Accelerating NumPy with NumExpr6.3.1 Fast expression processing6.3.2 How hardware architecture affects our results6.3.3 When NumExpr is not appropriate6.4 The performance implications of using the local network6.4.1 The sources of inefficiency with REST calls6.4.2 A naive client based on UDP and msgpack6.4.3 A UDP-based server6.4.4 Dealing with basic recovery on the client side6.4.5 Other suggestions for optimizing network computingSummary
Part 3. Applications and Libraries for Modern Data Processing
7 High-performance pandas and Apache Arrow
7.1 Optimizing memory and time when loading data7.1.1 Compressed vs. uncompressed data7.1.2 Type inference of columns7.1.3 The effect of data type precision7.1.4 Recoding and reducing data7.2 Techniques to increase data analysis speed7.2.1 Using indexing to accelerate access7.2.2 Row iteration strategies7.3 pandas on top of NumPy, Cython, and NumExpr7.3.1 Explicit use of NumPy7.3.2 pandas on top of NumExpr7.3.3 Cython and pandas7.4 Reading data into pandas with Arrow7.4.1 The relationship between pandas and Apache Arrow7.4.2 Reading a CSV file7.4.3 Analyzing with Arrow7.5 Using Arrow interop to delegate work to more efficient languages and systems7.5.1 Implications of Arrow’s language interop architecture7.5.2 Zero-copy operations on data with Arrow’s Plasma serverSummary
8 Storing big data
8.1 A unified interface for file access: fsspec8.1.1 Using fsspec to search for files in a GitHub repo8.1.2 Using fsspec to inspect zip files8.1.3 Accessing files using fsspec8.1.4 Using URL chaining to traverse different filesystems transparently8.1.5 Replacing filesystem backends8.1.6 Interfacing with PyArrow8.2 Parquet: An efficient format to store columnar data8.2.1 Inspecting Parquet metadata8.2.2 Column encoding with Parquet8.2.3 Partitioning with datasets8.3 Dealing with larger-than-memory datasets the old-fashioned way8.3.1 Memory mapping files with NumPy8.3.2 Chunk reading and writing of data frames8.4 Zarr for large-array persistence8.4.1 Understanding Zarr’s internal structure8.4.2 Storage of arrays in Zarr8.4.3 Creating a new array8.4.4 Parallel reading and writing of Zarr arraysSummary
Part 4. Advanced Topics
9 Data analysis using GPU computing
9.1 Making sense of GPU computing power9.1.1 Understanding the advantages of GPUs9.1.2 The relationship between CPUs and GPUs9.1.3 The internal architecture of GPUs9.1.4 Software architecture considerations9.2 Using Numba to generate GPU code9.2.1 Installation of GPU software for Python9.2.2 The basics of GPU programming with Numba9.2.3 Revisiting the Mandelbrot example using GPUs9.2.4 A NumPy version of the Mandelbrot code9.3 Performance analysis of GPU code: The case of a CuPy application9.3.1 GPU-based data analysis libraries9.3.2 Using CuPy: A GPU-based version of NumPy9.3.3 A basic interaction with CuPy9.3.4 Writing a Mandelbrot generator using Numba9.3.5 Writing a Mandelbrot generator using CUDA C9.3.6 Profiling tools for GPU codeSummary
10 Analyzing big data with Dask
10.1 Understanding Dask’s execution model10.1.1 A pandas baseline for comparison10.1.2 Developing a Dask-based data frame solution10.2 The computational cost of Dask operations10.2.1 Partitioning data for processing10.2.2 Persisting intermediate computations10.2.3 Algorithm implementations over distributed data frames10.2.4 Repartitioning the data10.2.5 Persisting distributed data frames10.3 Using Dask’s distributed scheduler10.3.1 The dask.distributed architecture10.3.2 Running code using dask.distributed10.3.3 Dealing with datasets larger than memorySummary
Appendix A. Setting up the environment
A.1 Setting up Anaconda PythonA.2 Installing your own Python distributionA.3 Using DockerA.4 Hardware considerations
Appendix B. Using Numba to generate efficient low-level code
B.1 Generating optimized code with NumbaB.2 Writing explicitly parallel functions in NumbaB.3 Writing NumPy-aware code in Numba
index

Content preview from Fast Python

front matter

preface

A few years ago, a Python-based pipeline that my team was working on suddenly ground to a halt. A process just kept using CPU and was not finalizing. This function was critical to the company and we needed to solve the problem sooner rather than later. We looked at the algorithm and it seemed OK—in fact, it was quite a simple implementation. After many hours with several engineers looking at the problem, we found that it all boiled down to searching on a list—a very big list. The problem was trivially solved after converting the list into a set. We ended up with a much smaller data structure with search times in milliseconds, not hours.

I had several epiphanies at that time:

It was a trivial problem, but our development ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781617297939Publisher Support Publisher Website Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Fast Python

by Tiago Antao

front matter

preface

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.