O'Reilly logo

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Mastering Concurrency in Python

Book Description

Take your programming skills to next level by mastering concepts of concurrency and parallelism in Python.

About This Book
  • Explore the core syntaxes and language features that enable concurrency in Python,
  • Understand when and where to use concurrency to keep data consistent and applications non-blocking, responsive, and reliable
  • A practical approach to utilize application scaffolding to design highly-scalable programs that are deeply rooted in go routines and channels.
Who This Book Is For

Programmers with some experience with Python wanting to build high-performance applications that scale by leveraging single-core, multicore, or distributed concurrency but have had difficulty following online materials on the topic. Readers should be familiar to Python programming syntax, error handling, and debugging.

What You Will Learn
  • Understand the idea of concurrency in programming and relevant concepts such as queues, threads, parallelism.
  • Explore the core syntax and language features that enable concurrency in simple Python problems, namely through concurrent, multiprocessing, asyncio.
  • Understand correct way to implement concurrency
  • Abstract methods to keep the data consistent and application non-blocking, responsive, and reliable.
  • Analyze problems commonly faced in concurrent programming.
  • Utilize application scaffolding to design highly scalable programs that are deeply rooted in goroutines and channels.
In Detail

Python is one of the most popular programming languages out there, with numerous libraries and frameworks that facilitate high-performance computing. While concurrency and parallelism in Python behave differently than those in other programming languages, it is still in every way possible to implement Python programs that run concurrently or in parallel and make a significant improvement in execution time. Mastering Concurrency in Python serves as a comprehensive introduction to various advanced concepts in concurrent engineering and programming.

This book starts by introducing the concepts and principles of the most important elements in concurrency, together with common problems that engineers and programmers face in concurrent programming. It then explores Python syntax and various functionalities that support concurrent and parallel programming that Python provides. Next the book discusses a number of advanced concepts in Python concurrency and how they interact with the Python ecosystem, including the aforementioned GIL. Finally, examples of how concurrency is being used to solve real-world problems will be discussed.

By the end of Mastering Concurrency in Python, you will have an extensive theoretical knowledge regarding concurrency, and a practical know-how of the ways concurrency is supported by the Python language.

Downloading the example code for this book. You can download the example code files for all Packt books you have purchased from your account at http://www.PacktPub.com. If you purchased this book elsewhere, you can visit http://www.PacktPub.com/support and register to have the code file.

Table of Contents

  1. Preface
  2. Concurrent and Parallel Programming - An Advanced Introduction
    1. What is concurrency?
      1. Concurrent vs. Sequential
      2. Concurrent versus parallel
    2. Not everything should be made concurrent
      1. Embarrassingly parallel
      2. Inherently sequential
      3. I/O bound
    3. The history, present, and future of concurrency
      1. The history
      2. The present
      3. The future
    4. A brief overview of mastering concurrency in Python
    5. Setting up your Python environment
    6. Summary
    7. Questions
    8. Further reading
  3. Amdahl's Law
    1. Technical requirements
    2. Amdahl’s law
      1. Terminologies
    3. Formula and interpretation
      1. Formula of Amdahl's law
        1. A quick example
      2. Implications
    4. Relation to the law of diminishing returns
    5. Simulation in Python
    6. Summary
    7. Questions
    8. Further reading
  4. Working with Threads in Python
    1. Technical requirements
    2. The Concept of a Thread
      1. Threads versus Processes
      2. Multithreading
      3. Introductory example in Python
    3. An overview of the threading module
      1. The thread module in Python 2
      2. The threading module in Python 3
    4. Creating a new thread in Python
      1. Starting a thread with the thread module
      2. Starting a thread with the threading module
    5. Synchronizing threads
      1. The concept of thread synchronization
      2. The threading.Lock class
      3. Example in Python
    6. Multithreaded priority queue
      1. A connection between real life and programatic queues
      2. The queue module
      3. Queue in concurrent programming
      4. Multithreaded priority queue
    7. Summary
    8. Questions
    9. Further reading
  5. Using the with Statement in Threads
    1. Technical Requirements
    2. Context Management
      1. Starting from Managing Files
      2. The with Statement as a Context Manager
      3. The Syntax of with Statement
    3. The with statement in Concurrent Programming
      1. Example of Deadlock Handling
    4. Summary
    5. Questions
    6. Further Reading
  6. Concurrent Web Scraping
    1. Technical Requirements
    2. The Basics of Web Scraping
      1. HTML
      2. GET and POST Requests
      3. Locating Elements in a Web Page
      4. Web Scraping with Python
        1. The requests Module
        2. BeautifulSoup
        3. Bringing It Together
    3. Concurrent Web Scraping
    4. Best Practices in Concurrent Web Scraping
      1. Consider Terms of Service and Web Scraping Policies
      2. Error Handling
      3. Update Your Program Regularly
      4. Avoid Over-Scraping
    5. Summary
    6. Questions
    7. Further Reading
  7. Working with Processes in Python
    1. Technical Requirements
    2. The Concept of a Process
      1. Processes vs. Threads
      2. Multiprocessing
      3. Introductory Example in Python
    3. An Overview of the multiprocessing Module
      1. The Process Class
      2. The Pool Class
      3. Determining the Current Process, Waiting and Terminating Processes
        1. Determining the Current Process
        2. Waiting for Processes
        3. Terminating Processes
    4. Interprocess Communication
      1. Message-Passing for a Single Worker
      2. Message-Passing between Several Workers
    5. Summary
    6. Questions
    7. Further Reading
  8. The Reduction Operation in Processes
  9. Concurrent Image Processing
  10. Introduction to Asynchronous I/O
  11. Asyncio – Pros and Cons
  12. TCP with Asyncio
  13. Deadlock
  14. Starvation
  15. Race Conditions
  16. The Global Interpreter Lock
  17. Designing Lock-Free and Lock-Based Concurrent Data Structures
  18. Memory Models and Operations on Atomic Types
  19. Building a Server from Scratch
  20. Testing, Debugging, and Scheduling Concurrent Applications