O'Reilly logo

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Python Cookbook, 3rd Edition

Book Description

If you need help writing programs in Python 3, or want to update older Python 2 code, this book is just the ticket. Packed with practical recipes written and tested with Python 3.3, this unique cookbook is for experienced Python programmers who want to focus on modern tools and idioms.

Inside, you’ll find complete recipes for more than a dozen topics, covering the core Python language as well as tasks common to a wide variety of application domains. Each recipe contains code samples you can use in your projects right away, along with a discussion about how and why the solution works.

Topics include:

  • Data Structures and Algorithms
  • Strings and Text
  • Numbers, Dates, and Times
  • Iterators and Generators
  • Files and I/O
  • Data Encoding and Processing
  • Functions
  • Classes and Objects
  • Metaprogramming
  • Modules and Packages
  • Network and Web Programming
  • Concurrency
  • Utility Scripting and System Administration
  • Testing, Debugging, and Exceptions
  • C Extensions

Table of Contents

  1. Preface
    1. Who This Book Is For
    2. Who This Book Is Not For
    3. Conventions Used in This Book
    4. Online Code Examples
    5. Using Code Examples
    6. Safari® Books Online
    7. How to Contact Us
    8. Acknowledgments
  2. 1. Data Structures and Algorithms
    1. 1.1. Unpacking a Sequence into Separate Variables
    2. 1.2. Unpacking Elements from Iterables of Arbitrary Length
    3. 1.3. Keeping the Last N Items
    4. 1.4. Finding the Largest or Smallest N Items
    5. 1.5. Implementing a Priority Queue
    6. 1.6. Mapping Keys to Multiple Values in a Dictionary
    7. 1.7. Keeping Dictionaries in Order
    8. 1.8. Calculating with Dictionaries
    9. 1.9. Finding Commonalities in Two Dictionaries
    10. 1.10. Removing Duplicates from a Sequence while Maintaining Order
    11. 1.11. Naming a Slice
    12. 1.12. Determining the Most Frequently Occurring Items in a Sequence
    13. 1.13. Sorting a List of Dictionaries by a Common Key
    14. 1.14. Sorting Objects Without Native Comparison Support
    15. 1.15. Grouping Records Together Based on a Field
    16. 1.16. Filtering Sequence Elements
    17. 1.17. Extracting a Subset of a Dictionary
    18. 1.18. Mapping Names to Sequence Elements
    19. 1.19. Transforming and Reducing Data at the Same Time
    20. 1.20. Combining Multiple Mappings into a Single Mapping
  3. 2. Strings and Text
    1. 2.1. Splitting Strings on Any of Multiple Delimiters
    2. 2.2. Matching Text at the Start or End of a String
    3. 2.3. Matching Strings Using Shell Wildcard Patterns
    4. 2.4. Matching and Searching for Text Patterns
    5. 2.5. Searching and Replacing Text
    6. 2.6. Searching and Replacing Case-Insensitive Text
    7. 2.7. Specifying a Regular Expression for the Shortest Match
    8. 2.8. Writing a Regular Expression for Multiline Patterns
    9. 2.9. Normalizing Unicode Text to a Standard Representation
    10. 2.10. Working with Unicode Characters in Regular Expressions
    11. 2.11. Stripping Unwanted Characters from Strings
    12. 2.12. Sanitizing and Cleaning Up Text
    13. 2.13. Aligning Text Strings
    14. 2.14. Combining and Concatenating Strings
    15. 2.15. Interpolating Variables in Strings
    16. 2.16. Reformatting Text to a Fixed Number of Columns
    17. 2.17. Handling HTML and XML Entities in Text
    18. 2.18. Tokenizing Text
    19. 2.19. Writing a Simple Recursive Descent Parser
    20. 2.20. Performing Text Operations on Byte Strings
  4. 3. Numbers, Dates, and Times
    1. 3.1. Rounding Numerical Values
    2. 3.2. Performing Accurate Decimal Calculations
    3. 3.3. Formatting Numbers for Output
    4. 3.4. Working with Binary, Octal, and Hexadecimal Integers
    5. 3.5. Packing and Unpacking Large Integers from Bytes
    6. 3.6. Performing Complex-Valued Math
    7. 3.7. Working with Infinity and NaNs
    8. 3.8. Calculating with Fractions
    9. 3.9. Calculating with Large Numerical Arrays
    10. 3.10. Performing Matrix and Linear Algebra Calculations
    11. 3.11. Picking Things at Random
    12. 3.12. Converting Days to Seconds, and Other Basic Time Conversions
    13. 3.13. Determining Last Friday’s Date
    14. 3.14. Finding the Date Range for the Current Month
    15. 3.15. Converting Strings into Datetimes
    16. 3.16. Manipulating Dates Involving Time Zones
  5. 4. Iterators and Generators
    1. 4.1. Manually Consuming an Iterator
    2. 4.2. Delegating Iteration
    3. 4.3. Creating New Iteration Patterns with Generators
    4. 4.4. Implementing the Iterator Protocol
    5. 4.5. Iterating in Reverse
    6. 4.6. Defining Generator Functions with Extra State
    7. 4.7. Taking a Slice of an Iterator
    8. 4.8. Skipping the First Part of an Iterable
    9. 4.9. Iterating Over All Possible Combinations or Permutations
    10. 4.10. Iterating Over the Index-Value Pairs of a Sequence
    11. 4.11. Iterating Over Multiple Sequences Simultaneously
    12. 4.12. Iterating on Items in Separate Containers
    13. 4.13. Creating Data Processing Pipelines
    14. 4.14. Flattening a Nested Sequence
    15. 4.15. Iterating in Sorted Order Over Merged Sorted Iterables
    16. 4.16. Replacing Infinite while Loops with an Iterator
  6. 5. Files and I/O
    1. 5.1. Reading and Writing Text Data
    2. 5.2. Printing to a File
    3. 5.3. Printing with a Different Separator or Line Ending
    4. 5.4. Reading and Writing Binary Data
    5. 5.5. Writing to a File That Doesn’t Already Exist
    6. 5.6. Performing I/O Operations on a String
    7. 5.7. Reading and Writing Compressed Datafiles
    8. 5.8. Iterating Over Fixed-Sized Records
    9. 5.9. Reading Binary Data into a Mutable Buffer
    10. 5.10. Memory Mapping Binary Files
    11. 5.11. Manipulating Pathnames
    12. 5.12. Testing for the Existence of a File
    13. 5.13. Getting a Directory Listing
    14. 5.14. Bypassing Filename Encoding
    15. 5.15. Printing Bad Filenames
    16. 5.16. Adding or Changing the Encoding of an Already Open File
    17. 5.17. Writing Bytes to a Text File
    18. 5.18. Wrapping an Existing File Descriptor As a File Object
    19. 5.19. Making Temporary Files and Directories
    20. 5.20. Communicating with Serial Ports
    21. 5.21. Serializing Python Objects
  7. 6. Data Encoding and Processing
    1. 6.1. Reading and Writing CSV Data
    2. 6.2. Reading and Writing JSON Data
    3. 6.3. Parsing Simple XML Data
    4. 6.4. Parsing Huge XML Files Incrementally
    5. 6.5. Turning a Dictionary into XML
    6. 6.6. Parsing, Modifying, and Rewriting XML
    7. 6.7. Parsing XML Documents with Namespaces
    8. 6.8. Interacting with a Relational Database
    9. 6.9. Decoding and Encoding Hexadecimal Digits
    10. 6.10. Decoding and Encoding Base64
    11. 6.11. Reading and Writing Binary Arrays of Structures
    12. 6.12. Reading Nested and Variable-Sized Binary Structures
    13. 6.13. Summarizing Data and Performing Statistics
  8. 7. Functions
    1. 7.1. Writing Functions That Accept Any Number of Arguments
    2. 7.2. Writing Functions That Only Accept Keyword Arguments
    3. 7.3. Attaching Informational Metadata to Function Arguments
    4. 7.4. Returning Multiple Values from a Function
    5. 7.5. Defining Functions with Default Arguments
    6. 7.6. Defining Anonymous or Inline Functions
    7. 7.7. Capturing Variables in Anonymous Functions
    8. 7.8. Making an N-Argument Callable Work As a Callable with Fewer Arguments
    9. 7.9. Replacing Single Method Classes with Functions
    10. 7.10. Carrying Extra State with Callback Functions
    11. 7.11. Inlining Callback Functions
    12. 7.12. Accessing Variables Defined Inside a Closure
  9. 8. Classes and Objects
    1. 8.1. Changing the String Representation of Instances
    2. 8.2. Customizing String Formatting
    3. 8.3. Making Objects Support the Context-Management Protocol
    4. 8.4. Saving Memory When Creating a Large Number of Instances
    5. 8.5. Encapsulating Names in a Class
    6. 8.6. Creating Managed Attributes
    7. 8.7. Calling a Method on a Parent Class
    8. 8.8. Extending a Property in a Subclass
    9. 8.9. Creating a New Kind of Class or Instance Attribute
    10. 8.10. Using Lazily Computed Properties
    11. 8.11. Simplifying the Initialization of Data Structures
    12. 8.12. Defining an Interface or Abstract Base Class
    13. 8.13. Implementing a Data Model or Type System
    14. 8.14. Implementing Custom Containers
    15. 8.15. Delegating Attribute Access
    16. 8.16. Defining More Than One Constructor in a Class
    17. 8.17. Creating an Instance Without Invoking init
    18. 8.18. Extending Classes with Mixins
    19. 8.19. Implementing Stateful Objects or State Machines
    20. 8.20. Calling a Method on an Object Given the Name As a String
    21. 8.21. Implementing the Visitor Pattern
    22. 8.22. Implementing the Visitor Pattern Without Recursion
    23. 8.23. Managing Memory in Cyclic Data Structures
    24. 8.24. Making Classes Support Comparison Operations
    25. 8.25. Creating Cached Instances
  10. 9. Metaprogramming
    1. 9.1. Putting a Wrapper Around a Function
    2. 9.2. Preserving Function Metadata When Writing Decorators
    3. 9.3. Unwrapping a Decorator
    4. 9.4. Defining a Decorator That Takes Arguments
    5. 9.5. Defining a Decorator with User Adjustable Attributes
    6. 9.6. Defining a Decorator That Takes an Optional Argument
    7. 9.7. Enforcing Type Checking on a Function Using a Decorator
    8. 9.8. Defining Decorators As Part of a Class
    9. 9.9. Defining Decorators As Classes
    10. 9.10. Applying Decorators to Class and Static Methods
    11. 9.11. Writing Decorators That Add Arguments to Wrapped Functions
    12. 9.12. Using Decorators to Patch Class Definitions
    13. 9.13. Using a Metaclass to Control Instance Creation
    14. 9.14. Capturing Class Attribute Definition Order
    15. 9.15. Defining a Metaclass That Takes Optional Arguments
    16. 9.16. Enforcing an Argument Signature on *args and **kwargs
    17. 9.17. Enforcing Coding Conventions in Classes
    18. 9.18. Defining Classes Programmatically
    19. 9.19. Initializing Class Members at Definition Time
    20. 9.20. Implementing Multiple Dispatch with Function Annotations
    21. 9.21. Avoiding Repetitive Property Methods
    22. 9.22. Defining Context Managers the Easy Way
    23. 9.23. Executing Code with Local Side Effects
    24. 9.24. Parsing and Analyzing Python Source
    25. 9.25. Disassembling Python Byte Code
  11. 10. Modules and Packages
    1. 10.1. Making a Hierarchical Package of Modules
    2. 10.2. Controlling the Import of Everything
    3. 10.3. Importing Package Submodules Using Relative Names
    4. 10.4. Splitting a Module into Multiple Files
    5. 10.5. Making Separate Directories of Code Import Under a Common Namespace
    6. 10.6. Reloading Modules
    7. 10.7. Making a Directory or Zip File Runnable As a Main Script
    8. 10.8. Reading Datafiles Within a Package
    9. 10.9. Adding Directories to sys.path
    10. 10.10. Importing Modules Using a Name Given in a String
    11. 10.11. Loading Modules from a Remote Machine Using Import Hooks
    12. 10.12. Patching Modules on Import
    13. 10.13. Installing Packages Just for Yourself
    14. 10.14. Creating a New Python Environment
    15. 10.15. Distributing Packages
  12. 11. Network and Web Programming
    1. 11.1. Interacting with HTTP Services As a Client
    2. 11.2. Creating a TCP Server
    3. 11.3. Creating a UDP Server
    4. 11.4. Generating a Range of IP Addresses from a CIDR Address
    5. 11.5. Creating a Simple REST-Based Interface
    6. 11.6. Implementing a Simple Remote Procedure Call with XML-RPC
    7. 11.7. Communicating Simply Between Interpreters
    8. 11.8. Implementing Remote Procedure Calls
    9. 11.9. Authenticating Clients Simply
    10. 11.10. Adding SSL to Network Services
    11. 11.11. Passing a Socket File Descriptor Between Processes
    12. 11.12. Understanding Event-Driven I/O
    13. 11.13. Sending and Receiving Large Arrays
  13. 12. Concurrency
    1. 12.1. Starting and Stopping Threads
    2. 12.2. Determining If a Thread Has Started
    3. 12.3. Communicating Between Threads
    4. 12.4. Locking Critical Sections
    5. 12.5. Locking with Deadlock Avoidance
    6. 12.6. Storing Thread-Specific State
    7. 12.7. Creating a Thread Pool
    8. 12.8. Performing Simple Parallel Programming
    9. 12.9. Dealing with the GIL (and How to Stop Worrying About It)
    10. 12.10. Defining an Actor Task
    11. 12.11. Implementing Publish/Subscribe Messaging
    12. 12.12. Using Generators As an Alternative to Threads
    13. 12.13. Polling Multiple Thread Queues
    14. 12.14. Launching a Daemon Process on Unix
  14. 13. Utility Scripting and System Administration
    1. 13.1. Accepting Script Input via Redirection, Pipes, or Input Files
    2. 13.2. Terminating a Program with an Error Message
    3. 13.3. Parsing Command-Line Options
    4. 13.4. Prompting for a Password at Runtime
    5. 13.5. Getting the Terminal Size
    6. 13.6. Executing an External Command and Getting Its Output
    7. 13.7. Copying or Moving Files and Directories
    8. 13.8. Creating and Unpacking Archives
    9. 13.9. Finding Files by Name
    10. 13.10. Reading Configuration Files
    11. 13.11. Adding Logging to Simple Scripts
    12. 13.12. Adding Logging to Libraries
    13. 13.13. Making a Stopwatch Timer
    14. 13.14. Putting Limits on Memory and CPU Usage
    15. 13.15. Launching a Web Browser
  15. 14. Testing, Debugging, and Exceptions
    1. 14.1. Testing Output Sent to stdout
    2. 14.2. Patching Objects in Unit Tests
    3. 14.3. Testing for Exceptional Conditions in Unit Tests
    4. 14.4. Logging Test Output to a File
    5. 14.5. Skipping or Anticipating Test Failures
    6. 14.6. Handling Multiple Exceptions
    7. 14.7. Catching All Exceptions
    8. 14.8. Creating Custom Exceptions
    9. 14.9. Raising an Exception in Response to Another Exception
    10. 14.10. Reraising the Last Exception
    11. 14.11. Issuing Warning Messages
    12. 14.12. Debugging Basic Program Crashes
    13. 14.13. Profiling and Timing Your Program
    14. 14.14. Making Your Programs Run Faster
  16. 15. C Extensions
    1. 15.1. Accessing C Code Using ctypes
    2. 15.2. Writing a Simple C Extension Module
    3. 15.3. Writing an Extension Function That Operates on Arrays
    4. 15.4. Managing Opaque Pointers in C Extension Modules
    5. 15.5. Defining and Exporting C APIs from Extension Modules
    6. 15.6. Calling Python from C
    7. 15.7. Releasing the GIL in C Extensions
    8. 15.8. Mixing Threads from C and Python
    9. 15.9. Wrapping C Code with Swig
    10. 15.10. Wrapping Existing C Code with Cython
    11. 15.11. Using Cython to Write High-Performance Array Operations
    12. 15.12. Turning a Function Pointer into a Callable
    13. 15.13. Passing NULL-Terminated Strings to C Libraries
    14. 15.14. Passing Unicode Strings to C Libraries
    15. 15.15. Converting C Strings to Python
    16. 15.16. Working with C Strings of Dubious Encoding
    17. 15.17. Passing Filenames to C Extensions
    18. 15.18. Passing Open Files to C Extensions
    19. 15.19. Reading File-Like Objects from C
    20. 15.20. Consuming an Iterable from C
    21. 15.21. Diagnosing Segmentation Faults
  17. A. Further Reading
    1. Online Resources
    2. Books for Learning Python
    3. Advanced Books
  18. Index
  19. Colophon
  20. Copyright