book

Programming Python, 3rd Edition

Name: Programming Python, 3rd Edition
Author: Mark Lutz
ISBN: 9780596009250

by Mark Lutz

August 2006

Intermediate to advanced

1600 pages

51h 46m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Programming Python, 3rd Edition
A Note Regarding Supplemental Files
Foreword
How Time Flies!
Coming Attractions
Concluding Remarks
Foreword to the Second Edition (2001)
Foreword from the First Edition (1996)
Preface
“And Now for Something Completely Different . . . Again”

About This Book
About This Edition
Python ChangesExample ChangesTopic ChangesFocus Unchanged
This Book’s Motivation
This Book’s Structure
This Edition’s Design
It’s Been Updated for Python 2.4 (and 2.5)It’s Been ReorganizedIt Covers New TopicsIt’s Still Mostly Platform-NeutralC integration code platform issuesIt’s Still Focused for a More Advanced AudienceIt’s Still Example-Oriented
Using the Book’s Examples
The Book Examples TreeThe Book Examples Distribution PackageRunning Examples: The Short StoryRunning Examples: The DetailsDemos and gadgetsSetup requirementsWeb-based examplesTop-level programs
Conventions Used in This Book
Safari® Enabled
Where to Look for Updates
Contacting O’Reilly
Using Code Examples
Acknowledgments
I. The Beginning
1. Introducing Python
1.1. “And Now for Something Completely Different”
1.2. Python Philosophy 101
1.3. The Life of Python
1.4. Signs of the Python Times
1.4.1. Recent Growth (As of 2005, at Least)
1.5. The Compulsory Features List
1.6. What’s Python Good For?
1.7. What’s Python Not Good For?
1.8. Truth in Advertising
2. A Sneak Preview
2.1. “Programming Python: The Short Story”
2.2. The Task
2.3. Step 1: Representing Records
2.3.1. Using Lists2.3.1.1. A database list2.3.1.2. Field labels2.3.2. Using Dictionaries2.3.2.1. Other ways to make dictionaries2.3.2.2. Lists of dictionaries2.3.2.3. Nested structures2.3.2.4. Dictionaries of dictionaries
2.4. Step 2: Storing Records Persistently
2.4.1. Using Formatted Files2.4.1.1. Test data script2.4.1.2. Data format script2.4.1.3. Utility scripts2.4.2. Using Pickle Files2.4.3. Using Per-Record Pickle Files2.4.4. Using Shelves
2.5. Step 3: Stepping Up to OOP
2.5.1. Using Classes2.5.2. Adding Behavior2.5.3. Adding Inheritance2.5.4. Refactoring Code2.5.4.1. Augmenting methods2.5.4.2. Display format2.5.4.3. Constructor customization2.5.4.4. Alternative classes2.5.5. Adding Persistence2.5.6. Other Database Options
2.6. Step 4: Adding Console Interaction
2.6.1. A Console Shelve Interface
2.7. Step 5: Adding a GUI
2.7.1. GUI Basics2.7.2. Using OOP for GUIs2.7.3. Getting Input from a User2.7.4. A GUI Shelve Interface2.7.4.1. Coding the GUI2.7.4.2. Using the GUI2.7.4.3. Future directions
2.8. Step 6: Adding a Web Interface
2.8.1. CGI Basics2.8.2. Running a Web Server2.8.3. Using Query Strings and urllib2.8.4. Formatting Reply Text2.8.5. A Web-Based Shelve Interface2.8.5.1. Coding the web site2.8.5.2. Using the web site2.8.5.3. Future directions
2.9. The End of the Demo
II. System Programming
3. System Tools
3.1. “The os.path to Knowledge”
3.1.1. Why Python Here?3.1.2. The Next Five Chapters
3.2. System Scripting Overview
3.2.1. Python System Modules3.2.2. Module Documentation Sources3.2.3. Paging Documentation Strings3.2.4. Introducing String Methods3.2.5. File Operation Basics3.2.6. Using Programs in Two Ways3.2.7. Python Library Manuals3.2.8. Commercially Published References
3.3. Introducing the sys Module
3.3.1. Platforms and Versions3.3.2. The Module Search Path3.3.3. The Loaded Modules Table3.3.4. Exception Details3.3.5. Other sys Module Exports
3.4. Introducing the os Module
3.4.1. The Big os Lists3.4.2. Administrative Tools3.4.3. Portability Constants3.4.4. Basic os.path Tools3.4.5. Running Shell Commands from Scripts3.4.5.1. What’s a shell command?3.4.5.2. Running shell commands3.4.5.3. Communicating with shell commands3.4.5.4. Shell command limitations3.4.6. Other os Module Exports
3.5. Script Execution Context
3.6. Current Working Directory
3.6.1. CWD, Files, and Import Paths3.6.2. CWD and Command Lines
3.7. Command-Line Arguments
3.8. Shell Environment Variables
3.8.1. Fetching Shell Variables3.8.2. Changing Shell Variables3.8.3. Shell Variable Details
3.9. Standard Streams
3.9.1. Redirecting Streams to Files and Programs3.9.1.1. Redirecting streams to files3.9.1.2. Chaining programs with pipes3.9.1.3. Coding alternatives3.9.1.4. Redirected streams and user interaction3.9.1.5. Reading keyboard input3.9.2. Redirecting Streams to Python Objects3.9.3. The StringIO Module3.9.4. Capturing the stderr Stream3.9.5. Redirection Syntax in Print Statements3.9.6. Other Redirection Options
4. File and Directory Tools
4.1. “Erase Your Hard Drive in Five Easy Steps!”
4.2. File Tools
4.2.1. Built-In File Objects4.2.1.1. Output files4.2.1.1.1. Opening4.2.1.1.2. Writing4.2.1.1.3. Closing4.2.1.2. Input files4.2.1.3. Reading lines with file iterators4.2.1.4. Other file object modes4.2.1.5. Binary datafiles4.2.1.6. End-of-line translations on Windows4.2.1.7. Parsing packed binary data with the struct module4.2.2. File Tools in the os Module4.2.2.1. Open mode flags4.2.2.2. Other os file tools4.2.3. File Scanners
4.3. Directory Tools
4.3.1. Walking One Directory4.3.1.1. Running shell listing commands with os.popen4.3.1.2. The glob module4.3.1.3. The os.listdir call4.3.1.4. Splitting and joining listing results4.3.2. Walking Directory Trees4.3.2.1. The os.path.walk visitor4.3.2.2. The os.walk generator4.3.2.3. Recursive os.listdir traversals4.3.3. Rolling Your Own find Module
5. Parallel System Tools
5.1. “Telling the Monkeys What to Do”
5.2. Forking Processes
5.2.1. The fork/exec Combination5.2.1.1. os.exec call formats5.2.1.2. Spawned child program
5.3. Threads
5.3.1. The thread Module5.3.1.1. Synchronizing access to global objects5.3.1.2. Waiting for spawned thread exits5.3.2. The threading Module5.3.3. The Queue Module5.3.4. The Global Interpreter Lock and Threads5.3.4.1. The thread switch interval5.3.4.2. Atomic operations5.3.4.3. C API thread considerations
5.4. Program Exits
5.4.1. os Module Exits5.4.2. Exit Status Codes5.4.3. Process Exit Status5.4.4. Thread Exits
5.5. Interprocess Communication
5.6. Pipes
5.6.1. Anonymous Pipe Basics5.6.2. Bidirectional IPC with Pipes5.6.2.1. Deadlocks, flushes, and unbuffered streams5.6.3. Named Pipes (Fifos)
5.7. Signals
5.8. Other Ways to Start Programs
5.8.1. The os.spawn Calls5.8.2. Launching Programs on Windows5.8.2.1. Using the DOS start command5.8.2.2. Using start in Python scripts5.8.2.3. The os.startfile call5.8.3. Other Program Launch Options
5.9. A Portable Program-Launch Framework
5.10. Other System Tools
6. System Examples: Utilities
6.1. “Splits and Joins and Alien Invasions”
6.2. Splitting and Joining Files
6.2.1. Splitting Files Portably6.2.1.1. Operation modes6.2.1.2. Binary file access6.2.1.3. Manually closing files6.2.2. Joining Files Portably6.2.2.1. Reading by blocks or files6.2.2.2. Sorting filenames6.2.3. Usage Variations
6.3. Generating Forward-Link Web Pages
6.3.1. Page Template File6.3.2. Page Generator Script
6.4. A Regression Test Script
6.5. Packing and Unpacking Files
6.5.1. Packing Files “++”6.5.2. Application Hierarchy Superclasses6.5.2.1. StreamApp: adding stream redirection6.5.2.2. App: the root class6.5.2.3. Why use classes here?
6.6. Automated Program Launchers
6.6.1. Launcher Module Clients6.6.2. Launching Programs Without Environment Settings6.6.3. Launching Web Browsers Portably6.6.3.1. Launching browsers with command lines6.6.3.2. Launching browsers with function calls6.6.3.3. Viewing multimedia in browsers6.6.4. A Portable Media File Player Tool6.6.4.1. The Python webbrowser module6.6.4.2. The Python mimetypes module
7. System Examples: Directories
7.1. “The Greps of Wrath”
7.2. Fixing DOS Line Ends
7.2.1. Converting Line Ends in One File7.2.1.1. Slinging bytes and verifying results7.2.1.2. Nonintrusive conversions7.2.1.3. Slicing strings out of bounds7.2.1.4. Binary file mode revisited7.2.2. Converting Line Ends in One Directory7.2.3. Converting Line Ends in an Entire Tree7.2.3.1. The view from the top
7.3. Fixing DOS Filenames
7.3.1. Rewriting with os.path.walk
7.4. Searching Directory Trees
7.4.1. Greps and Globs in Shells and Python7.4.1.1. Cleaning up bytecode files7.4.2. A Python Tree Searcher
7.5. Visitor: Walking Trees Generically
7.5.1. Editing Files in Directory Trees7.5.2. Global Replacements in Directory Trees7.5.3. Collecting Matched Files in Trees7.5.3.1. Suppressing status messages7.5.4. Recoding Fixers with Visitors7.5.5. Fixing File Permissions in Trees7.5.6. Changing Unix Executable Path Lines7.5.7. Summary: Counting Source Lines Four Ways
7.6. Copying Directory Trees
7.6.1. A Python Tree Copy Script7.6.2. Recoding Copies with a Visitor-Based Class
7.7. Deleting Directory Trees
7.7.1. Recoding Deletions for Generality
7.8. Comparing Directory Trees
7.8.1. Finding Directory Differences7.8.2. Finding Tree Differences7.8.3. Running the Script7.8.3.1. Verifying CD backups7.8.4. Reporting Differences
III. GUI Programming
8. Graphical User Interfaces
8.1. “Here’s Looking at You, Kid”
8.1.1. GUI Programming Topics
8.2. Python GUI Development Options
8.3. Tkinter Overview
8.3.1. Tkinter Pragmatics8.3.2. Tkinter Documentation8.3.3. Tkinter Extensions8.3.4. Tkinter Structure
8.4. Climbing the GUI Learning Curve
8.4.1. “Hello World” in Four Lines (or Less)
8.5. Tkinter Coding Basics
8.5.1. Making Widgets8.5.2. Geometry Managers8.5.3. Running GUI Programs8.5.3.1. Avoiding DOS consoles on Windows
8.6. Tkinter Coding Alternatives
8.6.1. Widget Resizing Basics8.6.2. Configuring Widget Options and Window Titles8.6.3. One More for Old Times’ Sake8.6.4. Packing Widgets Without Saving Them
8.7. Adding Buttons and Callbacks
8.7.1. Widget Resizing Revisited: Expansion
8.8. Adding User-Defined Callback Handlers
8.8.1. Lambda Callback Handlers8.8.1.1. Passing in values with default arguments8.8.1.2. Passing in values with enclosing scope references8.8.1.3. Enclosing scopes versus defaults8.8.2. Bound Method Callback Handlers8.8.3. Callable Class Object Callback Handlers8.8.4. Other Tkinter Callback Protocols8.8.5. Binding Events
8.9. Adding Multiple Widgets
8.9.1. Widget Resizing Revisited: Clipping8.9.2. Attaching Widgets to Frames8.9.3. Layout: Packing Order and Side Attachments8.9.4. The Packer’s Expand and Fill Revisited8.9.5. Using Anchor to Position Instead of Stretch
8.10. Customizing Widgets with Classes
8.11. Reusable GUI Components with Classes
8.11.1. Attaching Class Components8.11.2. Extending Class Components8.11.3. Standalone Container Classes
8.12. The End of the Tutorial
8.13. Python/Tkinter for Tcl/Tk Converts
9. A Tkinter Tour, Part 1
9.1. “Widgets and Gadgets and GUIs, Oh My!”
9.1.1. This Chapter’s Topics
9.2. Configuring Widget Appearance
9.3. Top-Level Windows
9.3.1. Toplevel and Tk Widgets9.3.2. Top-Level Window Protocols
9.4. Dialogs
9.4.1. Standard (Common) Dialogs9.4.1.1. A “smart” and reusable Quit button9.4.1.2. A dialog demo launcher bar9.4.1.3. Printing dialog results (and passing callback data with lambdas)9.4.1.4. Letting users select colors on the fly9.4.1.5. Other standard dialog calls9.4.2. The Old-Style Dialog Module9.4.3. Custom Dialogs9.4.3.1. Making custom dialogs modal9.4.3.2. Other ways to be modal
9.5. Binding Events
9.6. Message and Entry
9.6.1. Message9.6.2. Entry9.6.2.1. Programming Entry widgets9.6.2.2. Laying out input forms9.6.2.3. Going modal again9.6.2.4. Tkinter “variables”
9.7. Checkbutton, Radiobutton, and Scale
9.7.1. Checkbuttons9.7.1.1. Check buttons and variables9.7.2. Radio Buttons9.7.2.1. Radio buttons and variables9.7.2.2. Radio buttons without variables9.7.2.3. Hold onto your variables9.7.3. Scales (Sliders)9.7.3.1. Scales and variables
9.8. Running GUI Code Three Ways
9.8.1. Attaching Frames9.8.2. Independent Windows9.8.3. Running Programs9.8.3.1. Cross-program communication9.8.3.2. Coding for reusability
9.9. Images
9.9.1. Fun with Buttons and Pictures
9.10. Viewing and Processing Images with PIL
9.10.1. PIL Basics9.10.2. Displaying Other Image Types with PIL9.10.2.1. Displaying all images in a directory9.10.3. Creating Image Thumbnails with PIL9.10.3.1. Performance: saving thumbnail files9.10.3.2. Layout: gridding and fixed-size widgets9.10.3.3. Scrolling and canvases
10. A Tkinter Tour, Part 2
10.1. “On Today’s Menu: Spam, Spam, and Spam”
10.2. Menus
10.2.1. Top-Level Window Menus10.2.2. Frame- and Menubutton-Based Menus10.2.2.1. Using Menubuttons and Optionmenus10.2.3. Windows with Both Menus and Toolbars10.2.3.1. Automating menu construction
10.3. Listboxes and Scrollbars
10.3.1. Programming Listboxes10.3.2. Programming Scroll Bars10.3.3. Packing Scroll Bars
10.4. Text
10.4.1. Programming the Text Widget10.4.1.1. Text is a Python string10.4.1.2. String positions10.4.1.2.1. Text indexes10.4.1.2.2. Text marks10.4.1.2.3. Text tags10.4.2. Adding Text-Editing Operations10.4.2.1. Using the clipboard10.4.2.2. Composition versus inheritance10.4.2.3. It’s called “Simple” for a reason10.4.3. Advanced Text and Tag Operations
10.5. Canvas
10.5.1. Basic Canvas Operations10.5.2. Programming the Canvas Widget10.5.2.1. Coordinates10.5.2.2. Object construction10.5.2.3. Object identifiers and operations10.5.2.4. Canvas object tags10.5.3. Scrolling Canvases10.5.4. Scrollable Canvases and Image Thumbnails10.5.4.1. Scrolling images too: PyPhoto (ahead)10.5.5. Using Canvas Events10.5.5.1. Binding events on specific items
10.6. Grids
10.6.1. Grid Basics10.6.2. grid Versus pack10.6.3. Combining grid and pack10.6.4. Making Gridded Widgets Expandable10.6.4.1. Resizing in grids10.6.4.2. Spanning columns and rows10.6.5. Laying Out Larger Tables with grid
10.7. Time Tools, Threads, and Animation
10.7.1. Using Threads with GUIs10.7.2. Using the after Method10.7.2.1. Hiding and redrawing widgets and windows10.7.3. Simple Animation Techniques10.7.3.1. Using time.sleep loops10.7.3.2. Using widget.after events10.7.3.3. Using multiple time.sleep loop threads10.7.4. Other Animation Concepts: Threads and Toolkits
10.8. The End of the Tour
10.9. The PyDemos and PyGadgets Launchers
10.9.1. PyDemos Launcher Bar10.9.2. PyGadgets Launcher Bar
11. GUI Coding Techniques
11.1. “Building a Better Mouse Trap”
11.2. GuiMixin: Common Tool Mixin Classes
11.3. GuiMaker: Automating Menus and Toolbars
11.3.1. Subclass Protocols11.3.2. GuiMaker Classes11.3.3. GuiMaker Self-Test11.3.4. BigGui: A Client Demo Program
11.4. ShellGui: GUIs for Command-Line Tools
11.4.1. A Generic Shell-Tools Display11.4.2. Application-Specific Tool Set Classes11.4.3. Adding GUI Frontends to Command Lines
11.5. GuiStreams: Redirecting Streams to Widgets
11.5.1. Using Redirection for the Packing Scripts
11.6. Reloading Callback Handlers Dynamically
11.7. Wrapping Up Top-Level Window Interfaces
11.8. GUIs, Threads, and Queues
11.8.1. Placing Callbacks on Queues
11.9. More Ways to Add GUIs to Non-GUI Code
11.9.1. Popping up GUI Windows on Demand11.9.2. Adding a GUI As a Separate Program: Sockets11.9.3. Adding a GUI As a Separate Program: Pipes
12. Complete GUI Programs
12.1. “Python, Open Source, and Camaros”
12.1.1. Examples in Other Chapters12.1.2. This Chapter’s Strategy
12.2. PyEdit: A Text Editor Program/Object
12.2.1. Running PyEdit12.2.1.1. Menus and toolbars12.2.1.2. Dialogs12.2.1.3. Running program code12.2.1.4. New features in version 2.012.2.2. PyEdit Source Code
12.3. PyPhoto: An Image Viewer and Resizer
12.3.1. Running PyPhoto12.3.2. PyPhoto Source Code
12.4. PyView: An Image and Notes Slideshow
12.4.1. Running PyView12.4.2. PyView Source Code
12.5. PyDraw: Painting and Moving Graphics
12.5.1. Running PyDraw12.5.2. PyDraw Source Code
12.6. PyClock: An Analog/Digital Clock Widget
12.6.1. A Quick Geometry Lesson12.6.2. Running PyClock12.6.3. PyClock Source Code
12.7. PyToe: A Tic-Tac-Toe Game Widget
12.7.1. Running PyToe12.7.2. PyToe Source Code (Book Examples Distribution)
12.8. Where to Go from Here
IV. Internet Programming
13. Network Scripting
13.1. “Tune In, Log On, and Drop Out”
13.1.1. Internet Scripting Topics13.1.1.1. What we will cover13.1.1.2. What we won’t cover13.1.1.3. Running examples in this part of the book
13.2. Plumbing the Internet
13.2.1. The Socket Layer13.2.1.1. Machine identifiers13.2.2. The Protocol Layer13.2.2.1. Port number rules13.2.2.2. Clients and servers13.2.2.3. Protocol structures13.2.3. Python’s Internet Library Modules
13.3. Socket Programming
13.3.1. Socket Basics13.3.1.1. Server socket calls13.3.1.2. Client socket calls13.3.1.3. Running socket programs locally13.3.1.4. Running socket programs remotely13.3.1.5. Socket pragmatics13.3.1.6. Spawning clients in parallel13.3.1.7. Talking to reserved ports
13.4. Handling Multiple Clients
13.4.1. Forking Servers13.4.1.1. Running the forking server13.4.1.2. Forking processes13.4.1.3. Exiting from children13.4.1.4. Killing the zombies13.4.1.5. Preventing zombies with signal handlers13.4.2. Threading Servers13.4.3. Standard Library Server Classes13.4.4. Third-Party Server Tools: Twisted13.4.5. Multiplexing Servers with select13.4.5.1. A select-based echo server13.4.5.2. Running the select server13.4.6. Choosing a Server Scheme
13.5. A Simple Python File Server
13.5.1. Running the File Server and Clients13.5.2. Adding a User-Interface Frontend13.5.2.1. Using Frames and command lines13.5.2.2. Using grids and function calls13.5.2.3. Using a reusable form-layout class
14. Client-Side Scripting
14.1. “Socket to Me!”
14.2. FTP: Transferring Files over the Net
14.2.1. Fetching Files with ftplib14.2.2. Using urllib to FTP Files14.2.3. FTP get and put Utilities14.2.3.1. Download utility14.2.3.2. Upload utility14.2.3.3. Playing the Monty Python theme song14.2.3.4. Adding user interfaces14.2.4. Downloading Web Sites (Mirrors)14.2.5. Uploading Web Sites14.2.6. Refactoring Uploads and Downloads for Reuse14.2.6.1. Refactoring with functions14.2.6.2. Refactoring with classes14.2.7. Uploads and Deletes with Subdirectories14.2.7.1. Uploading local trees14.2.7.2. Deleting remote trees
14.3. Processing Internet Email
14.4. POP: Fetching Email
14.4.1. Mail Configuration Module14.4.2. POP Mail Reader Script14.4.3. Fetching Messages14.4.4. Fetching Email at the Interactive Prompt
14.5. SMTP: Sending Email
14.5.1. SMTP Mail Sender Script14.5.2. Sending Messages14.5.3. More Ways to Abuse the Net14.5.4. Back to the Big Internet Picture14.5.5. Sending Email from the Interactive Prompt
14.6. email: Parsing and Composing Mails
14.6.1. Message Objects14.6.2. Basic email Interfaces in Action
14.7. pymail: A Console-Based Email Client
14.7.1. Running the pymail Console Client
14.8. The mailtools Utility Package
14.8.1. Initialization File14.8.2. MailTool Class14.8.3. MailSender Class14.8.4. MailFetcher Class14.8.4.1. General usage14.8.4.2. Inbox synchronization tools14.8.5. MailParser Class14.8.6. Self-Test Script14.8.7. Updating the pymail Console Client
14.9. NNTP: Accessing Newsgroups
14.10. HTTP: Accessing Web Sites
14.11. Module urllib Revisited
14.11.1. Other urllib Interfaces
14.12. Other Client-Side Scripting Options
15. The PyMailGUI Client
15.1. “Use the Source, Luke”
15.1.1. Source Code Modules15.1.2. Why PyMailGUI?15.1.3. Running PyMailGUI15.1.4. Presentation Strategy15.1.5. New in This Edition
15.2. A PyMailGUI Demo
15.2.1. Getting Started15.2.2. Loading Mail15.2.3. Threading Model15.2.4. Load Server Interface15.2.5. Offline Processing with Save and Open15.2.6. Sending Email and Attachments15.2.7. Viewing Email and Attachments15.2.8. Email Replies and Forwards15.2.9. Deleting Email15.2.10. POP Message Numbers and Synchronization15.2.11. Multiple Windows and Status Messages
15.3. PyMailGUI Implementation
15.3.1. Code Reuse15.3.2. Code Structure15.3.3. PyMailGui2: The Main Module15.3.4. SharedNames: Program-Wide Globals15.3.5. ListWindows: Message List Windows15.3.6. ViewWindows: Message View Windows15.3.7. messagecache: Message Cache Manager15.3.8. popuputil: General-Purpose GUI Pop Ups15.3.9. wraplines: Line Split Tools15.3.10. mailconfig: User Configurations15.3.11. PyMailGuiHelp: User Help Text15.3.12. Ideas for Improvement
16. Server-Side Scripting
16.1. “Oh What a Tangled Web We Weave”
16.2. What’s a Server-Side CGI Script?
16.2.1. The Script Behind the Curtain16.2.2. Writing CGI Scripts in Python
16.3. Running Server-Side Examples
16.3.1. Web Server Options16.3.2. Running a Local Web Server16.3.3. The Server-Side Examples Root Page16.3.4. Viewing Server-Side Examples and Output
16.4. Climbing the CGI Learning Curve
16.4.1. A First Web Page16.4.1.1. HTML basics16.4.1.2. Internet addresses (URLs)16.4.1.3. Using minimal URLs16.4.1.4. HTML file permission constraints16.4.2. A First CGI Script16.4.2.1. Installing CGI scripts16.4.2.2. Finding Python on remote servers16.4.3. Adding Pictures and Generating Tables16.4.3.1. Table tags16.4.4. Adding User Interaction16.4.4.1. Submission page16.4.4.2. More on form tags16.4.4.3. Response script16.4.4.4. Passing parameters in URLs16.4.4.5. Testing outside browsers with the module urllib16.4.5. Using Tables to Lay Out Forms16.4.5.1. Converting strings in CGI scripts16.4.5.2. Debugging CGI scripts16.4.6. Adding Common Input Devices16.4.7. Changing Input Layouts16.4.7.1. Keeping display and logic separate16.4.8. Passing Parameters in Hardcoded URLs16.4.9. Passing Parameters in Hidden Form Fields
16.5. Saving State Information in CGI Scripts
16.5.1. URL Query Parameters16.5.2. Hidden Form Input Fields16.5.3. HTTP “Cookies”16.5.3.1. Creating a cookie16.5.3.2. Receiving a cookie16.5.3.3. Using cookies in CGI scripts16.5.3.4. Handling cookies with the module urllib216.5.4. Server-Side Databases16.5.5. Extensions to the CGI Model16.5.6. Combining Techniques
16.6. The Hello World Selector
16.6.1. Checking for Missing and Invalid Inputs
16.7. Refactoring Code for Maintainability
16.7.1. Step 1: Sharing Objects Between Pages—A New Input Form16.7.2. Step 2: A Reusable Form Mock-Up Utility16.7.3. Step 3: Putting It All Together—A New Reply Script
16.8. More on HTML and URL Escapes
16.8.1. URL Escape Code Conventions16.8.2. Python HTML and URL Escape Tools16.8.3. Escaping HTML Code16.8.4. Escaping URLs16.8.5. Escaping URLs Embedded in HTML Code16.8.5.1. HTML and URL conflicts: &16.8.5.2. Avoiding conflicts
16.9. Transferring Files to Clients and Servers
16.9.1. Displaying Arbitrary Server Files on the Client16.9.1.1. Handling private files and errors16.9.2. Uploading Client Files to the Server16.9.2.1. Handling client path formats16.9.3. More Than One Way to Push Bits over the Net
17. The PyMailCGI Server
17.1. “Things to Do When Visiting Chicago”
17.2. The PyMailCGI Web Site
17.2.1. Implementation Overview17.2.2. New in This Edition17.2.3. Presentation Overview17.2.4. Running This Chapter’s Examples
17.3. The Root Page
17.3.1. Configuring PyMailCGI
17.4. Sending Mail by SMTP
17.4.1. The Message Composition Page17.4.2. The Send Mail Script17.4.3. Error Pages17.4.4. Common Look-and-Feel17.4.5. Using the Send Mail Script Outside a Browser
17.5. Reading POP Email
17.5.1. The POP Password Page17.5.2. The Mail Selection List Page17.5.3. Passing State Information in URL Link Parameters17.5.4. Security Protocols17.5.4.1. Reading mail with direct URLs17.5.5. The Message View Page17.5.6. Passing State Information in HTML Hidden Input Fields17.5.7. Escaping Mail Text and Passwords in HTML
17.6. Processing Fetched Mail
17.6.1. Reply and Forward17.6.2. Delete17.6.3. Deletions and POP Message Numbers17.6.3.1. Inbox synchronization error potential17.6.3.2. Passing header text in hidden input fields (PyMailCGI_2.1)17.6.3.3. Server-side files for headers17.6.3.4. Delete on load
17.7. Utility Modules
17.7.1. External Components and Configuration17.7.2. POP Mail Interface17.7.3. POP Password Encryption17.7.3.1. Manual data encryption: rotor (defunct)17.7.3.2. Manual data encryption: PyCrypto17.7.3.3. Secure HTTP transmissions17.7.3.4. Secure cookies17.7.3.5. The secret.py module17.7.3.6. Rolling your own encryptor17.7.4. Common Utilities Module
17.8. CGI Script Trade-Offs
17.8.1. Other Approaches
18. Advanced Internet Topics
18.1. “Surfing on the Shoulders of Giants”
18.2. Zope: A Web Application Framework
18.2.1. Zope Overview18.2.1.1. Zope hierarchy model18.2.1.2. Zope scripting18.2.1.3. Zope components18.2.2. Zope Object Publishing18.2.3. A Zope External Method18.2.3.1. Calling through the Web18.2.3.2. Calling from other objects18.2.4. A Simple Zope Interactive Web Site
18.3. HTMLgen: Web Pages from Objects
18.3.1. A Brief HTMLgen Tutorial
18.4. Jython: Python for Java
18.4.1. A Quick Introduction to Jython18.4.2. Why Jython?18.4.3. A Simple Jython Example18.4.4. Interface Automation Tricks18.4.5. Writing Java Applets in Jython18.4.6. Jython Trade-Offs18.4.7. Picking Your Python
18.5. Grail: A Python-Based Web Browser
18.5.1. A Simple Grail Applet Example
18.6. XML Processing Tools
18.6.1. A Brief Introduction to XML Parsing
18.7. Windows Web Scripting Extensions
18.7.1. Active Scripting: Client-Side Embedding18.7.1.1. Active Scripting basics18.7.1.2. Embedding Python in HTML18.7.2. Active Server Pages: Server-Side Embedding18.7.2.1. A short ASP example18.7.3. The COM Connection18.7.3.1. A brief introduction to COM18.7.3.2. Python COM clients18.7.3.3. Python COM servers18.7.3.3.1. Using the Python server from a Python client18.7.3.3.2. Using the Python server from a Visual Basic client18.7.3.3.3. Using the Python server with client-side Active Scripting18.7.3.4. The bigger COM picture: DCOM
18.8. Python Server Pages
18.8.1. PSP in Webware and mod_python
18.9. Rolling Your Own Servers in Python
18.9.1. Standard Library Socket Servers18.9.2. Standard Library Web Servers18.9.3. Third-Party Solutions
18.10. And Other Cool Stuff
V. Tools and Techniques
19. Databases and Persistence
19.1. “Give Me an Order of Persistence, but Hold the Pickles”
19.2. Persistence Options in Python
19.3. DBM Files
19.3.1. Using DBM Files
19.4. Pickled Objects
19.4.1. Using Object Pickling19.4.2. Picking in Action19.4.3. Pickler Protocols and cPickle
19.5. Shelve Files
19.5.1. Using Shelves19.5.2. Storing Built-In Object Types in Shelves19.5.3. Storing Class Instances in Shelves19.5.4. Changing Classes of Objects Stored in Shelves19.5.5. Shelve Constraints19.5.5.1. Keys must be strings19.5.5.2. Objects are unique only within a key19.5.5.3. Updates must treat shelves as fetch-modify-store mappings19.5.5.4. Concurrent updates are not directly supported19.5.5.5. Underlying DBM format portability19.5.6. Pickled Class Constraints19.5.7. Other Shelve Limitations
19.6. The ZODB Object-Oriented Database
19.6.1. A ZODB Tutorial19.6.1.1. Installing ZODB19.6.1.2. The ZEO distributed object server19.6.1.3. Creating a ZODB database19.6.1.4. Fetching and changing19.6.2. Using Classes with ZODB19.6.3. A ZODB People Database19.6.4. ZODB Resources
19.7. SQL Database Interfaces
19.7.1. SQL Interface Overview19.7.2. An SQL Database API Tutorial19.7.2.1. The MySQL system19.7.2.2. Installation19.7.2.3. Getting started19.7.2.4. Making databases and tables19.7.2.5. Adding records19.7.2.6. Running queries19.7.2.7. Running updates19.7.3. Building Record Dictionaries19.7.3.1. Using table descriptions19.7.3.2. Record dictionaries19.7.3.3. Automating with scripts and modules19.7.4. Tying the Pieces Together19.7.5. Loading Database Tables from Files19.7.5.1. Loading with SQL and Python19.7.5.2. Python versus SQL19.7.6. SQL Utility Scripts19.7.6.1. Table load scripts19.7.6.2. Table display script19.7.6.3. Using the scripts19.7.7. SQL Resources
19.8. PyForm: A Persistent Object Viewer
19.8.1. Processing Shelves with Code19.8.2. Adding a Graphical Interface19.8.3. PyForm GUI Implementation19.8.4. PyForm Table Wrappers19.8.5. PyForm Creation and View Utility Scripts19.8.5.1. Creating and browsing custom databases19.8.6. Data as Code19.8.7. Browsing Other Kinds of Objects with PyForm19.8.8. Browsing Other Kinds of Databases with PyForm19.8.9. PyForm Limitations
20. Data Structures
20.1. “Roses Are Red, Violets Are Blue; Lists Are Mutable, and So Is Set Foo”
20.2. Implementing Stacks
20.2.1. A Stack Module20.2.2. A Stack Class20.2.3. Customization: Performance Monitors20.2.4. Optimization: Tuple Tree Stacks20.2.5. Optimization: In-Place List Modifications20.2.6. Timing the Improvements20.2.6.1. Results under Python 1.5.220.2.6.2. Results under Python 2.4
20.3. Implementing Sets
20.3.1. Set Functions20.3.1.1. Supporting multiple operands20.3.2. Set Classes20.3.3. Optimization: Moving Sets to Dictionaries20.3.3.1. Timing the results under Python 2.420.3.3.2. Timing results under Python 1.5.2: version skew20.3.3.3. Using the Python profiler20.3.4. Optimizing fastset by Coding Techniques (or Not)20.3.5. Adding Relational Algebra to Sets (External)
20.4. Subclassing Built-In Types
20.5. Binary Search Trees
20.6. Graph Searching
20.6.1. Moving Graphs to Classes
20.7. Reversing Sequences
20.8. Permuting Sequences
20.9. Sorting Sequences
20.9.1. Adding Comparison Functions
20.10. Data Structures Versus Python Built-Ins
20.11. PyTree: A Generic Tree Object Viewer
20.11.1. Running PyTree20.11.2. PyTree Source Code20.11.2.1. Tree-independent GUI implementation20.11.2.2. Tree wrappers and test widgets20.11.3. PyTree Does Parse Trees Too
21. Text and Language
21.1. “See Jack Hack. Hack, Jack, Hack”
21.2. Strategies for Parsing Text in Python
21.3. String Method Utilities
21.3.1. Templating with Replacements and Formats21.3.2. Parsing with Splits and Joins21.3.3. Summing Columns in a File21.3.4. Parsing and Unparsing Rule Strings21.3.5. More on the holmes Expert System Shell
21.4. Regular Expression Pattern Matching
21.4.1. First Examples21.4.2. Using the re Module21.4.2.1. Module functions21.4.2.2. Compiled pattern objects21.4.2.3. Match objects21.4.2.4. Regular expression patterns21.4.3. Basic Patterns21.4.4. Scanning C Header Files for Patterns21.4.5. A File Pattern Search Utility
21.5. Advanced Language Tools
21.6. Handcoded Parsers
21.6.1. The Expression Grammar21.6.2. The Parser’s Code21.6.3. Adding a Parse Tree Interpreter21.6.4. Parse Tree Structure21.6.5. Exploring Parse Trees with PyTree21.6.6. Parsers Versus Python
21.7. PyCalc: A Calculator Program/Object
21.7.1. A Simple Calculator GUI21.7.1.1. Building the GUI21.7.1.2. Running code strings21.7.1.3. Extending and attaching21.7.2. PyCalc—A Real Calculator GUI21.7.2.1. Running PyCalc21.7.2.2. Evaluating expressions with stacks21.7.2.3. PyCalc source code21.7.2.4. Using PyCalc as a component21.7.2.5. Adding new buttons in new components
VI. Integration
22. Extending Python
22.1. “I Am Lost at C”
22.2. Integration Modes
22.2.1. Presentation Notes
22.3. C Extensions Overview
22.4. A Simple C Extension Module
22.5. Extension Module Details
22.5.1. Compilation and Linking22.5.1.1. Dynamic binding22.5.1.2. Static binding22.5.1.3. Static versus dynamic binding22.5.2. Compiling with the Distutils System22.5.3. Anatomy of a C Extension Module22.5.4. Data Conversions22.5.4.1. Python to C: using Python argument lists22.5.4.2. Python to C: using Python return values22.5.4.3. C to Python: returning values to Python22.5.4.4. Common conversion codes22.5.5. Error Handling22.5.5.1. Raising Python exceptions in C22.5.5.2. Detecting errors that occur in Python22.5.6. Reference Counts22.5.7. Other Extension Tasks: Threads
22.6. The SWIG Integration Code Generator
22.6.1. A Simple SWIG Example22.6.2. SWIG Details
22.7. Wrapping C Environment Calls
22.7.1. Adding Wrapper Classes to Flat Libraries22.7.2. But Don’t Do That Either—SWIG
22.8. A C Extension Module String Stack
22.8.1. But Don’t Do That Either—SWIG
22.9. A C Extension Type String Stack
22.9.1. Anatomy of a C Extension Type22.9.2. Compiling and Running22.9.3. Timing the C Implementations22.9.4. Older Timing Results22.9.5. But Don’t Do That Either—SWIG
22.10. Wrapping C++ Classes with SWIG
22.10.1. A Simple C++ Extension Class22.10.2. Wrapping the C++ Class with SWIG22.10.3. Using the C++ Class in Python22.10.3.1. Using the low-level extension module22.10.3.2. Subclassing the C++ class in Python22.10.3.3. Exploring the wrappers interactively
22.11. Other Extending Tools
23. Embedding Python
23.1. “Add Python. Mix Well. Repeat.”
23.2. C Embedding API Overview
23.2.1. What Is Embedded Code?
23.3. Basic Embedding Techniques
23.3.1. Running Simple Code Strings23.3.1.1. Compiling and running23.3.2. Running Code Strings with Results and Namespaces23.3.3. Calling Python Objects23.3.4. Running Strings in Dictionaries23.3.5. Precompiling Strings to Bytecode
23.4. Registering Callback Handler Objects
23.5. Using Python Classes in C
23.6. A High-Level Embedding API: ppembed
23.6.1. Running Objects with ppembed23.6.2. Running Code Strings with ppembed23.6.3. Running Customizable Validations23.6.3.1. Running function-based validations23.6.3.2. Other validation components23.6.4. ppembed Implementation23.6.5. Other Integration Examples (External)
23.7. Other Integration Topics
23.7.1. Jython: Java Integration23.7.2. IronPython: C#/.NET Integration23.7.3. COM Integration on Windows23.7.4. CORBA Integration23.7.5. Other Languages23.7.6. Network-Based Integration Protocols23.7.7. Integration Versus Optimization23.7.7.1. Framework roles23.7.7.2. Extension module roles23.7.7.3. Picking an integration technology
VIII. The End
24. Conclusion: Python and the Development Cycle
24.1. “That’s the End of the Book, Now Here’s the Meaning of Life”
24.2. “Something’s Wrong with the Way We Program Computers”
24.3. The “Gilligan Factor”
24.4. Doing the Right Thing
24.4.1. The Static Language Build Cycle24.4.2. Artificial Complexities24.4.3. One Language Does Not Fit All
24.5. Enter Python
24.6. But What About That Bottleneck?
24.6.1. Python Provides Immediate Turnaround24.6.2. Python Is “Executable Pseudocode”24.6.3. Python Is OOP Done Right24.6.4. Python Fosters Hybrid Applications
24.7. On Sinking the Titanic
24.8. So What’s “Python: The Sequel”?
24.9. In the Final Analysis . . .
24.10. Postscript to the Second Edition (2000)
24.10.1. Integration Isn’t Everything24.10.2. The End of the Java Wars24.10.3. We’re Not Off That Island Yet
24.11. Postscript to the Third Edition (2006)
24.11.1. Proof of Concept24.11.2. Integration Today24.11.3. Quality Counts
Index
Colophon
Copyright

Content preview from Programming Python, 3rd Edition

Part V. Tools and Techniques

This part of the book presents a collection of additional Python application topics. Most of the tools presented along the way can be used in a wide variety of application domains. You’ll find the following chapters here:

Chapter 19, Databases and Persistence: This chapter covers commonly used and advanced Python techniques for storing information between program executions—DBM files, object pickling (serialization), object shelves, the ZODB object database, and Python’s SQL database interfaces. MySQL is used for the SQL examples, but the API is portable to other systems.
Chapter 20, Data Structures: This chapter explores techniques for implementing more advanced data structures in Python—stacks, sets, binary search trees, graphs, and the like. In Python, these take the form of object implementations.
Chapter 21, Text and Language: This chapter addresses Python tools and techniques for parsing text-based information—string splits and joins, regular expression matching, recursive descent parsing, and more advanced language-based topics.

This is the last pure Python part of the book, and it makes heavy use of tools presented earlier in the text, especially the Tkinter GUI library. For instance, a tree browser (PyTree) is used to illustrate various object structures, a form browser (PyForm) helps make database concepts more concrete, and a calculator GUI (PyCalc) serves to demonstrate language processing and code reuse concepts.

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 0596009259Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Programming Python, 3rd Edition

by Mark Lutz

Part V. Tools and Techniques

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

More than 5,000 organizations count on O’Reilly

Julian F.

Addison B.

Amir M.

Mark W.

You might also like

Learning Python, 3rd Edition

Learning Python, 5th Edition

Python Object-Oriented Programming - Fourth Edition

Python for Programmers

Publisher Resources