book

Programming Python, 4th Edition

Name: Programming Python, 4th Edition
Author: Mark Lutz
ISBN: 9780596158101

by Mark Lutz

December 2010

Beginner

1626 pages

54h 17m

English

O'Reilly Media, Inc.

Read now

Unlock full access

A Note Regarding Supplemental Files
Preface
“And Now for Something Completely Different…”About This BookThis Book’s EcosystemWhat This Book Is NotAbout This Fourth EditionSpecific Changes in This EditionWhat’s Left, Then?Python 3.X Impacts on This BookSpecific 3.X ChangesLanguage Versus Library: UnicodePython 3.1 Limitations: Email, CGIUsing Book ExamplesWhere to Look for Examples and UpdatesExample PortabilityDemo LaunchersCode Reuse PoliciesContacting O’ReillyConventions Used in This BookAcknowledgments
I. The Beginning
1. A Sneak Preview
“Programming Python: The Short Story”The TaskStep 1: Representing RecordsUsing ListsStart-up pointersA database listField labelsUsing DictionariesOther ways to make dictionariesLists of dictionariesNested structuresDictionaries of dictionariesStep 2: Storing Records PersistentlyUsing Formatted FilesTest data scriptFile name conventionsScript start-up pointersData format scriptUtility scriptsUsing Pickle FilesUsing Per-Record Pickle FilesUsing ShelvesStep 3: Stepping Up to OOPUsing ClassesAdding BehaviorAdding InheritanceRefactoring CodeAugmenting methodsDisplay formatConstructor customizationAlternative classesAdding PersistenceOther Database OptionsStep 4: Adding Console InteractionA Console Shelve InterfaceStep 5: Adding a GUIGUI BasicsUsing OOP for GUIsGetting Input from a UserA GUI Shelve InterfaceCoding the GUIUsing the GUIFuture directionsStep 6: Adding a Web InterfaceCGI BasicsGUIs versus the WebRunning a Web ServerUsing Query Strings and urllibFormatting Reply TextA Web-Based Shelve InterfaceCoding the websiteDirectories, string formatting, and securityUsing the websiteFuture directionsThe End of the Demo
II. System Programming
2. System Tools
“The os.path to Knowledge”Why Python Here?The Next Five ChaptersSystem Scripting OverviewPython System ModulesModule Documentation SourcesPaging Documentation StringsA Custom Paging ScriptString Method BasicsOther String Concepts in Python 3.X: Unicode and bytesFile Operation BasicsUsing Programs in Two WaysPython Library ManualsCommercially Published ReferencesIntroducing the sys ModulePlatforms and VersionsThe Module Search PathThe Loaded Modules TableException DetailsOther sys Module ExportsIntroducing the os ModuleTools in the os ModuleAdministrative ToolsPortability ConstantsCommon os.path ToolsRunning Shell Commands from ScriptsWhat’s a shell command?Running shell commandsCommunicating with shell commandsThe subprocess module alternativeShell command limitationsOther os Module Exports
3. Script Execution Context
“I’d Like to Have an Argument, Please”Current Working DirectoryCWD, Files, and Import PathsCWD and Command LinesCommand-Line ArgumentsParsing Command-Line ArgumentsShell Environment VariablesFetching Shell VariablesChanging Shell VariablesShell Variable Fine Points: Parents, putenv, and getenvStandard StreamsRedirecting Streams to Files and ProgramsChaining programs with pipesCoding alternatives for adders and sortersRedirected Streams and User InteractionRedirecting Streams to Python ObjectsThe io.StringIO and io.BytesIO Utility ClassesCapturing the stderr StreamRedirection Syntax in Print CallsOther Redirection Options: os.popen and subprocess RevisitedRedirecting input or output with os.popenRedirecting input and output with subprocess
4. File and Directory Tools
“Erase Your Hard Drive in Five Easy Steps!”File ToolsThe File Object Model in Python 3.XUsing Built-in File ObjectsOutput filesOpeningWritingClosingEnsuring file closure: Exception handlers and context managersInput filesReading lines with file iteratorsOther open optionsBinary and Text FilesUnicode encodings for text filesEnd-of-line translations for text filesParsing packed binary data with the struct moduleRandom access filesLower-Level File Tools in the os ModuleUsing os.open filesos.open mode flagsWrapping descriptors in file objectsOther os module file toolsFile ScannersFile filtersDirectory ToolsWalking One DirectoryRunning shell listing commands with os.popenThe glob moduleThe os.listdir callSplitting and joining listing resultsWalking Directory TreesThe os.walk visitorRecursive os.listdir traversalsHandling Unicode Filenames in 3.X: listdir, walk, globUnicode policies: File content versus file names
5. Parallel System Tools
“Telling the Monkeys What to Do”Forking ProcessesThe fork/exec Combinationos.exec call formatsSpawned child programThreadsThe _thread ModuleBasic usageOther ways to code threads with _threadRunning multiple threadsSynchronizing access to shared objects and namesWaiting for spawned thread exitsCoding alternatives: busy loops, arguments, and context managersThe threading ModuleOther ways to code threads with threadingSynchronizing access to shared objects and names revisitedThe queue ModuleArguments versus globalsProgram exit with child threadsRunning the scriptPreview: GUIs and ThreadsMore on the Global Interpreter LockThe thread switch intervalAtomic operationsC API thread considerationsA process-based alternative: multiprocessing (ahead)Program Exitssys Module Exitsos Module ExitsShell Command Exit Status CodesExit status with os.system and os.popenOutput stream buffering: A first lookExit status with subprocessProcess Exit Status and Shared StateThread Exits and Shared StateInterprocess CommunicationAnonymous PipesAnonymous pipe basicsWrapping pipe descriptors in file objectsAnonymous pipes and threadsBidirectional IPC with anonymous pipesOutput stream buffering revisited: Deadlocks and flushesNamed Pipes (Fifos)Named pipe basicsNamed pipe use casesSockets: A First LookSocket basicsSockets and independent programsSocket use casesSignalsThe multiprocessing ModuleWhy multiprocessing?The Basics: Processes and LocksImplementation and usage rulesIPC Tools: Pipes, Shared Memory, and Queuesmultiprocessing pipesShared memory and globalsQueues and subclassingStarting Independent ProgramsAnd Much MoreAnd a little less…Why multiprocessing? The ConclusionOther Ways to Start ProgramsThe os.spawn CallsThe os.startfile call on WindowsUsing the DOS start commandUsing start in Python scriptsThe os.startfile callA Portable Program-Launch FrameworkOther System Tools Coverage
6. Complete System Programs
“The Greps of Wrath”A Quick Game of “Find the Biggest Python File”Scanning the Standard Library DirectoryScanning the Standard Library TreeScanning the Module Search PathScanning the Entire MachinePrinting Unicode FilenamesSplitting and Joining FilesSplitting Files PortablyJoining Files PortablyUsage VariationsGenerating Redirection Web PagesPage Template FilePage Generator ScriptA Regression Test ScriptRunning the Test DriverCopying Directory TreesComparing Directory TreesFinding Directory DifferencesFinding Tree DifferencesRunning the ScriptVerifying BackupsReporting Differences and Other IdeasSearching Directory TreesGreps and Globs and FindsRolling Your Own find ModuleThe fnmatch moduleCleaning Up Bytecode FilesA Python Tree SearcherVisitor: Walking Directories “++”Editing Files in Directory Trees (Visitor)Global Replacements in Directory Trees (Visitor)Counting Source Code Lines (Visitor)Recoding Copies with Classes (Visitor)Other Visitor Examples (External)Playing Media FilesThe Python webbrowser ModuleThe Python mimetypes ModuleUsing mimetypes guesses for SearchVisitorRunning the ScriptAutomated Program Launchers (External)

III. GUI Programming
7. Graphical User Interfaces
“Here’s Looking at You, Kid”GUI Programming TopicsRunning the ExamplesPython GUI Development Optionstkinter Overviewtkinter Pragmaticstkinter Documentationtkinter Extensionstkinter StructureImplementation structureProgramming structureClimbing the GUI Learning Curve“Hello World” in Four Lines (or Less)tkinter Coding BasicsMaking WidgetsGeometry ManagersRunning GUI ProgramsAvoiding DOS consoles on Windowstkinter Coding AlternativesWidget Resizing BasicsConfiguring Widget Options and Window TitlesOne More for Old Times’ SakePacking Widgets Without Saving ThemAdding Buttons and CallbacksWidget Resizing Revisited: ExpansionAdding User-Defined Callback HandlersLambda Callback HandlersDeferring Calls with Lambdas and Object ReferencesCallback Scope IssuesArguments versus globalsPassing in enclosing scope values with default argumentsPassing in enclosing scope values with automatic referencesBut you must still sometimes use defaults instead of enclosing scopesBound Method Callback HandlersCallable Class Object Callback HandlersOther tkinter Callback ProtocolsBinding EventsAdding Multiple WidgetsWidget Resizing Revisited: ClippingAttaching Widgets to FramesLayout: Packing Order and Side AttachmentsThe Packer’s Expand and Fill RevisitedUsing Anchor to Position Instead of StretchCustomizing Widgets with ClassesStandardizing Behavior and AppearanceCommon behaviorCommon appearanceReusable GUI Components with ClassesAttaching Class ComponentsExtending Class ComponentsStandalone Container ClassesThe End of the TutorialPython/tkinter for Tcl/Tk Converts
8. A tkinter Tour, Part 1
“Widgets and Gadgets and GUIs, Oh My!”This Chapter’s TopicsConfiguring Widget AppearanceTop-Level WindowsToplevel and Tk WidgetsTop-Level Window ProtocolsDialogsStandard (Common) DialogsA “smart” and reusable Quit buttonA dialog demo launcher barPrinting dialog results and passing callback data with lambdasLetting users select colors on the flyOther standard dialog callsThe Old-Style Dialog ModuleCustom DialogsMaking custom dialogs modalOther ways to be modalBinding EventsOther bind EventsMore on <Destroy> events and the quit and destroy methodsMessage and EntryMessageEntryProgramming Entry widgetsLaying Out Input FormsGoing modal againtkinter “Variables” and Form Layout AlternativesCheckbutton, Radiobutton, and ScaleCheckbuttonsCheck buttons and variablesRadio ButtonsRadio buttons and variablesRadio buttons without variablesHold onto your variables!Scales (Sliders)Scales and variablesRunning GUI Code Three WaysAttaching FramesImporting by name stringConfiguring at construction timeIndependent WindowsRunning ProgramsLaunching GUIs as programs other ways: multiprocessingCross-program communicationCoding for reusabilityImagesFun with Buttons and PicturesViewing and Processing Images with PILPIL BasicsDisplaying Other Image Types with PILDisplaying all images in a directoryCreating Image Thumbnails with PILPerformance: Saving thumbnail filesLayout options: GriddingLayout options: Fixed-size buttonsScrolling and canvases (ahead)
9. A tkinter Tour, Part 2
“On Today’s Menu: Spam, Spam, and Spam”MenusTop-Level Window MenusFrame- and Menubutton-Based MenusUsing Menubuttons and OptionmenusWindows with Both Menus and ToolbarsUsing images in toolbars, tooAutomating menu constructionListboxes and ScrollbarsProgramming ListboxesProgramming Scroll BarsPacking Scroll BarsTextProgramming the Text WidgetText is a Python stringString positionsText indexesText marksText tagsAdding Text-Editing OperationsUsing the clipboardComposition versus inheritanceIt’s called “Simple” for a reason: PyEdit (ahead)Unicode and the Text WidgetString types in the Text widgetUnicode text in stringsUnicode text in filesUnicode and the Text widgetThe problem with treating text as bytesOther binary mode considerationsSupporting Unicode in PyEdit (ahead)Advanced Text and Tag OperationsCanvasBasic Canvas OperationsProgramming the Canvas WidgetCoordinatesObject constructionObject identifiers and operationsCanvas object tagsScrolling CanvasesScrollable Canvases and Image ThumbnailsScrolling images too: PyPhoto (ahead)Using Canvas EventsBinding events on specific itemsGridsWhy Grids?Grid Basics: Input Forms RevisitedComparing grid and packCombining grid and packMaking Gridded Widgets ExpandableResizing in gridsSpanning columns and rowsLaying Out Larger Tables with gridTime Tools, Threads, and AnimationUsing Threads with tkinter GUIsUsing the after MethodHiding and redrawing widgets and windowsSimple Animation TechniquesUsing time.sleep loopsUsing widget.after eventsUsing multiple time.sleep loop threadsOther Animation TopicsOther animation effectsThreads and animationGraphics and gaming toolkitsThe End of the TourOther Widgets and Options
10. GUI Coding Techniques
“Building a Better Mousetrap”GuiMixin: Common Tool Mixin ClassesWidget Builder FunctionsMixin Utility ClassesGuiMaker: Automating Menus and ToolbarsSubclass ProtocolsGuiMaker ClassesGuiMaker Self-TestBigGui: A Client Demo ProgramShellGui: GUIs for Command-Line ToolsA Generic Shell-Tools DisplayApplication-Specific Tool Set ClassesAdding GUI Frontends to Command LinesNon-GUI scriptsGUI input dialogsRoom for improvementGuiStreams: Redirecting Streams to WidgetsUsing Redirection for the Packing ScriptsReloading Callback Handlers DynamicallyWrapping Up Top-Level Window InterfacesGUIs, Threads, and QueuesPlacing Data on QueuesRecoding with classes and bound methodsThread exits in GUIsPlacing Callbacks on QueuesPassing bound method callbacks on queuesMore Ways to Add GUIs to Non-GUI CodePopping Up GUI Windows on DemandAdding a GUI As a Separate Program: Sockets (A Second Look)Adding a GUI As a Separate Program: Command PipesThe specter of blocking input callsUpdating GUIs within threads…and other nonsolutionsAvoiding blocking input calls with non-GUI threadsSockets and pipes: Compare and contrastOther uses for threaded pipe GUIsThe PyDemos and PyGadgets LaunchersPyDemos Launcher Bar (Mostly External)PyGadgets Launcher Bar
11. Complete GUI Programs
“Python, Open Source, and Camaros”Examples in Other ChaptersThis Chapter’s StrategyPyEdit: A Text Editor Program/ObjectRunning PyEditMenus and toolbarsDialogsRunning program codeMultiple windowsOther PyEdit examples and screenshots in this bookPyEdit Changes in Version 2.0 (Third Edition)Font dialogUndo, redo, and modified testsConfiguration modulePyEdit Changes in Version 2.1 (Fourth Edition)Modal dialog state fixCross-process change tests on QuitNew Grep dialog: Threaded and Unicode-aware file tree searchGrep threading modelGrep Unicode modelUpdate for initial positioningImprovements for running codeUnicode (Internationalized) text supportUnicode file and display modelUnicode options and choicesMore on Quit checks: The <Destroy> event revisitedPyEdit Source CodeUser configurations fileWindows (and other) launch filesMain implementation filePyPhoto: An Image Viewer and ResizerRunning PyPhotoPyPhoto Source CodePyView: An Image and Notes SlideshowRunning PyViewEmbedding PyEdit in PyViewPyView Source CodePyDraw: Painting and Moving GraphicsRunning PyDrawPyDraw Source CodePyClock: An Analog/Digital Clock WidgetA Quick Geometry LessonRunning PyClockPyClock Source CodePyToe: A Tic-Tac-Toe Game WidgetRunning PyToePyToe Source Code (External)Where to Go from Here
IV. Internet Programming
12. Network Scripting
“Tune In, Log On, and Drop Out”Internet Scripting TopicsWhat we will coverWhat we won’t coverOther themes in this part of the bookRunning Examples in This Part of the BookPython Internet Development OptionsPlumbing the InternetThe Socket LayerMachine identifiersThe Protocol LayerPort number rulesClients and serversProtocol structuresPython’s Internet Library ModulesSocket ProgrammingSocket BasicsServer socket callsTransferring byte strings and objectsClient socket callsRunning Socket Programs LocallyRunning Socket Programs RemotelySocket pragmaticsSpawning Clients in ParallelPreview: Denied client connectionsTalking to Reserved PortsBinding reserved port serversHandling Multiple ClientsForking ServersRunning the forking serverOther run modes: Local servers with Cygwin and remote clientsForked processes and socketsExiting from childrenKilling the zombies: Don’t fear the reaper!Preventing zombies with signal handlers on LinuxWhy multiprocessing doesn’t help with socket server portabilityThreading ServersStandard Library Server ClassesMultiplexing Servers with selectA select-based echo serverRunning the select serverSummary: Choosing a Server SchemeMaking Sockets Look Like Files and StreamsA Stream Redirection UtilityText-mode files and buffered output streamsStream requirementsLine bufferingSolutionsBuffering in other contexts: Command pipes revisitedSockets versus command pipesA Simple Python File ServerRunning the File Server and ClientsAdding a User-Interface FrontendUsing row frames and command linesUsing grids and function callsUsing a reusable form-layout class
13. Client-Side Scripting
“Socket to Me!”FTP: Transferring Files over the NetTransferring Files with ftplibUsing urllib to Download FilesFTP get and put UtilitiesDownload utilityUpload utilityPlaying the Monty Python theme songAdding a User InterfaceTransferring Directories with ftplibDownloading Site DirectoriesUploading Site DirectoriesRefactoring Uploads and Downloads for ReuseRefactoring with functionsRefactoring with classesTransferring Directory Trees with ftplibUploading Local TreesDeleting Remote TreesDownloading Remote TreesProcessing Internet EmailUnicode in Python 3.X and Email ToolsPOP: Fetching EmailMail Configuration ModulePOP Mail Reader ScriptFetching MessagesFetching Email at the Interactive PromptSMTP: Sending EmailSMTP Mail Sender ScriptSending MessagesVerifying receiptManipulating both From and ToSending Email at the Interactive Promptemail: Parsing and Composing Mail ContentMessage ObjectsBasic email Package Interfaces in ActionHandling multipart messagesUnicode, Internationalization, and the Python 3.1 email PackageParser decoding requirementText payload encodings: Handling mixed type resultsText payload encodings: Using header information to decodeMessage header encodings: email package supportMessage address header encodings and parsing, and header creationWorkaround: Message text generation for binary attachment payloads is brokenWorkaround: Message composition for non-ASCII text parts is brokenSummary: Solutions and workaroundsA Console-Based Email ClientRunning the pymail Console ClientThe mailtools Utility PackageInitialization FileMailTool ClassMailSender ClassUnicode issues for attachments, save files, and headersMailFetcher ClassGeneral usageUnicode decoding for full mail text on fetchesInbox synchronization toolsMailParser ClassUnicode decoding for text part payloads and message headersSelf-Test ScriptRunning the self-testUpdating the pymail Console ClientRunning the pymail2 console clientNNTP: Accessing NewsgroupsHTTP: Accessing WebsitesThe urllib Package RevisitedOther urllib InterfacesInvoking programs and escaping textOther Client-Side Scripting Options
14. The PyMailGUI Client
“Use the Source, Luke”Source Code Modules and SizeCode sizeCode StructureWhy PyMailGUI?Running PyMailGUIPresentation StrategyMajor PyMailGUI ChangesNew in Version 2.1 and 2.0 (Third Edition)New in Version 3.0 (Fourth Edition)Version 3.0 Unicode support policiesA PyMailGUI DemoGetting StartedLoading MailThreading ModelThreading model implementationLoad Server InterfaceOffline Processing with Save and OpenSending Email and AttachmentsViewing Email and AttachmentsEmail Replies and Forwards and Recipient OptionsDeleting EmailPOP Message Numbers and SynchronizationHandling HTML Content in EmailMail Content Internationalization SupportAlternative Configurations and AccountsMultiple Windows and Status MessagesPyMailGUI ImplementationPyMailGUI: The Main ModuleSharedNames: Program-Wide GlobalsListWindows: Message List WindowsViewWindows: Message View Windowsmessagecache: Message Cache Managerpopuputil: General-Purpose GUI Pop Upswraplines: Line Split Toolshtml2text: Extracting Text from HTML (Prototype, Preview)mailconfig: User ConfigurationstextConfig: Customizing Pop-Up PyEdit WindowsPyMailGUIHelp: User Help Text and Displayaltconfigs: Configuring for Multiple AccountsIdeas for Improvement
15. Server-Side Scripting
“Oh, What a Tangled Web We Weave”What’s a Server-Side CGI Script?The Script Behind the CurtainWriting CGI Scripts in PythonRunning Server-Side ExamplesWeb Server OptionsRunning a Local Web ServerThe Server-Side Examples Root PageViewing Server-Side Examples and OutputClimbing the CGI Learning CurveA First Web PageHTML basicsInternet addresses (URLs)Using minimal URLsHTML file permission constraintsA First CGI ScriptInstalling CGI scriptsFinding Python on remote serversAdding Pictures and Generating TablesTable tagsAdding User InteractionSubmission pageMore on form tagsResponse scriptPassing parameters in URLsTesting outside browsers with the module urllib.requestUsing Tables to Lay Out FormsConverting strings in CGI scriptsDebugging CGI scriptsAdding Common Input DevicesChanging Input LayoutsKeeping display and logic separatePassing Parameters in Hardcoded URLsPassing Parameters in Hidden Form FieldsSaving State Information in CGI ScriptsURL Query ParametersHidden Form Input FieldsHTTP “Cookies”Creating a cookieReceiving a cookieUsing cookies in CGI scriptsHandling cookies with the urllib.request moduleServer-Side DatabasesExtensions to the CGI ModelCombining TechniquesThe Hello World SelectorChecking for Missing and Invalid InputsRefactoring Code for MaintainabilityStep 1: Sharing Objects Between Pages—A New Input FormStep 2: A Reusable Form Mock-Up UtilityStep 3: Putting It All Together—A New Reply ScriptMore on HTML and URL EscapesURL Escape Code ConventionsPython HTML and URL Escape ToolsEscaping HTML CodeEscaping URLsEscaping URLs Embedded in HTML CodeHTML and URL conflicts: &Avoiding conflictsTransferring Files to Clients and ServersDisplaying Arbitrary Server Files on the ClientHandling private files and errorsUploading Client Files to the ServerHandling client path formatsMore Than One Way to Push Bits over the Net
16. The PyMailCGI Server
“Things to Do When Visiting Chicago”The PyMailCGI WebsiteImplementation OverviewNew in This Fourth Edition (Version 3.0)New in the Prior Edition (Version 2.0)Presentation OverviewRunning This Chapter’s ExamplesThe Root PageConfiguring PyMailCGISending Mail by SMTPThe Message Composition PageThe Send Mail ScriptError PagesCommon Look-and-FeelUsing the Send Mail Script Outside a BrowserReading POP EmailThe POP Password PageThe Mail Selection List PagePassing State Information in URL Link ParametersSecurity ProtocolsReading mail with direct URLsThe Message View PagePassing State Information in HTML Hidden Input FieldsEscaping Mail Text and Passwords in HTMLProcessing Fetched MailReply and ForwardDeleteDeletions and POP Message NumbersInbox synchronization error potentialAlternative: Passing header text in hidden input fields (PyMailCGI_2.1)Alternative: Server-side files for headersAlternative: Delete on loadUtility ModulesExternal Components and ConfigurationPOP Mail InterfacePOP Password EncryptionManual data encryption: rotor (defunct)Manual data encryption: PyCryptoHTTPS: Secure HTTP transmissionsSecure cookiesThe secret.py moduleRolling your own encryptorCommon Utilities ModuleWeb Scripting Trade-OffsPyMailCGI Versus PyMailGUIThe Web Versus the DesktopOther Approaches
V. Tools and Techniques
17. Databases and Persistence
“Give Me an Order of Persistence, but Hold the Pickles”Persistence Options in PythonDBM FilesUsing DBM FilesDBM Details: Files, Portability, and ClosePickled ObjectsUsing Object PicklingPickling in ActionPickle Details: Protocols, Binary Modes, and _pickleShelve FilesUsing ShelvesStoring Built-in Object Types in ShelvesStoring Class Instances in ShelvesChanging Classes of Objects Stored in ShelvesShelve ConstraintsKeys must be strings (and str)Objects are unique only within a keyUpdates must treat shelves as fetch-modify-store mappingsConcurrent updates are not directly supportedUnderlying DBM format portabilityPickled Class ConstraintsOther Shelve LimitationsThe ZODB Object-Oriented DatabaseThe Mostly Missing ZODB TutorialSQL Database InterfacesSQL Interface OverviewAn SQL Database API Tutorial with SQLiteGetting startedMaking databases and tablesAdding recordsRunning queriesRunning updatesBuilding Record DictionariesUsing table descriptionsRecord dictionaries constructionAutomating with scripts and modulesTying the Pieces TogetherLoading Database Tables from FilesLoading with SQL and PythonPython versus SQLSQL Utility ScriptsTable load scriptsTable display scriptUsing the scriptsSQL ResourcesORMs: Object Relational MappersPyForm: A Persistent Object Viewer (External)
18. Data Structures
“Roses Are Red, Violets Are Blue; Lists Are Mutable, and So Is Set Foo”Implementing StacksBuilt-in OptionsA Stack ModuleA Stack ClassCustomization: Performance MonitorsOptimization: Tuple Tree StacksOptimization: In-Place List ModificationsTiming the ImprovementsResults under Python 3.1More on performance analysisImplementing SetsBuilt-in OptionsSet FunctionsSupporting multiple operandsSet ClassesOptimization: Moving Sets to DictionariesTiming the results under Python 3.1Adding Relational Algebra to Sets (External)Subclassing Built-in TypesBinary Search TreesBuilt-in OptionsImplementing Binary TreesTrees with Both Keys and ValuesGraph SearchingImplementing Graph SearchMoving Graphs to ClassesPermuting SequencesReversing and Sorting SequencesImplementing ReversalsImplementing SortsAdding comparison functionsData Structures Versus Built-ins: The ConclusionPyTree: A Generic Tree Object Viewer
19. Text and Language
“See Jack Hack. Hack, Jack, Hack”Strategies for Processing Text in PythonString Method UtilitiesTemplating with Replacements and FormatsParsing with Splits and JoinsSumming Columns in a FileSumming with zips and comprehensionsSumming with dictionariesParsing and Unparsing Rule StringsMore on the holmes expert system shellRegular Expression Pattern MatchingThe re ModuleFirst ExamplesString Operations Versus PatternsUsing the re ModuleModule functionsCompiled pattern objectsMatch objectsRegular expression patternsMore Pattern ExamplesScanning C Header Files for PatternsXML and HTML ParsingXML Parsing in ActionRegular expression parsingSAX parsingDOM parsingElementTree parsingOther XML topicsHTML Parsing in ActionHandling HTML entity references (revisited)Extracting plain text from HTML (revisited)Advanced Language ToolsCustom Language ParsersThe Expression GrammarThe Parser’s CodeAdding a Parse Tree InterpreterParse Tree StructureExploring Parse Trees with the PyTree GUIParsers Versus PythonPyCalc: A Calculator Program/ObjectA Simple Calculator GUIBuilding the GUIRunning code stringsExtending and attachingPyCalc—A “Real” Calculator GUIRunning PyCalcEvaluating expressions with stacksPyCalc source codeUsing PyCalc as a componentAdding new buttons in new components
20. Python/C Integration
“I Am Lost at C”Extending and EmbeddingExtending Python in C: OverviewA Simple C Extension ModuleThe SWIG Integration Code GeneratorA Simple SWIG ExampleWrapping C Environment CallsAdding Wrapper Classes to Flat LibrariesWrapping C Environment Calls with SWIGWrapping C++ Classes with SWIGA Simple C++ Extension ClassWrapping the C++ Class with SWIGUsing the C++ Class in PythonUsing the low-level extension moduleSubclassing the C++ class in PythonExploring the wrappers interactivelyOther Extending ToolsEmbedding Python in C: OverviewThe C Embedding APIWhat Is Embedded Code?Basic Embedding TechniquesRunning Simple Code StringsCompiling and runningRunning Code Strings with Results and NamespacesCalling Python ObjectsRunning Strings in DictionariesPrecompiling Strings to BytecodeRegistering Callback Handler ObjectsRegistration ImplementationUsing Python Classes in COther Integration Topics
VI. The End
21. Conclusion: Python and the Development Cycle
“That’s the End of the Book, Now Here’s the Meaning of Life”“Something’s Wrong with the Way We Program Computers”The “Gilligan Factor”Doing the Right ThingThe Static Language Build CycleArtificial ComplexitiesOne Language Does Not Fit AllEnter PythonBut What About That Bottleneck?Python Provides Immediate TurnaroundPython Is “Executable Pseudocode”Python Is OOP Done RightPython Fosters Hybrid ApplicationsOn Sinking the Titanic“So What’s Python?”: The SequelIn the Final Analysis…
Index
About the Author
Colophon
Copyright

Content preview from Programming Python, 4th Edition

Part V. Tools and Techniques

This part of the book presents a collection of additional Python application topics. Most of the tools presented along the way can be used in a wide variety of application domains. You’ll find the following chapters here:

Chapter 17: This chapter covers commonly used and advanced Python techniques for storing information between program executions—DBM files, object pickling, object shelves, and Python’s SQL database API—and briefly introduces full-blown OODBs such as ZODB, as well as ORMs such as SQLObject and SQLAlchemy. The Python standard library’s SQLite support is used for the SQL examples, but the API is portable to enterprise-level systems such as MySQL.
Chapter 18: This chapter explores techniques for implementing more advanced data structures in Python—stacks, sets, binary search trees, graphs, and the like. In Python, these take the form of object implementations.
Chapter 19: This chapter addresses Python tools and techniques for parsing text-based information—string splits and joins, regular expression matching, XML parsing, recursive descent parsing, and more advanced language-based topics.
Chapter 20: This chapter introduces integration techniques—both extending Python with compiled libraries and embedding Python code in other applications. While the main focus here is on linking Python with compiled C code, we’ll also investigate integration with Java, .NET, and more. This chapter assumes that you know how to read C programs, and it is intended mostly ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781449398712Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Programming Python, 4th Edition

by Mark Lutz

Part V. Tools and Techniques

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.