book

Learning Python, 2nd Edition

by Mark Lutz, David Ascher

December 2003

Beginner

620 pages

17h 3m

English

O'Reilly Media, Inc.

Read now

Unlock full access

About This Second EditionPrerequisitesThis Book’s ScopeThis Book’s Style and StructureCore LanguageOuter LayersAppendixesBook UpdatesFont ConventionsAbout the Programs in This BookUsing Code ExamplesHow to Contact UsAcknowledgmentsMark Also Says:David Also Says:
1.1. Why Do People Use Python?1.1.1. Software Quality1.1.2. Developer Productivity1.2. Is Python a Scripting Language?1.3. Okay, But What’s the Downside?1.4. Who Uses Python Today?1.5. What Can I Do with Python?1.5.1. Systems Programming1.5.2. GUIs1.5.3. Internet Scripting1.5.4. Component Integration1.5.5. Database Programming1.5.6. Rapid Prototyping1.5.7. Numeric Programming1.5.8. Gaming, Images, AI, XML, and More1.6. What Are Python’s Technical Strengths?1.6.1. It’s Object-Oriented1.6.2. It’s Free1.6.3. It’s Portable1.6.4. It’s Powerful1.6.5. It’s Mixable1.6.6. It’s Easy to Use1.6.7. It’s Easy to Learn1.7. How Does Python Stack Up to Language X?
2.1. Introducing the Python Interpreter2.2. Program Execution2.2.1. The Programmer’s View2.2.2. Python’s View2.2.2.1. Byte code compilation2.2.2.2. Python Virtual Machine (PVM)2.2.2.3. Performance implications2.2.2.4. Development implications2.3. Execution Model Variations2.3.1. Python Implementation Alternatives2.3.1.1. CPython2.3.1.2. Jython2.3.1.3. Python.NET2.3.2. The Psyco Just-in-Time Compiler2.3.3. Frozen Binaries2.3.4. Future Possibilities?
3.1. Interactive Coding3.1.1. Using the Interactive Prompt3.2. System Command Lines and Files3.2.1. Using Command Lines and Files3.2.2. Unix Executable Scripts (#!)3.3. Clicking Windows File Icons3.3.1. Clicking Icons on Windows3.3.2. The raw_input Trick3.3.3. Other Icon Click Limitations3.4. Module Imports and Reloads3.4.1. The Grander Module Story: Attributes3.4.2. Import and Reload Usage Notes3.5. The IDLE User Interface3.5.1. IDLE Basics3.5.2. Using IDLE3.6. Other IDEs3.7. Embedding Calls3.8. Frozen Binary Executables3.9. Text Editor Launch Options3.10. Other Launch Options3.11. Future Possibilities?3.12. Which Option Should I Use?3.13. Part I Exercises
4.1. Python Program Structure4.2. Why Use Built-in Types?4.3. Numbers4.3.1. Number Literals4.3.2. Built-in Tools and Extensions4.4. Python Expression Operators4.4.1. Mixed Operators: Operator Precedence4.4.2. Parentheses Group Subexpressions4.4.3. Mixed Types: Converted Up4.4.4. Preview: Operator Overloading4.5. Numbers in Action4.5.1. Basic Operations and Variables4.5.2. Numeric Representation4.5.3. Division: Classic, Floor, and True4.5.4. B itwise Operations4.5.5. L ong Integers4.5.6. Complex Numbers4.5.7. Hexadecimal and Octal Notation4.5.8. Other Numeric Tools4.6. The Dynamic Typing Interlude4.6.1. How Assignments Work4.6.2. References and Changeable Objects4.6.3. References and Garbage Collection
5.1. String Literals5.1.1. Single- and Double-Quoted Strings Are the Same5.1.2. Escape Sequences Code Special Bytes5.1.3. Raw Strings Suppress Escapes5.1.4. Triple Quotes Code Multiline Block Strings5.1.5. Unicode Strings Encode Larger Character Sets5.2. Strings in Action5.2.1. Basic Operations5.2.2. Indexing and Slicing5.2.2.1. I ndexing (S[i]) fetches components at offsets5.2.2.2. S licing (S[i:j]) extracts contiguous sections of a sequence5.2.3. String Conversion Tools5.2.4. C hanging Strings5.3. String Formatting5.3.1. Advanced String Formatting5.4. String Methods5.4.1. String Method Examples: Changing Strings5.4.2. String Method Examples: Parsing Text5.4.3. The Original Module5.5. General Type Categories5.5.1. Types Share Operation Sets by Categories5.5.2. Mutable Types Can Be Changed in-Place

6.1. Lists6.2. Lists in Action6.2.1. Basic List Operations6.2.2. In dexing, Slicing, and Matrixes6.2.3. Changing Lists in-Place6.2.3.1. Index and slice assignment6.2.3.2. List method calls6.3. Dictionaries6.4. Dictionaries in Action6.4.1. Basic Dictionary Operations6.4.2. Changing Dictionaries in-Place6.4.3. More Dictionary Methods6.4.4. A Languages Table6.4.5. Dictionary Usage Notes6.4.5.1. Using dictionaries to simulate flexible lists6.4.5.2. Using dictionaries for sparse data structures6.4.5.3. Using dictionaries as “records”
7.1. Tuples7.1.1. Why Lists and Tuples?7.2. Files7.2.1. Files in Action7.3. Type Categories Revisited7.4. Object Generality7.5. References Versus Copies7.6. Comparisons, Equality, and Truth7.7. Python’s Type Hierarchies7.8. Other Types in Python7.9. Built-in Type Gotchas7.9.1. Assignment Creates References, Not Copies7.9.2. Repetition Adds One-Level Deep7.9.2.1. Solutions7.9.3. Cyclic Data Structures7.9.4. Immutable Types Can’t Be Changed in-Place7.10. Part II Exercises
8.1. Assignment Statements8.1.1. Variable Name Rules8.1.1.1. Naming conventions8.1.1.2. Names have no type, but objects do8.1.2. Augmented Assignment Statements8.2. Expression Statements8.3. Print Statements8.3.1. The Python “Hello World” Program8.3.2. Redirecting the Output Stream8.3.2.1. The print>>file extension
9.1. if Statements9.1.1. General Format9.1.2. Examples9.1.2.1. Multiway branching9.2. Python Syntax Rules9.2.1. Block Delimiters9.2.2. Statement Delimiters9.2.3. A Few Special Cases9.3. Truth Tests
10.1. while Loops10.1.1. General Format10.1.2. Examples10.2. break, continue, pass, and the Loop else10.2.1. General Loop Format10.2.2. Examples10.2.3. More on the Loop else10.3. for Loops10.3.1. General Format10.3.2. Examples10.4. Loop Variations10.4.1. Counter Loops: range10.4.2. Nonexhaustive Traversals: range10.4.3. Changing Lists: range10.4.4. Parallel Traversals: zip and map10.4.5. Dictionary Construction with zip
11.1. The Python Documentation Interlude11.1.1. Documentation Sources11.1.2. # Comments11.1.3. The dir Function11.1.4. Docstrings: __doc__11.1.4.1. User-defined docstrings11.1.4.2. Docstring standards11.1.4.3. Built-in docstrings11.1.5. PyDoc: The help Function11.1.6. PyDoc: HTML Reports11.1.7. Standard Manual Set11.1.8. Web Resources11.1.9. Published Books11.2. Common Coding Gotchas11.3. Part III Exercises
12.1. Why Use Functions?12.2. Coding Functions12.2.1. def Statements12.2.2. def Executes at Runtime12.3. A First Example: Definitions and Calls12.3.1. Definition12.3.2. Calls12.3.3. Polymorphism in Python12.4. A Second Example: Intersecting Sequences12.4.1. Definition12.4.2. Calls12.4.3. Polymorphism Revisited12.4.4. Local Variables
13.1. Scope Rules13.1.1. Python Scope Basics13.1.2. Name Resolution: The LEGB Rule13.1.3. Scope Example13.1.4. The Built-in Scope13.2. The global Statement13.3. Scopes and Nested Functions13.3.1. Nested Scope Details13.3.2. Nested Scope Examples13.4. Passing Arguments13.4.1. Arguments and Shared References13.4.2. Avoiding Mutable Argument Changes13.4.3. Simulating Output Parameters13.5. Special Argument Matching Modes13.5.1. Keyword and Default Examples13.5.2. Arbitrary Arguments Examples13.5.3. Combining Keywords and Defaults13.5.4. The min Wakeup Call13.5.4.1. Full credit13.5.4.2. Bonus points13.5.4.3. The punch line13.5.5. A More Useful Example: General Set Functions13.5.6. Argument Matching: The Gritty Details
14.1. Anonymous Functions: lambda14.1.1. lambda Expressions14.1.2. Why lambda?14.1.3. How (Not) to Obfuscate Your Python Code14.1.4. Nested lambdas and Scopes14.2. Applying Functions to Arguments14.2.1. The apply Built-in14.2.1.1. Passing keyword arguments14.2.2. Apply-Like Call Syntax14.3. Mapping Functions Over Sequences14.4. Functional Programming Tools14.5. List Comprehensions14.5.1. List Comprehension Basics14.5.2. Adding Tests and Nested Loops14.5.3. Comprehending List Comprehensions14.6. Generators and Iterators14.6.1. Generator Example14.6.2. Iterators and Built-in Types14.7. Function Design Concepts14.7.1. Functions Are Objects: Indirect Calls14.8. Function Gotchas14.8.1. Local Names Are Detected Statically14.8.2. Defaults and Mutable Objects14.8.3. Functions Without Returns14.9. Part IV Exercises
15.1. Why Use Modules?15.2. Python Program Architecture15.2.1. How to Structure a Program15.2.2. Imports and Attributes15.2.3. Standard Library Modules15.3. How Imports Work15.3.1. 1. Find It15.3.1.1. The module search path15.3.1.2. The sys.path list15.3.1.3. Module file selection15.3.2. 2. Compile It (Maybe)15.3.3. 3. Run It
16.1. Module Creation16.2. Module Usage16.2.1. The import Statement16.2.2. The from statement16.2.3. The from * statement16.2.4. Imports Happen Only Once16.2.5. import and from Are Assignments16.2.6. Cross-File Name Changes16.2.7. import and from Equivalence16.3. Module Namespaces16.3.1. Files Generate Namespaces16.3.2. Attribute Name Qualification16.3.3. Imports Versus Scopes16.3.4. Namespace Nesting16.4. Reloading Modules16.4.1. Reload Basics16.4.2. Reload Example
17.1. Package Import Basics17.1.1. Packages and Search Path Settings17.1.2. Package __init__.py Files17.2. Package Import Example17.2.1. from Versus import with Packages17.3. Why Use Package Imports?17.4. A Tale of Three Systems
18.1. Data Hiding in Modules18.1.1. Minimizing from* damage: _X and __all__18.2. Enabling Future Language Features18.3. Mixed Usage Modes: __name__ and __main__18.4. Changing the Module Search Path18.5. The import as Extension18.6. Module Design Concepts18.6.1. Modules Are Objects: Metaprograms18.7. Module Gotchas18.7.1. Importing Modules by Name String18.7.2. from Copies Names but Doesn’t Link18.7.3. Statement Order Matters in Top-Level Code18.7.4. Recursive “from” Imports May Not Work18.7.5. reload May Not Impact from Imports18.7.6. reload and from and Interactive Testing18.7.7. reload Isn’t Applied Transitively18.8. Part V Exercises
19.1. Why Use Classes?19.2. OOP from 30,000 Feet19.2.1. Attribute Inheritance Search19.2.2. Coding Class Trees19.2.3. OOP Is About Code Reuse
20.1. Classes Generate Multiple Instance Objects20.1.1. Class Objects Provide Default Behavior20.1.2. Instance Objects Are Concrete Items20.1.3. A First Example20.2. Classes Are Customized by Inheritance20.2.1. A Second Example20.2.2. Classes and Modules20.3. Classes Can Intercept Python Operators20.3.1. A Third Example20.3.2. Why Operator Overloading?
21.1. The Class Statement21.1.1. General Form21.1.2. Example21.2. Methods21.2.1. Example21.2.2. Calling Superclass Constructors21.2.3. Other Method Call Possibilities21.3. Inheritance21.3.1. Attribute Tree Construction21.3.2. Specializing Inherited Methods21.3.3. Class Interface Techniques21.3.4. Abstract Superclasses21.4. Operator Overloading21.4.1. Common Operator Overloading Methods21.4.2. __getitem__ Intercepts Index References21.4.3. __getitem__ and __iter__ Implement Iteration21.4.3.1. User-defined iterators21.4.4. __getattr__ and __setattr__ Catch Attribute References21.4.5. __repr__ and __str__Return String Representations21.4.6. __radd__ Handles Right-Side Addition21.4.7. __call__ Intercepts Calls21.4.8. __del__ Is a Destructor21.5. Namespaces: The Whole Story21.5.1. Unqualified Names: Global Unless Assigned21.5.2. Qualified Names: Object Namespaces21.5.3. Assignments Classify Names21.5.4. Namespace Dictionaries21.5.5. Namespace Links
22.1. Python and OOP22.1.1. Overloading by Call Signatures (or Not)22.2. Classes as Records22.3. OOP and Inheritance: “is-a” Relationships22.4. OOP and Composition: “has-a” Relationships22.4.1. Stream Processors Revisited22.5. OOP and Delegation22.6. Multiple Inheritance22.7. Classes Are Objects: Generic Object Factories22.7.1. Why Factories?22.8. Methods Are Objects: Bound or Unbound22.9. Documentation Strings Revisited22.10. Classes Versus Modules
23.1. Extending Built-in Types23.1.1. Extending Types by Embedding23.1.2. Extending Types by Subclassing23.2. Pseudo-Private Class Attributes23.2.1. Name Mangling Overview23.2.2. Why Use Pseudo-Private Attributes?23.3. “New Style” Classes in Python 2.223.3.1. Diamond Inheritance Change23.3.1.1. Diamond inheritance example23.3.1.2. Explicit conflict resolution23.3.2. Other New Style Class Extensions23.3.2.1. Static and class methods23.3.2.2. Instance slots23.3.2.3. Class properties23.3.2.4. New __getattribute__ overload method23.4. Class Gotchas23.4.1. Changing Class Attributes Can Have Side Effects23.4.2. Multiple Inheritance: Order Matters23.4.3. Class Function Attributes Are Special: Static Methods23.4.3.1. Solution (prior to 2.2, and in 2.2 normally)23.4.3.2. Static and class methods in Python 2.223.4.4. Methods, Classes, and Nested Scopes23.4.5. Overwrapping-itis23.5. Part VI Exercises
24.1. Why Use Exceptions?24.1.1. Exception Roles24.2. Exception Handling: The Short Story24.3. The try/except/else Statement24.3.1. Try Statement Clauses24.3.2. The try/else Clause24.3.3. Example: Default Behavior24.3.4. Example: Catching Built-in Exceptions24.4. The try/finally Statement24.4.1. Example: Coding Termination Actions with try/finally24.5. The raise Statement24.5.1. Example: Raising and Catching User-Defined Exceptions24.5.2. Example: Passing Extra Data with raise24.5.3. Example: Propagating Exceptions with raise24.6. The assert Statement24.6.1. Example: Trapping Constraints (but Not Errors)
25.1. String-Based Exceptions25.2. Class-Based Exceptions25.2.1. Class Exception Example25.2.2. Why Class Exceptions?25.2.3. Built-in Exception Classes25.2.4. Specifying Exception Text25.2.5. Sending Extra Data in Instances25.2.5.1. Example: extra data with classes and strings25.3. General raise Statement Forms
26.1. Nesting Exception Handlers26.1.1. Example: Control-Flow Nesting26.1.2. Example: Syntactic Nesting26.2. Exception Idioms26.2.1. Exceptions Aren’t Always Errors26.2.2. Functions Signal Conditions with raise26.2.3. Debugging with Outer try Statements26.2.4. Running in-Process Tests26.3. Exception Design Tips26.3.1. What Should Be Wrapped26.3.2. Catching Too Much26.3.3. Catching Too Little26.4. Exception Gotchas26.4.1. String Exceptions Match by Identity, Not Value26.4.2. Catching the Wrong Thing26.5. Core Language Summary26.5.1. The Python Toolset26.5.2. Development Tools for Larger Projects26.6. Part VII Exercises
27.1. Exploring on Your Own27.2. Conversions, Numbers, and Comparisons27.3. Manipulating Strings27.3.1. The string Module27.3.2. Complicated String Matches with Regular Expressions27.3.2.1. A real regular expression problem27.4. Data Structure Manipulations27.4.1. Making Copies27.4.1.1. The copy module27.4.2. Sorting27.4.3. Randomizing27.4.4. Making New Data Structures27.4.5. Making New Lists and Dictionaries27.5. Manipulating Files and Directories27.5.1. The os and os.path Modules27.5.2. Copying Files and Directories: The shutil Module27.5.3. Filenames and Directories27.5.4. Matching Sets of Files27.5.5. Using Temporary Files27.5.6. Modifying Input and Outputs27.5.7. Using Standard I/O to Process Files27.5.7.1. Finding all lines that start with a #27.5.7.2. Extracting the fourth column of a file (where columns are defined by whitespace)27.5.7.3. Extracting the fourth column of a file, where columns are separated by colons, and making it lowercase27.5.7.4. Printing the first 10 lines, the last 10 lines, and every other line27.5.7.5. Counting the number of times the word “Python” occurs in a file27.5.7.6. Changing a list of columns into a list of rows27.5.7.7. Choosing chunk sizes27.5.8. Doing Something to a Set of Files Specified on the Command Line27.5.9. Processing Each Line of One or More Files27.5.10. Dealing with Binary Data: The struct Module27.6. Internet-Related Modules27.6.1. The Common Gateway Interface: The cgi Module27.6.2. Manipulating URLs: The urllib and urlparse Modules27.6.3. Specific Internet Protocols27.6.4. Processing Internet Data27.6.5. XML Processing27.7. Executing Programs27.8. Debugging, Testing, Timing, Profiling27.8.1. Debugging with pdb27.8.2. Testing with unittest27.8.3. Timing27.9. Exercises
28.1. An Automated Complaint System28.1.1. Excerpt from the HTML File28.2. Interfacing with COM: Cheap Public Relations28.3. A Tkinter-Based GUI Editor for Managing Form Data28.3.1. The Main Program28.3.2. The Form Editor28.3.3. Design Considerations28.4. Jython: The Felicitous Union of Python and Java28.4.1. Jython Gives Python Programmers Access to Java Libraries28.4.2. Jython as a Java Scripting Language28.4.3. A Real Jython/Swing Application: grapher.py28.5. Exercises
29.1. Layers of Community29.1.1. The Core29.1.2. Local User Groups29.1.3. Conferences and Workshops29.1.4. Where to Get Help29.1.4.1. Python-help29.1.4.2. Python-tutor29.1.4.3. Python-list29.1.5. Special Interest Groups29.1.6. python-dev29.1.7. News Sources29.2. The Process29.3. Services and Products29.4. The Legal Framework: The Python Software Foundation29.5. Software29.5.1. Search the Web29.5.2. Search the Mailing List Archives29.5.3. Look in the Vaults of Parnassus29.5.4. Check the Python Package Index (PyPI)29.5.5. Look in the Python Cookbook29.6. Popular Third-Party Software29.6.1. Interfaces to Windows and the MacOS29.6.1.1. Windows29.6.1.2. Macintosh29.6.2. Special-Purpose Libraries29.6.2.1. Scientific computing libraries29.6.2.2. Relational database interfaces29.6.2.3. Graphics libraries29.6.2.3.1. Python Imaging Library (PIL)29.6.2.3.2. PyGame and PyOpenGL29.6.3. Interfaces to GUI Toolkits29.6.4. Interfaces to C/C++/Fortran29.6.4.1. SWIG and f2py29.6.4.2. CXX and Boost::Python29.6.4.3. Pyrex, Weave, and Psyco29.6.4.4. ctypes29.6.5. Little Gems29.6.6. Packaging Tools29.6.6.1. Installer29.6.6.2. py2exe29.7. Web Application Frameworks29.7.1. Zope29.7.2. Twisted29.7.3. Quixote29.7.4. Webware, Spyce, and More29.8. Tools for Python Developers
A.1. Installing the Python InterpreterA.1.1. Where to Get PythonA.1.2. Installation StepsA.1.3. Configuration StepsA.1.3.1. Python environment variablesA.1.4. How to Set Configuration OptionsA.1.4.1. UNIX/Linux shell variablesA.1.4.2. DOS variables (Windows)A.1.4.3. Other Windows optionsA.1.4.4. Path files
B.1. Part I, Getting Started B.2. Part II, Types and Operations B.3. Part III, Statements and Syntax B.4. Part IV, Functions B.5. Part V, Modules B.6. Part VI, Classes and OOP B.7. Part VII, Exceptions and Tools B.8. Part VIII, The Outer Layers B.8.1. Chapter 27, Common Tasks in Python B.8.2. Chapter 28, Frameworks

Content preview from Learning Python, 2nd Edition

Chapter 18. Advanced Module Topics

Part V concludes with a collection of more advanced module-related topics, along with the standard set of gotchas and exercises. Just like funtions, modules are more effective when their interfaces are defined well, so this chapter also takes a brief look at module design concepts. Some of the topics here, such as the __name__ trick, are very widely used, despite the word “advanced” in this chapter’s title.

Data Hiding in Modules

As we’ve seen, Python modules export all names assigned at the top level of their file. There is no notion of declaring which names should and shouldn’t be visible outside the module. In fact, there’s no way to prevent a client from changing names inside a module if they want to.

In Python, data hiding in modules is a convention, not a syntactical constraint. If you want to break a module by trashing its names, you can, but we have yet to meet a programmer who would want to. Some purists object to this liberal attitude towards data hiding and claim that it means Python can’t implement encapsulation. However, encapsulation in Python is more about packaging than about restricting.

Minimizing from* damage: _X and all

As a special case, prefixing names with a single underscore (e.g., _X) prevents them from being copied out when a client imports with a from* statement. This really is intended only to minimize namespace pollution; since from* copies out all names, you may get more than you bargained for (including names that ...