Programming Python, 3rd Edition

What’s Python Good For?

Because Python is used in a wide variety of ways, it’s almost impossible to give an authoritative answer to this question. As a general-purpose language, Python can be used for almost anything computers are capable of. Its feature set applies to both rapid and longer-term development modes. And from an abstract perspective, any project that can benefit from the inclusion of a language optimized for speed of development is a good target Python application domain. Given the ever-shrinking schedules in software development, this is a very broad category.

A more specific answer is less easy to formulate. For instance, some use Python as an embedded extension language, and others use it exclusively as a standalone programming tool. To some extent, this entire book will answer this very question—it explores some of Python’s most common roles. For now, here’s a summary of some of the more common ways Python is being applied today:

System utilities

Portable command-line tools, testing, system administration scripts

Internet scripting

CGI web sites, Java applets, XML, email, Zope/Plone, CherryPy, Webware, Twisted

GUIs

With tools such as Tk, wxPython, Qt, Gtk, PythonCard, Dabo, Swing, Anygui

Component integration

C/C++ library frontends, product customization

Database access

Persistent object stores, SQL database interfaces

Distributed programming

With client/server APIs like CORBA, CGI, COM, .NET, SOAP, XML-RPC

Rapid-prototyping/development

Tactical run-once programs or deliverable prototypes

Language-based modules

Replacing special-purpose parsers with Python

And more

Image processing, numeric programming, gaming, AI, etc.

On the other hand, Python is not really tied to any particular application area. For example, Python’s integration support makes it useful for almost any system that can benefit from a frontend, programmable interface. In abstract terms, Python provides services that span domains. It is all of the things described in the following list.

“Buses Considered Harmful”

The PSA organization described earlier was originally formed in response to an early thread on the Python newsgroup that posed the semiserious question: “What would happen if Guido was hit by a bus?” The more recent PSF group has been tasked to address similar questions.

These days, Python creator Guido van Rossum is still the ultimate arbiter of proposed Python changes. He was officially anointed the BDFL—Benevolent Dictator For Life—of Python, at the first Python conference and still makes final yes and no decisions on language changes (and usually says no: a good thing in the programming languages domain, because Python tends to change slowly and in backward-compatible ways).

But Python’s user base helps support the language, work on extensions, fix bugs, and so on. It is a true community project. In fact, Python development is now a completely open process—anyone can inspect the latest source-code files or submit patches by visiting a web site (see http://www.python.org for details).

As an open source package, Python development is really in the hands of a very large cast of developers working in concert around the world. Given Python’s popularity, bus attacks seem less threatening now than they once did; of course, I can’t speak for Guido.

• A dynamic programming language, ideal for situations in which a compile/link step is either impossible (on-site customization) or inconvenient (prototyping, rapid development, system utilities)

• A powerful but simple programming language designed for development speed, ideal for situations in which the complexity of larger languages can be a liability (prototyping, end-user coding, time to market)

• A generalized language tool, ideal for situations in which we might otherwise need to invent and implement yet another “little language” (programmable system interfaces, configuration tools)

Given these general properties, you can apply Python to any area you’re interested in by extending it with domain libraries, embedding it in an application, or using it all by itself. For instance, Python’s role as a system tools language is due as much to its built-in interfaces to operating system services as to the language itself.

In fact, because Python was built with integration in mind, it has naturally given rise to a growing library of extensions and tools, available as off-the-shelf components to Python developers. Table 1-2 names just a few as a random sample (with apologies to the very many systems omitted here). You can find more about most of these components in this book, on Python’s web site, at the Vaults of Parnassus and PyPI web sites mentioned earlier in this chapter, and by a simple Google web search.

How Python Runs Your Code

Today, Python is “interpreted” in the same way Java is: Python source code is automatically compiled (translated) to an intermediate and platform-neutral form called bytecode, which is then executed by the Python virtual machine (that is, the Python runtime system). Translation to bytecode happens when a module is first imported, and it is avoided when possible to speed program startup: bytecode is automatically saved in .pyc files and, unless you change the corresponding source file, loaded directly the next time your program runs.

This bytecode compilation model makes Python scripts portable and faster than a pure interpreter that runs raw source code lines. But it also makes Python slower than true compilers that translate source code to binary machine code. Bytecode is not machine code and is ultimately run by the Python (or other) virtual machine program, not directly by your computer’s hardware.

Keep in mind, though, that some of these details are specific to the standard Python implementation. For instance, the Jython system compiles Python scripts to Java bytecode, and the IronPython implementation compiles Python source code to the bytecode used by the C#/.NET environment. In addition, Python compiler-related projects have been spawned in the past and will likely continue into the future. For more details on this front, see the following:

• The Psyco just-in-time compiler for Python, which replaces portions of a running program’s bytecode with optimized binary machine code tailored to specific datatypes. Psyco can speed Python programs by any factor from 2 to 100. The high end is more likely for heavily algorithmic code, whereas I/O-bound programs don’t improve as much. (In my own experience, a 3x-5x speedup is common for typical programs—amazing for a simple install.)

• A related project, PyPy, which aims to reimplement the Python virtual machine to better support optimizations. The PyPy project may incorporate and subsume Psyco’s techniques.

• The Parrot project, which seeks to develop a bytecode and virtual machine that will be shared by many languages, including Python.

• The Installer, Py2Exe, and Freeze systems, which package Python programs as standalone executables known as “frozen binaries”—a combination of your bytecode and the Python virtual machine. Frozen binaries do not require that Python be installed on the receiving end.

• Other program distribution formats, including zip archives (with modules automatically extracted on imports); Python eggs (an emerging package format); Distutils (an installation script system); and encrypted bytecode (for instance, using PyCrypto and the import hooks).

• The emerging Shed Skin system, which translates Python source code to C++. This system assumes that your code will not use the full range of Python’s dynamic typing, but this constraint allows highly efficient code to be generated, which is by some accounts faster than Psyco and much faster than standard Python. Shed Skin’s own website reports speedups of 12 and 45 times faster on average than Psyco and standard CPython, respectively, though results can vary greatly.

Psyco may provide a simpler optimization path for some programs than linked-in C libraries, especially for algorithm-intensive code. Although Python’s extreme dynamic nature makes compilation complex (the behavior of “x + 1” cannot be easily predicted until runtime), a future optimizing Python compiler might also make many of the performance notes in this chapter moot points.

Table 1-2. Popular Python domains, tools, and extensions

Domain	Tools and extensions
Systems programming: support for all common system-level tools	Sockets, processes, threads, signals, pipes, RPC, directories, POSIX bindings...
GUIs: a variety of portable GUI toolkits and builders	Tkinter, wxPython, PyQt, PyGTK, Anygui, Swing, PythonCard, Dabo...
Database interfaces: interfaces for both relational and object-oriented databases	MySQL, Oracle, Sybase, PostgreSQL, SQLite, persistence, ZODB, DBM...
Microsoft Windows tools: access to a variety of Windows-specific tools	MFC wrappers, COM interfaces, ActiveX scripting, ASP, ODBC drivers, .NET...
Internet tools: sockets, CGI, client tools, server tools, web frameworks, parsers, Apache support, Java integration	Jython, XML, email, ElementTree, htmllib, telnetlib, urllib, Zope, CherryPy, Twisted, Webware, Django, mod_python, SSL...
Distributed objects: SOAP web services, XML-RPC, CORBA, DCOM	PySOAP, SOAPy, xmlrpclib, ILU, Fnorb, omniORB, PyWin32...
Other popular tools: graphics, language, visualization, numerics, cryptography, integration, gaming, wikis...	PIL, VPython, Blender, PyOpenGL, NLTK, YAPPS, VTK, NumPy, PyCrypto, SWIG, ctypes, PyGame, MoinMoin...