Programming Python, 3rd Edition

Signs of the Python Times

It’s been an exciting decade in the Python world. Since I wrote the first edition of this book in 1995 and 1996, Python has grown from a new kid on the scripting-languages block to an established and widely used tool in companies around the world. In fact, today the real question is not who is using Python, but who is not. Python is now used in some fashion in almost every software organization—whether as a tactical tool for quick tasks or an implementation language for longer-range strategic projects.

Although measuring the popularity of an open source, freely distributed tool such as Python is not always easy (there are no licenses to be tallied), most available statistics reveal exponential growth in Python’s popularity over the last decade. Among the most recent signs of Python’s explosive growth are:

Users

In 1999, one leading industry observer suggested that, based on various statistics, there were as many as 300,000 Python users worldwide. Other estimates are still more optimistic. In early 2000, for instance, the Python web site was already on track to service 500,000 new Python interpreter downloads by year end in addition to other Python distribution media. Python is also a standard preinstalled item on Linux, Macintosh, and some Windows computers today and is embedded in various applications and hardware.

Today, the best estimates, based on developer surveys and network activity, suggest that there are likely between 750,000 and 1 million Python users worldwide. A better estimate is impossible because of Python’s open source nature, but Python clearly enjoys a large and active user community.

Applications

Real organizations have adopted Python and Python-focused systems for real projects. It has been used to:

• Animate movies (Industrial Light & Magic, Sony Pictures Imageworks, Disney, Pixar)

• Perform searches on the Internet (Google, Infoseek)

• Script GIS mapping products (ESRI)

• Distribute content downloads on the Internet (BitTorrent)

• Predict the weather (U.S. National Weather Service, NOAA)

• Test computer hardware (Seagate, Intel, Hewlett-Packard, Micron, KLA)

• Do numeric analysis (NASA, Los Alamos National Laboratory, Lawrence Livermore National Laboratory, Fermi)

• Perform cryptography and stock market analysis (NSA, Getco)

• Script games and graphics (Origin, Corel, Blender, PyGame)

• Navigate spacecraft and control experiments (Jet Propulsion Laboratory)

• Serve up maps and directories on the Web (Yahoo!)

• Guide users through Linux installation and maintenance (Red Hat)

• Implement web sites and content (Disney, JPL, Zope, Plone, Twisted)

• Design missile defense systems (Lockheed Martin)

• Manage mail lists (Mailman)

• Deliver eGreeting cards (American Greetings)

• Implement Personal Information Managers (Chandler)

...and much more.^[*] Some of the Python-based systems in the preceding list are very popular in their own right. For example, the widely used Google search engine—arguably responsible for much of the Web’s success—makes heavy use of the Python language and is likely the most successful server-side application of Python so far. And in the latest release of its popular ArcGIS geographical information system (GIS), ESRI has begun recommending Python as the scripting tool for customization and automation to its reported 1 million licensees.

Of special note, BitTorrent, a distributed file-sharing system written in Python, is likely the most successful client-side Python program to date. It already records 42 million lifetime downloads on SourceForge.net as this chapter is being written, and it is listed as the number three package for all-time top downloads at that site (this does not include the roughly 2 million new downloads per month, or alternative clients that embed the BitTorrent Python backend). In addition, a late 2004 Reuters report noted that more than one-third of the Internet’s traffic was based on BitTorrent. Per other reports, BitTorrent accounted for 53 percent of all peer-to-peer (P2P) Internet traffic in mid-2004, and P2P traffic may be two-thirds of all Internet traffic today.

Books

When I started the first edition of this book in 1995, no Python books were available. As I wrote the second edition of this book in 2000, more than a dozen were available, with almost that many more on the way. And as I write this third edition in 2005, far more than 50 Python books are on the market, not counting non-English translations (a simple search for “Python programming” books currently yields 91 hits on Amazon.com). Some of these books are focused on a particular domain such as Windows or the Web, and some are available in German, French, Japanese, and other language editions.

Domains

Python has grown to embrace Microsoft Windows developers, with support for .NET, COM, and Active Scripting; Java developers, with the Jython Java-based implementation of the language; Mac OS X developers, with integration of tools such as Cocoa and standard inclusion in the Mac OS; and web developers, with a variety of toolkits such as Zope and Plone.

As we’ll see in this book, the COM support allows Python scripts to be both a server and a client of components and to interface with Microsoft Office products; Active Scripting allows Python code to be embedded in HTML web page code and run on either clients or servers. The Jython system compiles Python scripts to Java Virtual Machine (JVM) code so that they can be run in Java-aware systems and seamlessly integrate Java class libraries for use by Python code.

As an open source tool for simplifying web site construction, the Python-based Zope web application framework discussed in this edition has also captured the attention of webmasters and CGI coders. Dynamic behavior in Zope web sites is scripted with Python and rendered with a server-side templating system. By using a workflow model, the Plone web content management system, based on Zope and Python, also allows webmasters to delegate the management of web site content to people who produce the content. Other toolkits, such as Django, Twisted, CherryPy, and Webware, similarly support network-based applications.

Compilers

As I write this third edition, two Python compilers are under development for the Microsoft .NET framework and C# language environment—independent implementations of the Python language that provide seamless .NET integration for Python scripts.

For instance, the new IronPython implementation of Python for .NET and Mono compiles Python code for use in the .NET runtime environment (and is currently being developed in part by Microsoft employees). It promises to be a new, alternative implementation of Python, along with the standard C-based Python and the Jython Java-based implementation mentioned in the prior section.

Other systems, such as the Psyco just-in-time bytecode compiler and the PyPy project, which may subsume it the IronPython implementation, promise substantial speedups for Python programs. See this chapter’s sidebar "How Python Runs Your Code" for more details on program execution and compilers.

Newsgroup

User traffic on the main Python Internet newsgroup, comp.lang.python, has risen dramatically too. For instance, according to Yahoo! Groups (see http://groups.yahoo.com/group/python-list), 76 articles were posted on that list in January 1994 and 2,678 in January 2000—a 35-fold increase. Later months were busier still (e.g., 4,226 articles during June 2000, and 7,675 in February 2003—roughly 275 per day), and growth has been generally constant since the list’s inception.

Python Internet newsgroup user traffic—along with all other user-base figures cited in this chapter—is likely to have increased by the time you read this text. But even at current traffic rates, Python forums are easily busy enough to consume the full-time attention of anyone with full-time attention to give. Other online forums, such as weblogs (blogs), host additional Python-oriented discussions.

Conferences

There are now two or more annual Python conferences in the U.S., including the annual PyCon event, organized by the Python community, and the Python conference held as part of the Open Source Convention, organized by O’Reilly. Attendance at Python conferences roughly doubled in size every year in their initial years. At least two annual conferences are also now held in Europe each year, including EuroPython and PythonUK. Furthermore, there is now a PyCon conference in Brazil, and conferences have also been held in other places around the world.

Press

Python is regularly featured in industry publications. In fact, since 1995, Python creator Guido van Rossum has appeared on the cover of prominent tech magazines such as Linux Journal and Dr. Dobb’s Journal; the latter publication gave him a programming excellence award for Python. Linux Journal also published a special Python supplement with its May 2000 issue, and a Python-specific magazine, PyZine, was started up in recently.

Group therapy

Regional Python user groups have begun springing up in numerous sites in the U.S. and abroad, including Oregon, San Francisco, Washington D.C., Colorado, Italy, Korea, and England. Such groups work on Python-related enhancements, organize Python events, and more.

Services

On the pragmatics front, commercial support, consulting, prepackaged distributions, and professional training for Python are now readily available from a variety of sources. For instance, the Python interpreter can be obtained on CDs and packages sold by various companies (including ActiveState), and Python usually comes prebuilt and free with most Linux and recent Macintosh operating systems. In addition, there are now two primary sites for finding third-party add-ons for Python programming: the Vaults of Parnassus and PyPI (see http://www.python.org for links).

Jobs

It’s now possible to make money as a Python programmer (without having to resort to writing large, seminal books). As I write this book, the Python job board at http://www.python.org/Jobs.html lists some 60 companies seeking Python programmers in the U.S. and abroad, in a wide variety of domains. Searches for Python at popular employment sites such as Monster.com and Dice.com yield hundreds of hits for Python-related jobs. And according to one report, the number of Python jobs available in the Silicon Valley area increased 400 percent to 600 percent in the year ending in mid-2005. Not that anyone should switch jobs, of course, but it’s nice to know that you can now make a living by using a language that also happens to be fun.

Tools

Python has also played host to numerous tool development efforts. Among the most prominent are the Software Carpentry project, which developed new core software tools in Python; ActiveState, which provides a set of Windows- and Linux-focused Python development products; the Eclipse development environment; and PythonWare, which offers a handful of Python tools.

Education

Python has also begun attracting the attention of educators, many of whom see Python as the “Pascal of the 2000s”—an ideal language for teaching programming due to its simplicity and structure. Part of this appeal was spawned by Guido van Rossum’s proposed Computer Programming for Everybody (CP4E) project, aimed at making Python the language of choice for first-time programmers worldwide.

CP4E itself is now defunct, but an active Python Special Interest Group (SIG) has been formed to address education-related topics. Regardless of any particular initiative’s outcome, Python promises to make programming more accessible to the masses. As people grow tired of clicking preprogrammed links, they may evolve from computer users to computer scripters.

Recent Growth (As of 2005, at Least)

As I was writing this third edition, I found that all signs pointed toward continued growth in the Python world:

• Python.org traffic had increased 30 percent for the year that ended in March 2005.

• PyCon conference attendance essentially doubled, increasing to 400–500 attendees in 2005 compared to 200–300 in 2004.

• Python 2.4 was given a Jolt productivity award in early 2005 by Software Development Magazine.

• Per a survey conducted by InfoWorld, Python popularity nearly doubled in 2004 (usage by developers grew to 14 percent in late 2004, versus 8 percent in the prior year; another survey in the same period measured Python use to be roughly 16 percent).

• Based on the InfoWorld survey and the number of all developers, the Python user base is now estimated to be from 750,000 to 1 million worldwide.

• Google, maker of the leading web search engine, launched an open source code site whose initially featured components were mostly Python code.

• The IronPython port being developed in part by Microsoft reported an 80 percent performance boost over the standard C-based Python 2.4 release on some benchmarks.

• As mentioned, the number of Python jobs available in Silicon Valley have reportedly increased by a factor of 4 to 6.

• A web site that automatically tracks the frequency of references to programming languages in online forums found that Python chatter more than doubled between 2004 and 2005. This site also found that among scripting languages, only Python traffic showed the early stages of a rapid growth curve.

• According to an article by O’Reilly, industry-wide book sales data shows that the Python book market grew to two-thirds the size of the Perl book market as of April 2005. Two years earlier, the Python book market was approximately one-sixth the size of the Perl book market. (Perl is an older scripting language optimized for text processing tasks, which some see as being in competition with Python for mindshare.)

In other words, it’s not 1995 anymore. Much of the preceding list was unimaginable when the first edition of this book was conceived. Naturally, this list is doomed to be out-of-date even before this book hits the shelves, but it is nonetheless representative of the sorts of milestones that have occurred over the last five years and will continue to occur for years to come. As a language optimized to address the productivity demands of today’s software world, Python’s best is undoubtedly yet to come.

What’s in a Name?

Python gets its name from the 1970s British TV comedy series Monty Python’s Flying Circus. According to Python folklore, Guido van Rossum, Python’s creator, was watching reruns of the show at about the same time he needed a name for a new language he was developing. And as they say in show business, “the rest is history.”

Because of this heritage, references to the comedy group’s work often show up in examples and discussion. For instance, the words spam, lumberjack, and shrubbery have a special connotation to Python users, and confrontations are sometimes referred to as “The Spanish Inquisition.” As a rule, if a Python user starts using phrases that have no relation to reality, they’re probably borrowed from the Monty Python series or movies. Some of these phrases might even pop up in this book. You don’t have to run out and rent The Meaning of Life or The Holy Grail to do useful work in Python, of course, but it can’t hurt.

While “Python” turned out to be a distinctive name, it has also had some interesting side effects. For instance, when the Python newsgroup, comp.lang.python, came online in 1994, its first few weeks of activity were almost entirely taken up by people wanting to discuss topics from the TV show. More recently, a special Python supplement in the Linux Journal magazine featured photos of Guido garbed in an obligatory “nice red uniform.”

Python’s news list still receives an occasional post from fans of the show. For instance, one poster innocently offered to swap Monty Python scripts with other fans. Had he known the nature of the forum, he might have at least mentioned whether they ran on Windows or Unix.

---

^[*] See http://www.python.org/moin/OrganizationsUsingPython or search Python.org (http://www.python.org/about/success) for more examples of Python-based applications. Some companies don’t disclose their Python use for competitive reasons, though many eventually become known when one of their web pages crashes and displays a Python error message in a browser. O’Reilly has also published a list of Python success stories derived from a list of testimonials maintained by people interested in Python advocacy; see the advocacy group’s list at http://www.pythonology.com/success.