Python in a Nutshell, 3rd Edition

Chapter 1. Introduction to Python

Python, a general-purpose programming language, has been around for quite a while: Guido van Rossum, Python’s creator, started developing Python back in 1990. This stable and mature language is very high-level, dynamic, object-oriented, and cross-platform—all very attractive characteristics. Python runs on all major hardware platforms and operating systems, so it doesn’t constrain your choices.

Python offers high productivity for all phases of the software life cycle: analysis, design, prototyping, coding, testing, debugging, tuning, documentation, deployment, and, of course, maintenance. Python’s popularity has seen steady, unflagging growth over the years. Today, familiarity with Python is an advantage for every programmer, as Python has infiltrated every niche and has useful roles to play as a part of any software solution.

Python provides a unique mix of elegance, simplicity, practicality, and sheer power. You’ll quickly become productive with Python, thanks to its consistency and regularity, its rich standard library, and the many third-party packages and tools that are readily available for it. Python is easy to learn, so it is quite suitable if you are new to programming, yet is also powerful enough for the most sophisticated expert.

The Python Language

The Python language, while not minimalist, is spare, for good pragmatic reasons. Once a language offers one good way to express a design idea, adding other ways has, at best, modest benefits, while cost, in terms of language complexity, grows more than linearly with the number of features. A complicated language is harder to learn and master (and to implement efficiently and without bugs) than a simpler one. Complications and quirks in a language hamper productivity in software development and maintenance, particularly in large projects, where many developers cooperate and often maintain code originally written by others.

Python is simple, but not simplistic. It adheres to the idea that, if a language behaves a certain way in some contexts, it should ideally work similarly in all contexts. Python also follows the principle that a language should not have “convenient” shortcuts, special cases, ad hoc exceptions, overly subtle distinctions, or mysterious and tricky under-the-covers optimizations. A good language, like any other well-designed artifact, must balance such general principles with taste, common sense, and a high degree of practicality.

Python is a general-purpose programming language: Python’s traits are useful in just about any area of software development. There is no area where Python cannot be part of a great solution. “Part” is an important word here; while many developers find that Python fills all of their needs, it does not have to stand alone: Python programs can cooperate with a variety of other software components, making it an ideal language for gluing together components written in other languages. One of the language’s design goals was that it should “play well with others.”

Python is a very high-level language (VHLL). This means that Python uses a higher level of abstraction, conceptually further from the underlying machine, than do classic compiled languages such as C, C++, and Fortran, which are traditionally called “high-level languages.” Python is simpler, faster to process (both for human brains and for programmatic tools), and more regular than classic high-level languages. This enables high programmer productivity and makes Python an attractive development tool. Good compilers for classic compiled languages can generate binary machine code that runs faster than Python code. However, in most cases, the performance of Python-coded applications is sufficient. When it isn’t, apply the optimization techniques covered in “Optimization” to improve your program’s performance while keeping the benefit of high productivity.

Somewhat newer languages such as Java and C# are slightly higher-level than classic ones such as C and Fortran, and share some characteristics of classic languages (such as the need to use declarations) as well as some of VHLLs like Python (such as the use of portable bytecode as the compilation target in typical implementations, and garbage collection to relieve programmers from the need to manage memory). If you find you are more productive with Java or C# than with C or Fortran, try Python (possibly in the Jython or IronPython implementations, covered in “Python Implementations”) to become even more productive.

In terms of language level, Python is comparable to other powerful VHLLs like JavaScript, Ruby, and Perl. The advantages of simplicity and regularity, however, remain on Python’s side.

Python is an object-oriented programming language, but lets you develop code using both object-oriented and procedural styles, and a touch of functional programming, too, mixing and matching as your application requires. Python’s object-oriented features are conceptually similar to those of C++, but simpler to use.

The Python Standard Library and Extension Modules

There is more to Python programming than just the Python language: the standard library and other extension modules are nearly as important for effective Python use as the language itself. The Python standard library supplies many well-designed, solid, 100 percent pure Python modules for convenient reuse. It includes modules for such tasks as representing data, processing text, interacting with the operating system and filesystem, and web programming. Because these modules are written in Python, they work on all platforms supported by Python.

Extension modules, from the standard library or from elsewhere, let Python code access functionality supplied by the underlying operating system or other software components, such as graphical user interfaces (GUIs), databases, and networks. Extensions also afford maximal speed in computationally intensive tasks such as XML parsing and numeric array computations. Extension modules that are not coded in Python, however, do not necessarily enjoy the same automatic cross-platform portability as pure Python code.

You can write special-purpose extension modules in lower-level languages to achieve maximum performance for small, computationally intensive parts that you originally prototyped in Python. You can also use tools such as Cython and CFFI to wrap existing C/C++ libraries into Python extension modules, as covered in “Extending Python Without Python’s C API”. Last but not least, you can embed Python in applications coded in other languages, exposing existing application functionality to Python scripts via app-specific Python extension modules.

This book documents many modules, both from the standard library and from other sources, in areas such as client- and server-side network programming, databases, manipulation of text and binary files, and interaction with the operating system.

Python Implementations

Python currently has four production-quality implementations (CPython, Jython, IronPython, PyPy) and a new high-performance implementation in early development, Pyston, which we do not cover further. We also mention a number of other, more experimental, implementations in a following section.

This book primarily addresses CPython, the most widely used implementation, which we usually refer to as just “Python” for simplicity. However, the distinction between a language and its implementations is an important one.

CPython

Classic Python (AKA CPython, often just called Python) is the most up-to-date, solid, and complete production-quality implementation of Python. It can be considered the “reference implementation” of the language. CPython is a compiler, interpreter, and set of built-in and optional extension modules, all coded in standard C. CPython can be used on any platform where the C compiler complies with the ISO/IEC 9899:1990 standard (i.e., all modern, popular platforms). In “Installation”, we explain how to download and install CPython. All of this book, except a few sections explicitly marked otherwise, applies to CPython. CPython supports both v3 (version 3.5 or better) and v2 (version 2.7).

Jython

Jython is a Python implementation for any Java Virtual Machine (JVM) compliant with Java 7 or better. Such JVMs are available for all popular, modern platforms. As of this writing, Jython supports v2, but not yet v3. With Jython, you can use all Java libraries and frameworks. For optimal use of Jython, you need some familiarity with fundamental Java classes. You do not have to code in Java, but documentation and examples for Java libraries are couched in Java terms, so you need a nodding acquaintance with Java to read and understand them. You also need to use Java-supporting tools for tasks such as manipulating .jar files and signing applets. This book deals with Python, not with Java. To use Jython, you should complement this book with Jython Essentials, by Noel Rappin and Samuele Pedroni (O’Reilly), Java in a Nutshell, by David Flanagan (O’Reilly), and/or The Definitive Guide to Jython (Open Source Version) by Josh Juneau, Jim Baker, et al. (Apress, available on O’Reilly’s Safari and on Jython.org), as well as some of the many other Java resources available.

IronPython

IronPython is a Python implementation (as of this writing, only v2, not yet v3) for the Microsoft-designed Common Language Runtime (CLR), most commonly known as .NET, which is now open source and ported to Linux and macOS. With IronPython, you can use all CLR libraries and frameworks. In addition to Microsoft’s own implementation, a cross-platform implementation of the CLR known as Mono works with other, non-Microsoft operating systems, as well as with Windows. For optimal use of IronPython, you need some familiarity with fundamental CLR libraries. You do not have to code in C#, but documentation and examples for existing CLR libraries are often couched in C# terms, so you need a nodding acquaintance with C# to read and understand them. You also need to use CLR supporting tools for tasks such as making CLR assemblies. This book deals with Python, not with the CLR. For IronPython usage, you should complement this book with IronPython’s own online documentation, and, if needed, some of the many other resources available about .NET, the CLR, C#, Mono, and so on.

PyPy

PyPy is a fast and flexible implementation of Python, coded in a subset of Python itself, able to target several lower-level languages and virtual machines using advanced techniques such as type inferencing. PyPy’s greatest strength is its ability to generate native machine code “just in time” as it runs your Python program. PyPy implements v2 and, at this writing, Python 3.5 support is being released in alpha. PyPy has substantial advantages in speed and memory management, and is seeing production-level use.

Choosing Between CPython, Jython, IronPython, and PyPy

If your platform, as most are, is able to run all of CPython, Jython, IronPython, and PyPy, how do you choose among them? First of all, don’t choose prematurely: download and install them all. They coexist without problems, and they’re all free. Having them all on your development machine costs only some download time and a little extra disk space, and lets you compare them directly.

The primary difference between the implementations is the environment in which they run and the libraries and frameworks they can use. If you need a JVM environment, then Jython is your best choice. If you need a CLR (AKA “.NET”) environment, take advantage of IronPython. If you need a custom version of Python, or need high performance for long-running programs, consider PyPy.

If you’re mainly working in a traditional environment, CPython is an excellent fit. If you don’t have a strong preference for one or the other, start with the standard CPython reference implementation, which is most widely supported by third-party add-ons and extensions, and offers the highly preferable v3 (in full 3.5 glory).¹

In other words, when you’re experimenting, learning, and trying things out, use CPython, most widely supported and mature. To develop and deploy, your best choice depends on the extension modules you want to use and how you want to distribute your programs. CPython applications are often faster than Jython or IronPython, particularly if you use extension modules such as NumPy (covered in “Array Processing”); however, PyPy can often be even faster, thanks to just-in-time compilation to machine code. It may be worthwhile to benchmark your CPython code against PyPy.

CPython is most mature: it has been around longer, while Jython, IronPython, and PyPy are newer and less proven in the field. The development of CPython versions tends to proceed ahead of that of Jython, IronPython, and PyPy: at the time of writing, for example, the language level supported is, for v2, 2.7 for all of them; for v3, 3.5 and 3.6 are now available in CPython, and PyPy has 3.5 in alpha.

However, Jython can use any Java class as an extension module, whether the class comes from a standard Java library, a third-party library, or a library you develop yourself. Similarly, IronPython can use any CLR class, whether from the standard CLR libraries, or coded in C#, Visual Basic .NET, or other CLR-compliant languages. PyPy is drop-in compatible with most standard CPython libraries, and with C-coded extensions using Cython, covered in “Cython”, and CFFI, mentioned in “Extending Python Without Python’s C API”.

A Jython-coded application is a 100 percent pure Java application, with all of Java’s deployment advantages and issues, and runs on any target machine having a suitable JVM. Packaging opportunities are also identical to Java’s. Similarly, an IronPython-coded application is entirely compliant with .NET’s specifications.

Jython, IronPython, PyPy, and CPython are all good, faithful implementations of Python, reasonably close to each other in terms of usability and performance. Since each of the JVM and CLR platforms carries a lot of baggage, but also supplies large amounts of useful libraries, frameworks, and tools, any of the implementations may enjoy decisive practical advantages in a specific deployment scenario. It is wise to become familiar with the strengths and weaknesses of each, and then choose optimally for each development task.

Other Developments, Implementations, and Distributions

Python has become popular enough that there are a number of other groups and individuals who have taken an interest in its development and provided features and implementations outside the core development team’s focus.

Programmers new to Python used to often find it troublesome to install. Nowadays almost all Unix-based systems include a v2 implementation, and recent releases of Ubuntu include v3 as the “system Python.” If you are serious about software development in Python, the first thing you should do is leave your system-installed Python alone! Quite apart from anything else, Python is increasingly used by the operating system itself, so tweaking the Python installation could lead to trouble.

Therefore, consider installing one or more Python implementations that you can freely use for your development convenience, sure in the knowledge that nothing you do affects the operating system. We also recommend the use of virtual environments (see “Python Environments”) to isolate projects from each other, allowing them to make use of what might otherwise be conflicting dependencies (e.g., if two of your projects require different versions of the same module).

Python’s popularity has led to the creation of many active communities, and the language’s ecosystem is very active. The following sections outline some of the more interesting recent developments, but our failure to include a project here reflects limitations of space and time rather than implying any disapproval on our part.

Pyjion

Pyjion is an open source project from Microsoft whose primary goal is to add an API to CPython for managing just-in-time (JIT) compilers. Secondary goals include the creation of a JIT compiler for Microsoft’s core CLR environment, which although at the heart of the .NET environment is now open sourced, and a framework for the development of JIT compilers. It is therefore not an implementation of Python, but we mention it because it offers the potential to translate CPython’s bytecode into highly efficient code for many different environments. Integration of Pyjion into CPython will be enabled by PEP 523, which was implemented in Python 3.6.

IPython

IPython enhances standard CPython to make it more powerful and convenient for interactive use. IPython extends the interpreter’s capabilities by allowing abbreviated function call syntax, and extensible functionality known as magics introduced by the percent (%) character. It also provides shell escapes, allowing a Python variable to receive the result of a shell command. You can use a question mark to query an object’s documentation (two question marks for extended documentation); all the standard features of Python are also available.

IPython has made particular strides in the scientific and data-focused world, and has slowly morphed (through the development of IPython Notebook, now refactored and renamed the Jupyter Notebook and shown in the following figure) into an interactive programming environment that, among snippets of code, also lets you embed commentary in literate programming style (including mathematical notation) and show the output of executing code, with advanced graphics produced by such subsystems as matplotlib and bokeh. An example of matplotlib graphics is shown in the bottom half of the figure. Reassuringly grant-funded in recent years, Jupyter/IPython is one of Python’s most prominent success stories.

MicroPython: Is your child already a programmer?

The continued trend in miniaturization has brought Python well within the range of the hobbyist. Single-board computers like the Raspberry Pi and the Beagle Board let you run Python in a full Linux environment. Below this level, there is an interesting class of devices known as microcontrollers, programmable chips with configurable hardware. This extends the scope of hobby and professional projects—for example, by making analog and digital sensing easy, enabling such applications as light and temperature measurements with little additional hardware.

Typical examples at the time of writing were the Adafruit Feather and the WiPy, but new devices appear (and disappear) all the time. Thanks to the MicroPython project, these devices, some of which come with advanced functionality like WiFi, can now be programmed in Python.

MicroPython is a Python 3 implementation that can produce bytecode or executable machine code (though many users will be happily unaware of the latter fact). It implements the syntax of v3, but lacks most of the standard library. Special hardware control modules allow configuration and driving of the various pieces of built-in configurable hardware, and access to Python’s socket library lets devices be clients for Internet services. External devices and timer events can trigger code.

A device typically offers interpreter access through a USB serial port or over the Telnet protocol (we aren’t aware of any ssh implementations yet, though, so take care to firewall these devices properly: they should not be directly accessible across the Internet without proper precautions!). You can program the device’s power-on bootstrap sequence in Python by creating a boot.py file in the device’s memory, and this file can execute arbitrary Python code of any complexity.

Thanks to MicroPython, the Python language can be a full player in the Internet of Things. At the time of writing the Python-capable BBC micro:bit device is being distributed to every year-7 schoolchild in the UK (roughly one million in number). Are you sure your children don’t already know some Python?

Anaconda and Miniconda

One of the more successful Python distributions in recent years is Anaconda from Continuum Analytics. This open source package comes with an enormous number of preconfigured and tested modules in addition to the standard library. In many cases you might find that it contains all the necessary dependencies for your work. On Unix-based systems it installs very simply in a single directory: to activate it, just add the Anaconda bin directory at the front of your shell PATH.

Anaconda is a Python distribution based on a packaging technology called conda. A sister implementation, Miniconda, gives access to the same libraries but does not come with them preloaded, making it more suitable for creating tailored environments. Conda does not use the standard virtual environments but contains equivalent facilities to allow separation of the dependencies for multiple projects.

Nuitka: Convert your Python to C++

Nuitka is an alternative compiler that converts your Python code to C++, with support for v2 and v3. Nuitka allows convenient optimization of your code, with current speeds over twice as fast as CPython. Future milestones include better type inferencing and even more speed. The Nuitka homepage has more information.

Grumpy: A Python-to-Go transpiler

Grumpy is an experimental compiler that converts your Python code to Go, with support for v2 only; the Go code then compiles to a native executable, allowing the import of Go-coded modules (though not of normal Python extensions) and high scalability. Grumpy is optimized for serving large parallel workloads with many lightweight threads, one day potentially replacing CPython as the runtime used to serve YouTube’s millions of requests per second, as explained on the Google Open Source blog. Grumpy’s maintained on the GitHub Grumpy project.

Transcrypt: Convert your Python to JavaScript

Many attempts have been made to determine how Python could become a browser-based language, but JavaScript’s hold has been tenacious. The Transcrypt system is a pip-installable Python package to convert Python code into browser-executable JavaScript. You have full access to the browser’s DOM, allowing your code to dynamically manipulate window content and use JavaScript libraries.

Although it creates minified code, Transcrypt provides full sourcemaps that allow you to debug with reference to the Python source rather than the generated JavaScript. You can write browser event handlers in Python, mixing it freely with HTML and JavaScript. Python may never make it as a first-class browser language, but Transcrypt means you might no longer need to worry about that.

Another (more mature but very active) project to let you script your web pages with Python is Brython, and there are others yet. To help you pick among them, you can find useful info and links on this Stack Overflow thread.

Licensing and Price Issues

CPython is covered by the Python Software Foundation License Version 2, which is GNU Public License (GPL) compatible, but lets you use Python for any proprietary, free, or other open source software development, similar to BSD/Apache/MIT licenses. Jython’s, IronPython’s, and PyPy’s licenses are similarly liberal. Anything you download from the main Python, Jython, IronPython, and PyPy sites won’t cost you a penny. These licenses do not constrain what licensing and pricing conditions you can use for software you develop using the tools, libraries, and documentation they cover.

However, not everything Python-related is free from licensing costs or hassles. Many third-party Python sources, tools, and extension modules that you can freely download have liberal licenses, similar to that of Python itself. Others are covered by the GPL or Lesser GPL (LGPL), constraining the licensing conditions you can place on derived works. Some commercially developed modules and tools may require you to pay a fee, either unconditionally or if you use them for profit.

There is no substitute for careful examination of licensing conditions and prices. Before you invest time and energy into any software tool or component, check that you can live with its license. Often, especially in a corporate environment, such legal matters may involve consulting lawyers. Modules and tools covered in this book, unless we explicitly say otherwise, can be taken to be, at the time of this writing, freely downloadable, open source, and covered by a liberal license akin to Python’s. However, we claim no legal expertise, and licenses can change over time, so double-checking is always prudent.

Python Development and Versions

Python is developed, maintained, and released by a team of core developers led by Guido van Rossum, Python’s inventor, architect, and Benevolent Dictator For Life (BDFL). This title means that Guido has the final say on what becomes part of the Python language and standard library. Python’s intellectual property is vested in the Python Software Foundation (PSF), a nonprofit corporation devoted to promoting Python, described in “Python Software Foundation”. Many PSF Fellows and members have commit privileges to Python’s reference source repositories (currently Mercurial, but transitioning to git, as documented in the Python Developer’s Guide), and most Python committers are members or Fellows of the PSF.

Proposed changes to Python are detailed in public docs called Python Enhancement Proposals (PEPs), debated (sometimes an advisory vote is taken) by Python developers and the wider Python community, and finally approved or rejected by Guido (or his delegates), who take debates and votes into account but are not bound by them. Hundreds of people contribute to Python development through PEPs, discussion, bug reports, and patches to Python sources, libraries, and docs.

The Python core team releases minor versions of Python (3.x for growing values of x), also known as “feature releases,” currently at a pace of about once every 18 months. Python 2.7, first released in July 2010, was the last feature release of the 2.x series. Current and future releases of v2 are and will be bug-fix point releases only, not adding any features. As of this writing, the current bug-fix release of 2.7.13 is available. There will never be a 2.8 release: we recommend migrating to v3.

Each v3 minor release (as opposed to bug-fix point release) adds features that make Python more powerful and simpler to use, but also takes care to maintain backward compatibility. Python 3.0, which was allowed to break backward compatibility in order to remove redundant “legacy” features and simplify the language, was first released in December 2008. Python 3.5 (the version we cover as “v3”) was first released in September 2015; 3.6 was released in December 2016, just as we are finishing writing this book.

Each minor release 3.x is first made available in alpha releases, tagged as 3.x a0, 3.x a1, and so on. After the alphas comes at least one beta release, 3.x b1, and after the betas, at least one release candidate, 3.x rc1. By the time the final release of 3.x (3.x.0) comes out, it is quite solid, reliable, and tested on all major platforms. Any Python programmer can help ensure this by downloading alphas, betas, and release candidates, trying them out, and filing bug reports for any problems that emerge.

Once a minor release is out, part of the attention of the core team switches to the next minor release. However, a minor release normally gets successive point releases (i.e., 3.x.1, 3.x.2, and so on) that add no functionality, but can fix errors, port Python to new platforms, enhance documentation, and add optimizations and tools.

This book focuses on Python 2.7 and 3.5 (and all their point releases), the most stable and widespread releases at the time of this writing (with side notes about the highlights of the newest release, 3.6). Python 2.7 is what we call v2, while 3.5 (with its successors) is what we call v3. We document which parts of the language and libraries exist in v3 only and cannot be used in v2, as well as v3 ones that can be used in v2—for example, with from __future__ import syntax (see Imports from __future__). Whenever we say that a feature is in v3, we mean it’s in Python 3.5 (and presumably all future versions, including 3.6, but not necessarily in, say, 3.4); saying that a feature is in v2 means it’s in Python 2.7.x (currently 2.7.13) but not necessarily in 2.6 or earlier. In a few minor cases, we specify a point release, such as 3.5.3, for language changes where the point release matters.

This book does not address older versions of Python, such as 2.5 or 2.6; such versions are more than five years old and should not be used for any development. However, you might have to worry about such legacy versions if they are embedded in some application you need to script. Fortunately, Python’s backward compatibility is quite good within major releases: v2 is able to properly process almost every valid Python program that was written for Python 1.5.2 or later. As mentioned, however, Python 3 was a major release that did break backward compatibility, although it is possible to write code that works in both v2 and v3 if care is taken. Please see Chapter 26 for more on migrating legacy code to v3. You can find code and documentation online for all old releases of Python.

Python Resources

The richest Python resource is the web. The best starting point is Python’s homepage, full of interesting links to explore. The Jython site is a must if you’re interested in Jython. IronPython has a .net site. PyPy is documented at the PyPy site.

Documentation

Python, Jython, IronPython, and PyPy all come with good documentation. The manuals are available in many formats, suitable for viewing, searching, and printing. You can read CPython’s manuals online (we often refer to these as “the online docs”): both v2 and v3 are available. You can also find various downloadable formats for v2 and v3.The Python Documentation page has links to a large variety of documents. The Jython Documentation page has links to Jython-specific documents as well as general Python ones; there are also documentation pages for IronPython and for PyPy. You can also find Frequently Asked Questions (FAQ) documents for Python, Jython, IronPython, and PyPy online.

Python documentation for nonprogrammers

Most Python documentation (including this book) assumes some software-development knowledge. However, Python is quite suitable for first-time programmers, so there are exceptions to this rule. Good introductory online texts for non-programmers include:

• Josh Cogliati’s “Non-Programmers Tutorial For Python,” available online in two versions: for v2, and for v3

• Alan Gauld’s “Learning to Program,” available online (covers both v2 and v3)

• “Think Python, 2nd edition,” by Allen Downey, available online (covers primarily v3, with notes for v2 usage)

An excellent resource for learning Python (for nonprogrammers, and for less experienced programmers too) is the Beginners’ Guide wiki, which includes a wealth of links and advice. It’s community-curated, so it will likely stay up-to-date as available books, courses, tools, and so on keep evolving and improving.

Extension modules and Python sources

A good starting point to Python extensions and sources is the Python Package Index (still fondly known to old-timers as “The Cheese Shop,” and generally referred to now as PyPI), which at the time of this writing offers over 95,000 packages,² each with descriptions and pointers.

The standard Python source distribution contains excellent Python source code in the standard library and in the Demos and Tools directories, as well as C source for the many built-in extension modules. Even if you have no interest in building Python from source, we suggest you download and unpack the Python source distribution for study purposes; or, if you choose, peruse it online.

Many Python modules and tools covered in this book also have dedicated sites. We include references to such sites in the appropriate chapters in this book.

Books

Although the web is a rich source of information, books still have their place (if we didn’t agree on this, we wouldn’t have written this book, and you wouldn’t be reading it). Books about Python are numerous. Here are a few we recommend, although some cover older versions rather than current ones:

• If you’re just learning to program, Python for Dummies, by Stef and Aahz Maruch (For Dummies), while dated (2006), is a great introduction to programming, and to elementary Python (v2) in particular.

• If you know some programming, but are just starting to learn Python, Head First Python, 2nd edition, by Paul Barry (O’Reilly), may serve you well. Like all Head First series books, it uses graphics and humor to teach its subject.

• Dive Into Python and Dive Into Python 3, by Mark Pilgrim (APress), teach by example, in a fast-paced and thorough way that is quite suitable for people who are already expert programmers in other languages. You can also freely download the book, in any of several formats, for either v2 or v3.

• Beginning Python: From Novice to Professional, by Magnus Lie Hetland (APress), teaches both by thorough explanations and by fully developing 10 complete programs in various application areas.

• Python Essential Reference, by David Beazley (New Riders), is a complete reference to the Python language and its standard libraries.

• For a concise summary reference and reminder of Python’s essentials, check out Python Pocket Reference, by Mark Lutz (O’Reilly).

Community

One of the greatest strengths of Python is its robust, friendly, welcoming community. Python programmers and contributors meet f2f³ at conferences, “hackathons” (often known as sprints in the Python community), and local user groups; actively discuss shared interests; and help each other on mailing lists and social media.

Python Software Foundation

Besides holding the intellectual property rights for the Python programming language, a large part of the mission of the Python Software Foundation (PSF) is to “support and facilitate the growth of a diverse and international community of Python programmers.” The PSF sponsors user groups, conferences, and sprints, and provides grants for development, outreach, and education, among other activities. The PSF has dozens of Fellows (nominated for their contributions to Python, and including all of the Python core team, as well as all authors of this book); hundreds of members who contribute time, work, and money; and dozens of corporate sponsors. Under the new open membership model, anyone who uses and supports Python can become a member of the PSF. Check out the membership page for information on becoming a member of the PSF. If you’re interested in contributing to Python itself, see the Python Developer’s Guide.

Python conferences

There are lots of Python conferences worldwide. General Python conferences include international and regional ones, such as PyCon and EuroPython, and other more localized ones such as PyOhio and Pycon Italia. Topical conferences include SciPy, PyData, and DjangoCon. Conferences are often followed by coding sprints, where Python contributors get together for several days of coding focused on particular open source projects and on abundant camaraderie. You can find a listing of conferences at the Community Workshops page. Videos of talks from previous conferences can be found at the pyvideo site.

User groups and Meetups

The Python community has local user groups in every continent except Antarctica, as listed at the Local User Groups wiki. In addition, there are Python Meetups listed for over 70 countries. Women in Python have an international PyLadies group, as well as local PyLadies chapters.

Special Interest Groups

Discussions on some specialized subjects related to Python take place on the mailing lists of Python Special Interest Groups (SIGs). The SIGS page has a list, and pointers to general and specific information about them. Over a dozen SIGs are active at the time of this writing. Here are a few examples:

• cplusplus-sig: Bindings between C++ and Python

• mobile-sig: Python on mobile devices

• edu-sig: Python in Education

Social media

For an RSS feed of Python-related blogs, see planetpython or follow them on Twitter @planetpython. On Twitter, you can also follow @ThePSF. FreeNode on IRC hosts several Python-related channels: the main one is #python. On Facebook, you can join the Python Software Developers public group; on Google+, the Unofficial Python Community; on LinkedIn, check its Python page for more. Technical questions and answers about Python can also be found and followed on stackoverflow.com under a variety of tags, including python, python-2.7, python-3.x, and others. Python is an actively curated topic on Stack Overflow, where many useful answers with illuminating discussions can be found.

Mailing lists and newsgroups

The Community lists page has links to Python-related mailing lists and newsgroups. The Usenet newsgroup for Python discussions is comp.lang.python. The newsgroup is also available as a mailing list; to subscribe, fill out the form. Python-related official announcements are posted to python announce, or its mailing-list equivalent. To ask for individual help with specific problems, write to [e-mail]. For help learning or teaching Python, write to [e-mail] or join the list. For issues or discussion about porting between v2 and v3, join the python-porting list.

For a useful weekly round-up of Python-related news and articles, you can subscribe to the Python Weekly.

In 2015, the Python Software Foundation created a community-oriented mailing list to support its new open membership model, which you can subscribe to.

Installation

You can install Python in classic (CPython), JVM (Jython), .NET (IronPython), and PyPy versions, on most platforms. With a suitable development system (C for CPython, Java for Jython, .NET for IronPython; PyPy, coded in Python itself, only needs CPython installed first), you can install Python versions from the respective source code distributions. On popular platforms, you also have the (highly recommended!) alternative of installing from prebuilt binary distributions.

Installing Python if it comes preinstalled

If your platform comes with a preinstalled version of Python, you’re still best advised to install another, independent, updated version. When you do, you must not remove or overwrite your platform’s original version—rather, install the other version “side by side” with the first one. This way, you make sure you are not going to disturb any other software that is part of your platform: such software might rely on the exact Python version that came with the platform itself.

Installing CPython from a binary distribution is faster, saves you substantial work on some platforms, and is the only possibility if you have no suitable C compiler. Installing from source code gives you more control and flexibility, and is a must if you can’t find a suitable prebuilt binary distribution for your platform. Even if you install from binaries, it’s best to also download the source distribution, as it includes examples and demos that may be missing from prebuilt binaries.

Installing Python from Binaries

If your platform is popular and current, you’ll find prebuilt, packaged binary versions of Python ready for installation. Binary packages are typically self-installing, either directly as executable programs, or via appropriate system tools, such as the RedHat Package Manager (RPM) on some versions of Linux and the Microsoft Installer (MSI) on Windows. After downloading a package, install it by running the program and choosing installation parameters, such as the directory where Python is to be installed. In Windows, select the option labeled “Add Python 3.5 to PATH” to have the installer add the install location into the PATH in order to easily use Python at a command prompt (see “The python Program”).

To download “official” Python binaries, visit the Python site, click Downloads, and on the next page, click the button labeled Python 2.7 or Python 3.x to download the binary suitable for the browser’s platform.

Many third parties supply free binary Python installers for other platforms. Installers exist for Linux distributions, whether your distribution is RPM-based (RedHat, Fedora, Mandriva, SUSE, etc.), or Debian and Ubuntu. The Other Downloads page provides links to binary distributions for now-somewhat-exotic platforms such as OS/2, RISC OS, IBM AS/400, and so forth.

Anaconda is a binary distribution including either v2 or v3, plus the conda package manager, and hundreds of third-party extensions, particularly for science, math, engineering, and data analysis. It’s available for Linux, Windows, and macOS. The same package, but without all of those extensions (you can selectively install subsets of them with conda), is also available, and is known as Miniconda.

macOS

Apple’s macOS comes with v2 (text-mode only), as of this writing. Nevertheless, we recommend you install the latest version and enhancements by following the instructions and links on the Python site under Downloads; due to Apple’s release cycles, the Python version included with your version of macOS may be out of date or incomplete. Python’s latest version installs in addition to, not instead of, Apple’s supplied one; Apple uses its own version of Python and proprietary extensions to implement some of the software it distributes as a part of macOS, so it’s unwise to risk disturbing that version in any way.

The popular third-party macOS open source package manager Homebrew offers, among many other packages, excellent versions of Python, both v2 and v3. Beware, though, that usually only one release of each is available.

Installing Python from Source Code

To install CPython from source code, you need a platform with an ISO-compliant C compiler and tools such as make. On Windows, the normal way to build Python is with Visual Studio (ideally both VS2008 and VS2010 for v2, or VS 2015 for v3).

To download Python source code, visit the Python site, click Downloads, then pick v2 or v3. For more options, hover on Downloads, and use the menu that shows.

The .tgz file extension under the link labeled “Gzipped source tarball” is equivalent to .tar.gz (i.e., a tar archive of files, compressed by the powerful and popular gzip compressor). Alternatively, use the link labeled “XZ compressed source tarball” to get a version with an extension of .tar.xz instead of .tgz, compressed with the even more powerful xz compressor, if you have tools that can deal with XZ compression.

Microsoft Windows

On Windows, installing Python from source code can be a chore unless you are already familiar with Visual Studio and are used to working in the text-oriented windows known as the command prompt.

If the following instructions give you any trouble, stick with “Installing Python from Binaries”. It’s best to do a separate installation from binaries anyway, even if you also install from source. If you notice anything strange while using the version you installed from source, double-check with the installation from binaries. If the strangeness goes away, it must be due to some quirk in your installation from source, so you know you must double-check the latter.

In the following sections, for clarity, we assume you have made a new folder called %USERPROFILE%\py, (e.g., c:\users\tim\py). Open this folder by typing, for example, %USERPROFILE% in the address bar of a Windows Explorer window and going from there. Download the source tgz file—for example, Python-3.5.2.tgz (or other Python version of your choice) to that folder. Of course, name and place the folder as it best suits you: our name choice is just for expository purposes.

Uncompressing and unpacking the Python source code

You can uncompress and unpack a .tgz or .tar.xz file, for example, with the free program 7-Zip. Download and install 7-Zip from the download page, and run it on the sources’ tgz file (e.g., c:\users\tim\py\Python-3.5.2.tgz) you downloaded. (The tgz file may have downloaded to %USERPROFILE%\downloads, in which case you need to move it to your \py folder before unpacking it with 7-Zip.) You now have a folder %USERPROFILE%\py\Python-3.5.2 (or other version originally downloaded), the root of a tree that contains the entire standard Python distribution in source form.

Building the Python source code

Open the text file %USERPROFILE%\py\Python-3.5.2\PCBuild\readme.txt (or whichever version of Python you downloaded) with the text editor of your choice, and follow the detailed instructions found there.

Unix-Like Platforms

On Unix-like platforms, installing Python from source code is generally simple. In the following sections, for clarity, we assume you have created a new directory named ~/Py and downloaded the sources’ tgz file—for example, Python-3.5.2.tgz (or other Python version of your choice)—to that directory. Of course, name and place the directory as it best suits you: our name choice is just for expository purposes.

Uncompressing and unpacking the Python source code

You can uncompress and unpack a .tgz or .tar.xz file with the popular GNU version of tar. Just type the following at a shell prompt:

$ cd ~/Py
$ tar xzf Python-3.5.2.tgz

You now have a directory ~/Py/Python-3.5.2, the root of a tree that contains the entire standard Python distribution in source form.

Configuring, building, and testing

Detailed notes are in ~/Py/Python-3.5.2/README under the heading “Build instructions,” and we recommend studying those notes. In the simplest case, however, all you need may be to give the following commands at a shell prompt:

$ cd ~/Py/Python-3.5.2
$ ./configure
    [configure writes much information - snipped here]
$ make
    [make takes quite a while, and emits much information]

If you run make without first running ./configure, make implicitly runs ./configure. When make finishes, check that the Python you have just built works as expected:

$ make test
    [takes quite a while, emits much information]

Usually, make test confirms that your build is working, but also informs you that some tests have been skipped because optional modules were missing.

Some of the modules are platform-specific (e.g., some may work only on machines running SGI’s ancient Irix operating system), so don’t worry about them. However, other modules are skipped because they depend on other open source packages that are not installed on your machine. For example, module _tkinter—needed to run the Tkinter GUI package and the IDLE integrated development environment, which come with Python—can be built only if ./configure can find an installation of Tcl/Tk 8.0 or later on your machine. See ~/Py/Python-3.5.2/README for more details and specific caveats about different Unix and Unix-like platforms.

Building from source code lets you tweak your configuration in several ways. For example, you can build Python in a special way that helps you track down memory leaks when you develop C-coded Python extensions, covered in “Building and Installing C-Coded Python Extensions”. ./configure --help is a good source of information about the configuration options you can use.

Installing after the build

By default, ./configure prepares Python for installation in /usr/local/bin and /usr/local/lib. You can change these settings by running ./configure with option --prefix before running make. For example, if you want a private installation of Python in subdirectory py35 of your home directory, run:

$ cd ~/Py/Python-3.5.2
$ ./configure --prefix=~/py35

and continue with make as in the previous section. Once you’re done building and testing Python, to perform the actual installation of all files, run:

$ make install

The user running make install must have write permissions on the target directories. Depending on your choice of target directories, and permissions on those directories, you may need to su to root, bin, or some other user when you run make install. A common idiom for this purpose is sudo make install: if sudo prompts for a password, enter your current user’s password, not root’s. An alternative, and recommended, approach is to install into a virtual environment, as covered in “Python Environments”.

Installing Jython

To download Jython, visit the Jython homepage and follow the link labeled Download. The latest version, at the time of this writing, is Jython 2.7.0, which supports Python 2.7 and comes as two executable JARs: one for just running Jython and one for embedding it in Java applications. Perform the installation by following the installation instructions.

Installing IronPython

To install IronPython, you need to have a current Common Language Runtime (CLR, AKA .NET) implementation installed on your machine. Both Mono and the Microsoft .NET Framework work fine with IronPython. To download IronPython, visit the IronPython homepage, and click the arrow-shaped icon labeled Download IronPython 2.7; this gives you an executable installer—run it to install IronPython.

Installing PyPy

To install PyPy, download your chosen version of PyPy’s sources (the latest version, at the time of this writing, is PyPy 5.6.0, but don’t let its numbering confuse you—it fully supports Python 2.7, which we call v2 in this book!), then follow the instructions if you want to build PyPy from sources. Alternatively, download a binary installer for your platform (if supported), uncompress it from its .zip or .tar.bz2 format, and then run the resulting executable file.

¹ Throughout this book, we’ll mention some highlights of Python 3.6, just barely released as we write—while 3.6 may not yet be mature enough for you to use in production releases, knowing what’s coming as 3.6 does mature can’t fail to be helpful to your technical choices.

² By the time you read this, there will likely be more than 100,000 packages.

³ Face-to-face (i.e., in-person).