book

Mastering Python

Name: Mastering Python
Author: Rick Hattem
ISBN: 9781785289729

by Rick Hattem

April 2016

Intermediate to advanced

486 pages

9h 21m

English

Packt Publishing

Read now

Unlock full access

Mastering Python
Table of Contents
Mastering Python
Credits
About the Author
About the Reviewers
www.PacktPub.com
eBooks, discount offers, and moreWhy subscribe?
Preface
What this book covers
What you need for this book
Who this book is for

Conventions
Reader feedback
Customer support
Downloading the example codeErrataPiracyQuestions
1. Getting Started – One Environment per Project
Creating a virtual Python environment using venvCreating your first venvvenv argumentsDifferences between virtualenv and venv
Bootstrapping pip using ensurepip
ensurepip usageManual pip install
Installing C/C++ packages
Debian and UbuntuRed Hat, CentOS, and FedoraOS XWindows
Summary
2. Pythonic Syntax, Common Pitfalls, and Style Guide
Code style – or what is Pythonic code?Formatting strings – printf-style or str.format?PEP20, the Zen of PythonBeautiful is better than uglyExplicit is better than implicitSimple is better than complexFlat is better than nestedSparse is better than denseReadability countsPracticality beats purityErrors should never pass silentlyIn the face of ambiguity, refuse the temptation to guessOne obvious way to do itNow is better than neverHard to explain, easy to explainNamespaces are one honking great ideaConclusionExplaining PEP8Duck typingDifferences between value and identity comparisonsLoopsMaximum line lengthVerifying code quality, pep8, pyflakes, and moreflake8Pep8pyflakesMcCabeflake8Pylint
Common pitfalls
Scope matters!Function argumentsClass propertiesModifying variables in the global scopeOverwriting and/or creating extra built-insModifying while iteratingCatching exceptions – differences between Python 2 and 3Late binding – be careful with closuresCircular importsImport collisions
Summary
3. Containers and Collections – Storing Data the Right Way
Time complexity – the big O notation
Core collections
list – a mutable list of itemsdict – unsorted but a fast map of itemsset – like a dict without valuestuple – the immutable list
Advanced collections
ChainMap – the list of dictionariescounter – keeping track of the most occurring elementsdeque – the double ended queuedefaultdict – dictionary with a default valuenamedtuple – tuples with field namesenum – a group of constantsOrderedDict – a dictionary where the insertion order mattersheapq – the ordered listbisect – the sorted list
Summary
4. Functional Programming – Readability Versus Brevity
Functional programming
list comprehensions
dict comprehensions
set comprehensions
lambda functions
The Y combinator
functools
partial – no need to repeat all arguments every timereduce – combining pairs into a single resultImplementing a factorial functionProcessing trees
itertools
accumulate – reduce with intermediate resultschain – combining multiple resultscombinations – combinatorics in Pythonpermutations – combinations where the order matterscompress – selecting items using a list of Booleansdropwhile/takewhile – selecting items using a functioncount – infinite range with decimal stepsgroupby – grouping your sorted iterableislice – slicing any iterable
Summary
5. Decorators – Enabling Code Reuse by Decorating
Decorating functionsWhy functools.wraps is importantHow are decorators useful?Memoization using decoratorsDecorators with (optional) argumentsCreating decorators using classes
Decorating class functions
Skipping the instance – classmethod and staticmethodProperties – smart descriptor usage
Decorating classes
Singletons – classes with a single instanceTotal ordering – sortable classes the easy way
Useful decorators
Single dispatch – polymorphism in PythonContextmanager, with statements made easyValidation, type checks, and conversionsUseless warnings – how to ignore them
Summary
6. Generators and Coroutines – Infinity, One Step at a Time
What are generators?Advantages and disadvantages of generatorsPipelines – an effective use of generatorstee – using an output multiple timesGenerating from generatorsContext managers
Coroutines
A basic examplePrimingClosing and throwing exceptionsBidirectional pipelinesUsing the state
Summary
7. Async IO – Multithreading without Threads
Introducing the asyncio libraryThe async and await statementsPython 3.4Python 3.5Choosing between the 3.4 and 3.5 syntaxA simple example of single-threaded parallel processingConcepts of asyncioFutures and tasksEvent loopsEvent loop implementationsEvent loop policiesEvent loop usageProcessesAsynchronous servers and clientsBasic echo server
Summary
8. Metaclasses – Making Classes (Not Instances) Smarter
Dynamically creating classesA basic metaclassArguments to metaclassesAccessing metaclass attributes through classes
Abstract classes using collections.abc
Internal workings of the abstract classesCustom type checksUsing abc.ABC before Python 3.4
Automatically registering a plugin system
Importing plugins on-demandImporting plugins through configurationImporting plugins through the file system
Order of operations when instantiating classes
Finding the metaclassPreparing the namespaceExecuting the class bodyCreating the class object (not instance)Executing the class decoratorsCreating the class instanceExample
Storing class attributes in definition order
The classic solution without metaclassesUsing metaclasses to get a sorted namespace
Summary
9. Documentation – How to Use Sphinx and reStructuredText
The reStructuredText syntaxGetting started with reStructuredTextInline markupHeadersListsEnumerated listBulleted listOption listDefinition listNested listsLinks, references, and labelsImagesSubstitutionsBlocks, code, math, comments, and quotesConclusion
The Sphinx documentation generator
Getting started with SphinxUsing sphinx-quickstartUsing sphinx-apidocSphinx directivesThe table of contents tree directive (toctree)Autodoc, documenting Python modules, classes, and functionsSphinx roles
Documenting code
Documenting a class with the Sphinx styleDocumenting a class with the Google styleDocumenting a class with the NumPy styleWhich style to choose
Summary
10. Testing and Logging – Preparing for Bugs
Using examples as tests with doctestA simple doctest exampleWriting doctestsTesting with pure documentationThe doctest flagsTrue and False versus 1 and 0Normalizing whitespaceEllipsisDoctest quirksTesting dictionariesTesting floating-point numbersTimes and durations
Testing with py.test
The difference between the unittest and py.test outputThe difference between unittest and py.test testsSimplifying assertionsParameterizing testsAutomatic arguments using fixturesCacheCustom fixturesPrint statements and loggingPluginspytest-covpytest-pep8 and pytest-flakesConfiguring plugins
Mock objects
Using unittest.mockUsing py.test monkeypatch
Logging
ConfigurationBasic logging configurationDictionary configurationJSON configurationIni file configurationThe network configurationLoggerUsage
Summary
11. Debugging – Solving the Bugs
Non-interactive debuggingInspecting your script using traceDebugging using loggingShowing call stack without exceptionsDebugging asyncioHandling crashes using faulthandler
Interactive debugging
Console on demandDebugging using pdbBreakpointsCatching exceptionsCommandsDebugging using ipdbOther debuggersDebugging services
Summary
12. Performance – Tracking and Reducing Your Memory and CPU Usage
What is performance?
Timeit – comparing code snippet performance
cProfile – finding the slowest components
First profiling runCalibrating your profilerSelective profiling using decoratorsUsing profile statistics
Line profiler
Improving performance
Using the right algorithmGlobal interpreter lockTry versus ifLists versus generatorsString concatenationAddition versus generatorsMap versus generators and list comprehensionsCachingLazy importsUsing optimized librariesJust-in-time compilingConverting parts of your code to C
Memory usage
TracemallocMemory profilerMemory leaksReducing memory usageGenerators versus listsRecreating collections versus removing itemsUsing slots
Performance monitoring
Summary
13. Multiprocessing – When a Single CPU Core Is Not Enough
Multithreading versus multiprocessing
Hyper-threading versus physical CPU cores
Creating a pool of workers
Sharing data between processes
Remote processes
Distributed processing using multiprocessingDistributed processing using IPyparallelipython_config.pyipython_kernel_config.pyipcontroller_config.pyipengine_config.pyipcluster_config.py
Summary
14. Extensions in C/C++, System Calls, and C/C++ Libraries
IntroductionDo you need C/C++ modules?WindowsOS XLinux/Unix
Calling C/C++ with ctypes
Platform-specific librariesWindowsLinux/UnixOS XMaking it easyCalling functions and native typesComplex data structuresArraysGotchas with memory management
CFFI
Complex data structuresArraysABI or API?CFFI or ctypes?
Native C/C++ extensions
A basic exampleC is not Python – size mattersThe example explainedstaticPyObject*Parsing argumentsC is not Python – errors are silent or lethalCalling Python from C – handling complex types
Summary
15. Packaging – Creating Your Own Libraries or Applications
Installing packages
Setup parameters
Packages
Entry points
Creating global commandsCustom setup.py commands
Package data
Testing packages
Unittestpy.testNosetests
C/C++ extensions
Regular extensionsCython extensions
Wheels – the new eggs
Distributing to the Python Package Index
Summary
Index

Content preview from Mastering Python

Improving performance

Much can be said about performance optimization, but truthfully, if you have read the entire book up to this point, you know most of the Python-specific techniques to write fast code. The most important factor in application performance will always be the choice of algorithms, and by extension, the data structures. Searching for an item within list is almost always a worse idea than searching for an item in dict or set.

Using the right algorithm

Within any application, the right choice of algorithm is by far the most important performance characteristic, which is why I am repeating it to illustrate the results of a bad choice:

In [1]: a = list(range(1000000)) In [2]: b = dict.fromkeys(range(1000000)) In [3]: %timeit 'x' in a ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781785289729

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Mastering Python

by Rick Hattem

Improving performance

Using the right algorithm

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.