book

Mastering Python

by Rick Hattem

April 2016

Intermediate to advanced

486 pages

9h 21m

English

Packt Publishing

Read now

Unlock full access

eBooks, discount offers, and moreWhy subscribe?
What this book covers

Downloading the example codeErrataPiracyQuestions
Creating a virtual Python environment using venvCreating your first venvvenv argumentsDifferences between virtualenv and venv
ensurepip usageManual pip install
Debian and UbuntuRed Hat, CentOS, and FedoraOS XWindows
Code style – or what is Pythonic code?Formatting strings – printf-style or str.format?PEP20, the Zen of PythonBeautiful is better than uglyExplicit is better than implicitSimple is better than complexFlat is better than nestedSparse is better than denseReadability countsPracticality beats purityErrors should never pass silentlyIn the face of ambiguity, refuse the temptation to guessOne obvious way to do itNow is better than neverHard to explain, easy to explainNamespaces are one honking great ideaConclusionExplaining PEP8Duck typingDifferences between value and identity comparisonsLoopsMaximum line lengthVerifying code quality, pep8, pyflakes, and moreflake8Pep8pyflakesMcCabeflake8Pylint
Scope matters!Function argumentsClass propertiesModifying variables in the global scopeOverwriting and/or creating extra built-insModifying while iteratingCatching exceptions – differences between Python 2 and 3Late binding – be careful with closuresCircular importsImport collisions
Time complexity – the big O notation
list – a mutable list of itemsdict – unsorted but a fast map of itemsset – like a dict without valuestuple – the immutable list
ChainMap – the list of dictionariescounter – keeping track of the most occurring elementsdeque – the double ended queuedefaultdict – dictionary with a default valuenamedtuple – tuples with field namesenum – a group of constantsOrderedDict – a dictionary where the insertion order mattersheapq – the ordered listbisect – the sorted list
Functional programming
The Y combinator
partial – no need to repeat all arguments every timereduce – combining pairs into a single resultImplementing a factorial functionProcessing trees
accumulate – reduce with intermediate resultschain – combining multiple resultscombinations – combinatorics in Pythonpermutations – combinations where the order matterscompress – selecting items using a list of Booleansdropwhile/takewhile – selecting items using a functioncount – infinite range with decimal stepsgroupby – grouping your sorted iterableislice – slicing any iterable
Decorating functionsWhy functools.wraps is importantHow are decorators useful?Memoization using decoratorsDecorators with (optional) argumentsCreating decorators using classes
Skipping the instance – classmethod and staticmethodProperties – smart descriptor usage
Singletons – classes with a single instanceTotal ordering – sortable classes the easy way
Single dispatch – polymorphism in PythonContextmanager, with statements made easyValidation, type checks, and conversionsUseless warnings – how to ignore them
What are generators?Advantages and disadvantages of generatorsPipelines – an effective use of generatorstee – using an output multiple timesGenerating from generatorsContext managers
A basic examplePrimingClosing and throwing exceptionsBidirectional pipelinesUsing the state
Introducing the asyncio libraryThe async and await statementsPython 3.4Python 3.5Choosing between the 3.4 and 3.5 syntaxA simple example of single-threaded parallel processingConcepts of asyncioFutures and tasksEvent loopsEvent loop implementationsEvent loop policiesEvent loop usageProcessesAsynchronous servers and clientsBasic echo server
Dynamically creating classesA basic metaclassArguments to metaclassesAccessing metaclass attributes through classes
Internal workings of the abstract classesCustom type checksUsing abc.ABC before Python 3.4
Importing plugins on-demandImporting plugins through configurationImporting plugins through the file system
Finding the metaclassPreparing the namespaceExecuting the class bodyCreating the class object (not instance)Executing the class decoratorsCreating the class instanceExample
The classic solution without metaclassesUsing metaclasses to get a sorted namespace
The reStructuredText syntaxGetting started with reStructuredTextInline markupHeadersListsEnumerated listBulleted listOption listDefinition listNested listsLinks, references, and labelsImagesSubstitutionsBlocks, code, math, comments, and quotesConclusion
Getting started with SphinxUsing sphinx-quickstartUsing sphinx-apidocSphinx directivesThe table of contents tree directive (toctree)Autodoc, documenting Python modules, classes, and functionsSphinx roles
Documenting a class with the Sphinx styleDocumenting a class with the Google styleDocumenting a class with the NumPy styleWhich style to choose
Using examples as tests with doctestA simple doctest exampleWriting doctestsTesting with pure documentationThe doctest flagsTrue and False versus 1 and 0Normalizing whitespaceEllipsisDoctest quirksTesting dictionariesTesting floating-point numbersTimes and durations
The difference between the unittest and py.test outputThe difference between unittest and py.test testsSimplifying assertionsParameterizing testsAutomatic arguments using fixturesCacheCustom fixturesPrint statements and loggingPluginspytest-covpytest-pep8 and pytest-flakesConfiguring plugins
Using unittest.mockUsing py.test monkeypatch
ConfigurationBasic logging configurationDictionary configurationJSON configurationIni file configurationThe network configurationLoggerUsage
Non-interactive debuggingInspecting your script using traceDebugging using loggingShowing call stack without exceptionsDebugging asyncioHandling crashes using faulthandler
Console on demandDebugging using pdbBreakpointsCatching exceptionsCommandsDebugging using ipdbOther debuggersDebugging services
What is performance?
First profiling runCalibrating your profilerSelective profiling using decoratorsUsing profile statistics
Using the right algorithmGlobal interpreter lockTry versus ifLists versus generatorsString concatenationAddition versus generatorsMap versus generators and list comprehensionsCachingLazy importsUsing optimized librariesJust-in-time compilingConverting parts of your code to C
TracemallocMemory profilerMemory leaksReducing memory usageGenerators versus listsRecreating collections versus removing itemsUsing slots
Multithreading versus multiprocessing
Distributed processing using multiprocessingDistributed processing using IPyparallelipython_config.pyipython_kernel_config.pyipcontroller_config.pyipengine_config.pyipcluster_config.py
IntroductionDo you need C/C++ modules?WindowsOS XLinux/Unix
Platform-specific librariesWindowsLinux/UnixOS XMaking it easyCalling functions and native typesComplex data structuresArraysGotchas with memory management
Complex data structuresArraysABI or API?CFFI or ctypes?
A basic exampleC is not Python – size mattersThe example explainedstaticPyObject*Parsing argumentsC is not Python – errors are silent or lethalCalling Python from C – handling complex types
Installing packages
Creating global commandsCustom setup.py commands
Unittestpy.testNosetests
Regular extensionsCython extensions
Distributing to the Python Package Index

Content preview from Mastering Python

Summary

This chapter has shown us how multiprocessing works, how we can pool a lot of jobs, and how we should share data between multiple processes. But more interestingly, it has also shown how we can distribute processing across multiple machines which helps a lot in speeding up heavy calculations.

The most important lesson you can learn from this chapter is that you should always try to avoid data sharing and synchronisation between multiple processes or servers, as it is slow and will thus slow down your applications a lot. Whenever possible, keep your calculations and data local.

In the next chapter we will learn about creating extensions in C/C++ to increase performance and allow low-level access to memory and other hardware resources. While ...