book

Mastering Python 2E - Second Edition

by Rick Hattem

May 2022

Intermediate to advanced

710 pages

16h 41m

English

Packt Publishing

Read now

Unlock full access

Who this book is forWhat this book coversTo get the most out of this bookGet in touch
Virtual environmentsWhy virtual environments are a good ideaUsing venv and virtualenvCreating a venvActivating a venv/virtualenvInstalling packagesUsing pyenvUsing AnacondaGetting started with Anaconda NavigatorGetting started with condaManaging dependenciesUsing pip and a requirements.txt fileVersion specifiersInstalling through source control repositoriesAdditional dependencies using extrasConditional dependencies using environment markersAutomatic project management using poetryCreating a new poetry projectAdding dependenciesUpgrading dependenciesRunning commandsAutomatic dependency tracking using pipenvUpdating your packagesDeploying to productionRunning cron commandsExercisesReading the Python Enhancement Proposals (PEPs)Combining pyenv and poetry or pipenvConverting an existing project to a poetry projectSummary
The Python interpreterModifying the interpreterEnabling and enhancing autocompletionAlternative interpretersbpythonRewinding your sessionReloading modulesptpythonIPython and JupyterBasic interpreter usageSaving and loading sessionsRegular Python prompt/doctest modeIntrospection and helpAutocompletionJupyterInstalling JupyterIPython summaryExercisesSummary
A brief history of PythonCode style – What is Pythonic code?Whitespace instead of bracesFormatting strings – printf, str.format, or f-strings?Simple formattingNamed variablesArbitrary expressionsPEP 20, the Zen of PythonBeautiful is better than uglyExplicit is better than implicitSimple is better than complexFlat is better than nestedSparse is better than denseReadability countsPracticality beats purityErrors should never pass silentlyIn the face of ambiguity, refuse the temptation to guessOne obvious way to do itHard to explain, easy to explainNamespaces are one honking great ideaExplaining PEP 8Duck typingDifferences between value and identity comparisonsLoopsMaximum line lengthVerifying code quality, pep8, pyflakes, and moreRecent additions to the Python syntaxPEP 572: Assignment expressions/the walrus operatorPEP 634: Structural pattern matching, the switch statementCommon pitfallsScope matters!Global variablesMutable function default argumentsClass propertiesOverwriting and/or creating extra built-insModifying while iteratingCatching and storing exceptionsLate binding and closuresCircular importsImport collisionsSummary
Time complexity – The big O notationCore collectionslist – A mutable list of itemsdict – A map of itemsset – Like a dict without valuestuple – The immutable listPythonic patterns using advanced collectionsSmart data storage with type hinting using dataclassesCombining multiple scopes with ChainMapDefault dictionary values using defaultdictenum – A group of constantsSorting collections using heapqSearching through sorted collections using bisectGlobal instances using Borg or Singleton patternsNo need for getters and setters with propertiesDict union operatorsExercisesSummary
Functional programmingPurely functionalFunctional programming and PythonAdvantages of functional programminglist, set, and dict comprehensionsBasic list comprehensionsset comprehensionsdict comprehensionsComprehension pitfallslambda functionsThe Y combinatorfunctoolspartial – Prefill function argumentsreduce – Combining pairs into a single resultImplementing a factorial functionProcessing treesReducing in the other direction itertoolsaccumulate – reduce with intermediate resultschain – Combining multiple resultscompress – Selecting items using a list of Booleansdropwhile/takewhile – Selecting items using a functioncount – Infinite range with decimal stepsgroupby – Grouping your sorted iterableExercisesSummary
Decorating functionsGeneric function decoratorsThe importance of functools.wrapsChaining or nesting decoratorsRegistering functions using decoratorsMemoization using decoratorsDecorators with (optional) argumentsCreating decorators using classesDecorating class functionsSkipping the instance – classmethod and staticmethodProperties – Smart descriptor usageDecorating classesSingletons – Classes with a single instanceTotal ordering – Making classes sortableUseful decoratorsSingle dispatch – Polymorphism in Pythoncontextmanager — with statements made easyValidation, type checks, and conversionsUseless warnings – How to ignore them safelyExercisesSummary
GeneratorsCreating generatorsCreating infinite generatorsGenerators wrapping iterablesGenerator comprehensionsClass-based generators and iteratorsGenerator examplesBreaking an iterable up into chunks/groupsitertools.islice – Slicing iterablesitertools.chain – Concatenating multiple iterablesitertools.tee – Using an output multiple timescontextlib.contextmanager – Creating context managersCoroutinesA basic examplePrimingClosing and throwing exceptionsMixing generators and coroutinesUsing the stateExercisesSummary
Dynamically creating classesA basic metaclassArguments to metaclassesAccessing metaclass attributes through classesAbstract classes using collections.abcInternal workings of the abstract classesCustom type checksAutomatically registering plugin systemsImporting plugins on-demandImporting plugins through configurationImporting plugins through the filesystemDataclassesOrder of operations when instantiating classesFinding the metaclassPreparing the namespaceExecuting the class bodyCreating the class object (not instance)Executing the class decoratorsCreating the class instanceExampleStoring class attributes in definition orderThe classic solution without metaclassesUsing metaclasses to get a sorted namespaceExercisesSummary
Type hintingBasic exampleCustom typesGenericsType checkingPython type interface filesType hinting conclusionreStructuredText and MarkdownGetting started with reStructuredTextGetting started with MarkdownInline markupHeadersHeaders with reStructuredTextHeaders with MarkdownListsEnumerated listsBulleted listsOption listsDefinition lists (reST only)Nested listsLinks, references, and labelsImagesImages with reStructuredTextImages with MarkdownSubstitutionsBlocks, code, math, comments, and quotesConclusionThe Sphinx documentation generatorGetting started with SphinxUsing sphinx-quickstartUsing sphinx-apidocSphinx directivesSphinx rolesDocumenting codeDocumenting a class with the Sphinx styleDocumenting a class with the Google styleDocumenting a class with the NumPy styleWhich style to chooseExercisesSummary

Using documentation as tests with doctestA simple doctest exampleWriting doctestsTesting with documentationThe doctest flagsTrue and False versus 1 and 0Normalizing whitespaceEllipsisDoctest quirksTesting dictionariesTesting floating-point numbersTimes and durationsTesting with py.testThe difference between the unittest and py.test outputThe difference between unittest and py.test testsSimplifying assertionsParameterizing testsAutomatic arguments using fixturesPrint statements and loggingPluginsMock objectsUsing unittest.mockUsing py.test monkeypatchTesting multiple environments with toxGetting started with toxThe tox.ini config fileRunning toxLoggingConfigurationBasic logging configurationDictionary configurationJSON configurationini file configurationThe network configurationLoggerUsageFormattingModern formatting using f-strings and str.formatLogging pitfallsDebugging loggersExercisesSummary
Non-interactive debuggingInspecting your script using traceDebugging using loggingShowing the call stack without exceptionsHandling crashes using faulthandlerInteractive debuggingConsole on demandDebugging using Python debugger (pdb)BreakpointsCatching exceptionsAliasescommandsDebugging with IPythonDebugging with JupyterOther debuggersDebugging servicesExercisesSummary
What is performance?Measuring CPU performance and execution timeTimeit – comparing code snippet performancecProfile – Finding the slowest componentsFirst profiling runCalibrating your profilerSelective profiling using decoratorsUsing profile statisticsLine profiler – Tracking performance per lineImproving execution timeUsing the right algorithmGlobal interpreter locktry versus ifLists versus generatorsString concatenationAddition versus generatorsMap versus generators and list comprehensionsCachingLazy importsUsing slotsUsing optimized librariesJust-in-time compilingConverting parts of your code to CMemory usagetracemallocMemory ProfilerMemory leaksCircular referencesAnalyzing memory usage using the garbage collectorWeak referencesWeakref limitations and pitfallsReducing memory usageGenerators versus listsRecreating collections versus removing itemsUsing slotsPerformance monitoringExercisesSummary
Introduction to asyncioBackward compatibility and async/await statementsPython 3.4Python 3.5Python 3.7A basic example of parallel executionasyncio conceptsCoroutines, Futures, and TasksEvent loopsExecutorsAsynchronous examplesProcessesInteractive processesEcho client and serverAsynchronous file operationsCreating async generators to support async forAsynchronous constructors and destructorsDebugging asyncioForgetting to await a coroutineSlow blocking functionsForgetting to check the results or exiting earlyExiting before all tasks are doneExercisesSummary
The Global Interpreter Lock (GIL)The use of multiple threadsWhy do we need the GIL?Why do we still have the GIL?Multiple threads and processesBasic examplesconcurrent.futuresthreadingmultiprocessingCleanly exiting long-running threads and processesBatch processing using concurrent.futuresBatch processing using multiprocessingSharing data between threads and processesShared memory between processesThread safetyDeadlocksThread-local variablesProcesses, threads, or a single thread?threading versus concurrent.futuresmultiprocessing versus concurrent.futuresHyper-threading versus physical CPU coresRemote processesDistributed processing using multiprocessingDistributed processing using DaskInstalling DaskBasic exampleRunning a single threadDistributed execution across multiple machinesDistributed processing using ipyparallelipython_config.pyipython_kernel_config.pyipcontroller_config.pyipengine_config.pyipcluster_config.pySummary
Installing the packagesArrays and matricesNumPy – Fast arrays and matricesNumba – Faster Python on CPU or GPUSciPy – Mathematical algorithms and NumPy utilitiesSparse matricesPandas – Real-world data analysisInput and output optionsPivoting and groupingMergingRolling or expanding windowsStatsmodels – Statistical models on top of Pandasxarray – Labeled arrays and datasetsSTUMPY – Finding patterns in time seriesMathematics and precise calculationsgmpy2 – Fast and precise calculationsSage – An alternative to Mathematica/Maple/MATLABmpmath – Convenient, precise calculationsSymPy – Symbolic mathematicsPatsy – Describing statistical models Plotting, graphing, and chartingMatplotlibSeabornYellowbrickPlotlyBokehDatashaderExercisesSummary
Introduction to artificial intelligenceTypes of AIInstalling the packagesImage processing scikit-imageInstalling scikit-imageEdge detectionFace detectionscikit-image overviewOpenCVInstalling OpenCV for PythonEdge detectionObject detectionOpenCV versus scikit-imageNatural language processingNLTK – Natural Language ToolkitspaCy – Natural language processing with CythonGensim – Topic modeling for humansMachine learningTypes of machine learningSupervised learningReinforcement learningUnsupervised learningCombinations of learning methodsDeep learningArtificial neural networks and deep learningTensorsPyTorch – Fast (deep) neural networksPyTorch Lightning and PyTorch Ignite – High-level PyTorch APIsSkorch – Mixing PyTorch and scikit-learnTensorFlow/Keras – Fast (deep) neural networksTensorFlow versus PyTorchEvolutionary algorithmsSupport-vector machinesBayesian networksVersatile AI libraries and utilitiesscikit-learn – Machine learning in PythonSupervised learningUnsupervised learningauto-sklearn – Automatic scikit-learnmlxtend – Machine learning extensionsscikit-lego – scikit-learn utilitiesXGBoost – eXtreme Gradient BoostingFeaturetools – Feature detection and predictionSnorkel – Improving your ML data automaticallyTPOT – Optimizing ML models using genetic programmingExercisesSummary
Setting up toolingDo you need C/C++ modules?WindowsOS XLinux/UnixCalling C/C++ with ctypesPlatform-specific librariesWindowsLinux/UnixOS XMaking it easyCalling functions and native typesComplex data structuresArraysGotchas with memory managementCFFIComplex data structuresArraysABI or API?CFFI or ctypes?Native C/C++ extensionsA basic exampleC is not Python – Size mattersThe example explainedstaticPyObject*Parsing argumentsC is not Python – Errors are silent or lethalCalling Python from C – Handling complex typesExercisesSummary
Introduction to packagesTypes of packagesWheels – The new eggsSource packagesPackage toolsPackage versioningBuilding packagesPackaging using pyproject.tomlCreating a basic packageInstalling packages for developmentAdding code and dataAdding executable commandsManaging dependenciesBuilding the packageBuilding C/C++ extensionsPackaging using setuptools with setup.py or setup.cfgCreating a basic packageInstalling the package for developmentAdding packagesAdding package dataManaging dependenciesAdding executable commandsBuilding the packagePublishing packagesAdding URLsPyPI trove classifiersUploading to PyPIC/C++ extensionsRegular C/C++ extensionsCython extensionsTestingunittestpy.testExercisesSummary

Content preview from Mastering Python 2E - Second Edition

12 Performance – Tracking and Reducing Your Memory and CPU Usage

Before we talk about performance, there is a quote by Donald Knuth you need to consider first:

“The real problem is that programmers have spent far too much time worrying about efficiency in the wrong places and at the wrong times; premature optimization is the root of all evil (or at least most of it) in programming”.

Donald Knuth is often called the father of algorithm analysis. His book series, The Art of Computer Programming, can be considered the Bible of all fundamental algorithms.

As long as you pick the correct data structures with the right algorithms, performance should not be something to worry about. That does not mean you should ignore performance entirely, but ...