book

Mastering Python 2E - Second Edition

Name: Mastering Python 2E - Second Edition
Author: Rick Hattem
ISBN: 9781800207721

by Rick Hattem

May 2022

Intermediate to advanced

710 pages

16h 41m

English

Packt Publishing

Read now

Unlock full access

Preface
Who this book is forWhat this book coversTo get the most out of this bookGet in touch
Getting Started – One Environment per Project
Virtual environmentsWhy virtual environments are a good ideaUsing venv and virtualenvCreating a venvActivating a venv/virtualenvInstalling packagesUsing pyenvUsing AnacondaGetting started with Anaconda NavigatorGetting started with condaManaging dependenciesUsing pip and a requirements.txt fileVersion specifiersInstalling through source control repositoriesAdditional dependencies using extrasConditional dependencies using environment markersAutomatic project management using poetryCreating a new poetry projectAdding dependenciesUpgrading dependenciesRunning commandsAutomatic dependency tracking using pipenvUpdating your packagesDeploying to productionRunning cron commandsExercisesReading the Python Enhancement Proposals (PEPs)Combining pyenv and poetry or pipenvConverting an existing project to a poetry projectSummary
Interactive Python Interpreters
The Python interpreterModifying the interpreterEnabling and enhancing autocompletionAlternative interpretersbpythonRewinding your sessionReloading modulesptpythonIPython and JupyterBasic interpreter usageSaving and loading sessionsRegular Python prompt/doctest modeIntrospection and helpAutocompletionJupyterInstalling JupyterIPython summaryExercisesSummary
Pythonic Syntax and Common Pitfalls
A brief history of PythonCode style – What is Pythonic code?Whitespace instead of bracesFormatting strings – printf, str.format, or f-strings?Simple formattingNamed variablesArbitrary expressionsPEP 20, the Zen of PythonBeautiful is better than uglyExplicit is better than implicitSimple is better than complexFlat is better than nestedSparse is better than denseReadability countsPracticality beats purityErrors should never pass silentlyIn the face of ambiguity, refuse the temptation to guessOne obvious way to do itHard to explain, easy to explainNamespaces are one honking great ideaExplaining PEP 8Duck typingDifferences between value and identity comparisonsLoopsMaximum line lengthVerifying code quality, pep8, pyflakes, and moreRecent additions to the Python syntaxPEP 572: Assignment expressions/the walrus operatorPEP 634: Structural pattern matching, the switch statementCommon pitfallsScope matters!Global variablesMutable function default argumentsClass propertiesOverwriting and/or creating extra built-insModifying while iteratingCatching and storing exceptionsLate binding and closuresCircular importsImport collisionsSummary
Pythonic Design Patterns
Time complexity – The big O notationCore collectionslist – A mutable list of itemsdict – A map of itemsset – Like a dict without valuestuple – The immutable listPythonic patterns using advanced collectionsSmart data storage with type hinting using dataclassesCombining multiple scopes with ChainMapDefault dictionary values using defaultdictenum – A group of constantsSorting collections using heapqSearching through sorted collections using bisectGlobal instances using Borg or Singleton patternsNo need for getters and setters with propertiesDict union operatorsExercisesSummary
Functional Programming – Readability Versus Brevity
Functional programmingPurely functionalFunctional programming and PythonAdvantages of functional programminglist, set, and dict comprehensionsBasic list comprehensionsset comprehensionsdict comprehensionsComprehension pitfallslambda functionsThe Y combinatorfunctoolspartial – Prefill function argumentsreduce – Combining pairs into a single resultImplementing a factorial functionProcessing treesReducing in the other direction itertoolsaccumulate – reduce with intermediate resultschain – Combining multiple resultscompress – Selecting items using a list of Booleansdropwhile/takewhile – Selecting items using a functioncount – Infinite range with decimal stepsgroupby – Grouping your sorted iterableExercisesSummary
Decorators – Enabling Code Reuse by Decorating
Decorating functionsGeneric function decoratorsThe importance of functools.wrapsChaining or nesting decoratorsRegistering functions using decoratorsMemoization using decoratorsDecorators with (optional) argumentsCreating decorators using classesDecorating class functionsSkipping the instance – classmethod and staticmethodProperties – Smart descriptor usageDecorating classesSingletons – Classes with a single instanceTotal ordering – Making classes sortableUseful decoratorsSingle dispatch – Polymorphism in Pythoncontextmanager — with statements made easyValidation, type checks, and conversionsUseless warnings – How to ignore them safelyExercisesSummary
Generators and Coroutines – Infinity, One Step at a Time
GeneratorsCreating generatorsCreating infinite generatorsGenerators wrapping iterablesGenerator comprehensionsClass-based generators and iteratorsGenerator examplesBreaking an iterable up into chunks/groupsitertools.islice – Slicing iterablesitertools.chain – Concatenating multiple iterablesitertools.tee – Using an output multiple timescontextlib.contextmanager – Creating context managersCoroutinesA basic examplePrimingClosing and throwing exceptionsMixing generators and coroutinesUsing the stateExercisesSummary
Metaclasses – Making Classes (Not Instances) Smarter
Dynamically creating classesA basic metaclassArguments to metaclassesAccessing metaclass attributes through classesAbstract classes using collections.abcInternal workings of the abstract classesCustom type checksAutomatically registering plugin systemsImporting plugins on-demandImporting plugins through configurationImporting plugins through the filesystemDataclassesOrder of operations when instantiating classesFinding the metaclassPreparing the namespaceExecuting the class bodyCreating the class object (not instance)Executing the class decoratorsCreating the class instanceExampleStoring class attributes in definition orderThe classic solution without metaclassesUsing metaclasses to get a sorted namespaceExercisesSummary
Documentation – How to Use Sphinx and reStructuredText
Type hintingBasic exampleCustom typesGenericsType checkingPython type interface filesType hinting conclusionreStructuredText and MarkdownGetting started with reStructuredTextGetting started with MarkdownInline markupHeadersHeaders with reStructuredTextHeaders with MarkdownListsEnumerated listsBulleted listsOption listsDefinition lists (reST only)Nested listsLinks, references, and labelsImagesImages with reStructuredTextImages with MarkdownSubstitutionsBlocks, code, math, comments, and quotesConclusionThe Sphinx documentation generatorGetting started with SphinxUsing sphinx-quickstartUsing sphinx-apidocSphinx directivesSphinx rolesDocumenting codeDocumenting a class with the Sphinx styleDocumenting a class with the Google styleDocumenting a class with the NumPy styleWhich style to chooseExercisesSummary

Testing and Logging – Preparing for Bugs
Using documentation as tests with doctestA simple doctest exampleWriting doctestsTesting with documentationThe doctest flagsTrue and False versus 1 and 0Normalizing whitespaceEllipsisDoctest quirksTesting dictionariesTesting floating-point numbersTimes and durationsTesting with py.testThe difference between the unittest and py.test outputThe difference between unittest and py.test testsSimplifying assertionsParameterizing testsAutomatic arguments using fixturesPrint statements and loggingPluginsMock objectsUsing unittest.mockUsing py.test monkeypatchTesting multiple environments with toxGetting started with toxThe tox.ini config fileRunning toxLoggingConfigurationBasic logging configurationDictionary configurationJSON configurationini file configurationThe network configurationLoggerUsageFormattingModern formatting using f-strings and str.formatLogging pitfallsDebugging loggersExercisesSummary
Debugging – Solving the Bugs
Non-interactive debuggingInspecting your script using traceDebugging using loggingShowing the call stack without exceptionsHandling crashes using faulthandlerInteractive debuggingConsole on demandDebugging using Python debugger (pdb)BreakpointsCatching exceptionsAliasescommandsDebugging with IPythonDebugging with JupyterOther debuggersDebugging servicesExercisesSummary
Performance – Tracking and Reducing Your Memory and CPU Usage
What is performance?Measuring CPU performance and execution timeTimeit – comparing code snippet performancecProfile – Finding the slowest componentsFirst profiling runCalibrating your profilerSelective profiling using decoratorsUsing profile statisticsLine profiler – Tracking performance per lineImproving execution timeUsing the right algorithmGlobal interpreter locktry versus ifLists versus generatorsString concatenationAddition versus generatorsMap versus generators and list comprehensionsCachingLazy importsUsing slotsUsing optimized librariesJust-in-time compilingConverting parts of your code to CMemory usagetracemallocMemory ProfilerMemory leaksCircular referencesAnalyzing memory usage using the garbage collectorWeak referencesWeakref limitations and pitfallsReducing memory usageGenerators versus listsRecreating collections versus removing itemsUsing slotsPerformance monitoringExercisesSummary
asyncio – Multithreading without Threads
Introduction to asyncioBackward compatibility and async/await statementsPython 3.4Python 3.5Python 3.7A basic example of parallel executionasyncio conceptsCoroutines, Futures, and TasksEvent loopsExecutorsAsynchronous examplesProcessesInteractive processesEcho client and serverAsynchronous file operationsCreating async generators to support async forAsynchronous constructors and destructorsDebugging asyncioForgetting to await a coroutineSlow blocking functionsForgetting to check the results or exiting earlyExiting before all tasks are doneExercisesSummary
Multiprocessing – When a Single CPU Core Is Not Enough
The Global Interpreter Lock (GIL)The use of multiple threadsWhy do we need the GIL?Why do we still have the GIL?Multiple threads and processesBasic examplesconcurrent.futuresthreadingmultiprocessingCleanly exiting long-running threads and processesBatch processing using concurrent.futuresBatch processing using multiprocessingSharing data between threads and processesShared memory between processesThread safetyDeadlocksThread-local variablesProcesses, threads, or a single thread?threading versus concurrent.futuresmultiprocessing versus concurrent.futuresHyper-threading versus physical CPU coresRemote processesDistributed processing using multiprocessingDistributed processing using DaskInstalling DaskBasic exampleRunning a single threadDistributed execution across multiple machinesDistributed processing using ipyparallelipython_config.pyipython_kernel_config.pyipcontroller_config.pyipengine_config.pyipcluster_config.pySummary
Scientific Python and Plotting
Installing the packagesArrays and matricesNumPy – Fast arrays and matricesNumba – Faster Python on CPU or GPUSciPy – Mathematical algorithms and NumPy utilitiesSparse matricesPandas – Real-world data analysisInput and output optionsPivoting and groupingMergingRolling or expanding windowsStatsmodels – Statistical models on top of Pandasxarray – Labeled arrays and datasetsSTUMPY – Finding patterns in time seriesMathematics and precise calculationsgmpy2 – Fast and precise calculationsSage – An alternative to Mathematica/Maple/MATLABmpmath – Convenient, precise calculationsSymPy – Symbolic mathematicsPatsy – Describing statistical models Plotting, graphing, and chartingMatplotlibSeabornYellowbrickPlotlyBokehDatashaderExercisesSummary
Artificial Intelligence
Introduction to artificial intelligenceTypes of AIInstalling the packagesImage processing scikit-imageInstalling scikit-imageEdge detectionFace detectionscikit-image overviewOpenCVInstalling OpenCV for PythonEdge detectionObject detectionOpenCV versus scikit-imageNatural language processingNLTK – Natural Language ToolkitspaCy – Natural language processing with CythonGensim – Topic modeling for humansMachine learningTypes of machine learningSupervised learningReinforcement learningUnsupervised learningCombinations of learning methodsDeep learningArtificial neural networks and deep learningTensorsPyTorch – Fast (deep) neural networksPyTorch Lightning and PyTorch Ignite – High-level PyTorch APIsSkorch – Mixing PyTorch and scikit-learnTensorFlow/Keras – Fast (deep) neural networksTensorFlow versus PyTorchEvolutionary algorithmsSupport-vector machinesBayesian networksVersatile AI libraries and utilitiesscikit-learn – Machine learning in PythonSupervised learningUnsupervised learningauto-sklearn – Automatic scikit-learnmlxtend – Machine learning extensionsscikit-lego – scikit-learn utilitiesXGBoost – eXtreme Gradient BoostingFeaturetools – Feature detection and predictionSnorkel – Improving your ML data automaticallyTPOT – Optimizing ML models using genetic programmingExercisesSummary
Extensions in C/C++, System Calls, and C/C++ Libraries
Setting up toolingDo you need C/C++ modules?WindowsOS XLinux/UnixCalling C/C++ with ctypesPlatform-specific librariesWindowsLinux/UnixOS XMaking it easyCalling functions and native typesComplex data structuresArraysGotchas with memory managementCFFIComplex data structuresArraysABI or API?CFFI or ctypes?Native C/C++ extensionsA basic exampleC is not Python – Size mattersThe example explainedstaticPyObject*Parsing argumentsC is not Python – Errors are silent or lethalCalling Python from C – Handling complex typesExercisesSummary
Packaging – Creating Your Own Libraries or Applications
Introduction to packagesTypes of packagesWheels – The new eggsSource packagesPackage toolsPackage versioningBuilding packagesPackaging using pyproject.tomlCreating a basic packageInstalling packages for developmentAdding code and dataAdding executable commandsManaging dependenciesBuilding the packageBuilding C/C++ extensionsPackaging using setuptools with setup.py or setup.cfgCreating a basic packageInstalling the package for developmentAdding packagesAdding package dataManaging dependenciesAdding executable commandsBuilding the packagePublishing packagesAdding URLsPyPI trove classifiersUploading to PyPIC/C++ extensionsRegular C/C++ extensionsCython extensionsTestingunittestpy.testExercisesSummary
Other Books You May Enjoy
Index

Content preview from Mastering Python 2E - Second Edition

14 Multiprocessing – When a Single CPU Core Is Not Enough

In the previous chapter, we discussed asyncio, which can use the threading and multiprocessing modules but mainly uses single-thread/single-process parallelization. In this chapter, we will see how we can directly use multiple threads or processes to speed up our code and what caveats to keep in mind. This chapter can actually be seen as an extension to the list of performance tips.

The threading module makes it possible to run code in parallel in a single process. This makes threading very useful for I/O-related tasks such as reading/writing files or network communication, but a useless option for slow and heavy calculations, which is where the multiprocessing module shines.

With the ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Advanced Python Programming - Second Edition

Publisher Resources

ISBN: 9781800207721

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Mastering Python 2E - Second Edition

by Rick Hattem

14

Multiprocessing – When a Single CPU Core Is Not Enough

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.