book

Expert Python Programming - Fourth Edition

by Michał Jaworski, Tarek Ziade, Tarek Ziadé

May 2021

Intermediate to advanced

630 pages

15h 31m

English

Packt Publishing

Read now

Unlock full access

Who this book is forWhat this book coversTo get the most out of this bookGet in touch
Where are we now and where are we going?What to do with Python 2Keeping up to datePEP documentsActive communitiesOther resourcesSummary
Technical requirementsPython's packaging ecosystemInstalling Python packages using pipIsolating the runtime environmentApplication-level isolation versus system-level isolationApplication-level environment isolationPoetry as a dependency management systemSystem-level environment isolationContainerization versus virtualizationVirtual environments using DockerWriting your first DockerfileRunning containersSetting up complex environmentsUseful Docker and Docker Compose recipes for PythonVirtual development environments using VagrantPopular productivity toolsCustom Python shellsUsing IPythonIncorporating shells in your own scripts and programsInteractive debuggersOther productivity toolsSummary
Technical requirementsRecent language additionsDictionary merge and update operatorsAlternative – Dictionary unpackingAlternative – ChainMap from the collections moduleAssignment expressionsType-hinting genericsPositional-only parameterszoneinfo modulegraphlib moduleNot that new, but still shinybreakpoint() functionDevelopment modeModule-level __getattr__() and __dir__() functionsFormatting strings with f-stringsUnderscores in numeric literalssecrets moduleWhat may come in the future?Union types with the | operatorStructural pattern matchingSummary
Technical requirementsClass model and object-oriented programming Accessing super-classesMultiple inheritance and Method Resolution OrderClass instance initializationAttribute access patternsDescriptorsReal-life example – lazily evaluated attributesPropertiesDynamic polymorphismOperator overloadingDunder methods (language protocols)Comparison to C++Function and method overloadingSingle-dispatch functionsData classesFunctional programmingLambda functionsThe map(), filter(), and reduce() functionsPartial objects and partial functionsGeneratorsGenerator expressionsDecoratorsEnumerationsSummary
Technical requirementsInterfacesA bit of history: zope.interfaceUsing function annotations and abstract base classesUsing collections.abcInterfaces through type annotationsInversion of control and dependency injectionInversion of control in applicationsUsing dependency injection frameworksSummary
Technical requirementsWhat is concurrency?MultithreadingWhat is multithreading?How Python deals with threadsWhen should we use multithreading?Application responsivenessMultiuser applicationsWork delegation and background processingAn example of a multithreaded applicationUsing one thread per itemUsing a thread poolUsing two-way queuesDealing with errors in threadsThrottlingMultiprocessingThe built-in multiprocessing moduleUsing process poolsUsing multiprocessing.dummy as the multithreading interfaceAsynchronous programmingCooperative multitasking and asynchronous I/OPython async and await keywordsA practical example of asynchronous programmingIntegrating non-asynchronous code with async using futuresExecutors and futuresUsing executors in an event loopSummary
Technical requirementsWhat exactly is event-driven programming?Event-driven != asynchronousEvent-driven programming in GUIsEvent-driven communicationVarious styles of event-driven programmingCallback-based styleSubject-based styleTopic-based styleEvent-driven architecturesEvent and message queuesSummary
Technical requirementsWhat is metaprogramming?Using decorators to modify function behavior before use One step deeper: class decoratorsIntercepting the class instance creation processMetaclassesThe general syntaxMetaclass usageMetaclass pitfallsUsing the __init__subclass__() method as an alternative to metaclassesCode generationexec, eval, and compileThe abstract syntax tree Import hooksNotable examples of code generation in PythonFalcon's compiled routerHySummary
Technical requirementsC and C++ as the core of Python extensibilityCompiling and loading Python C extensionsThe need to use extensionsImproving performance in critical code sectionsIntegrating existing code written in different languagesIntegrating third-party dynamic librariesCreating efficient custom datatypesWriting extensionsPure C extensionsA closer look at the Python/C APICalling and binding conventionsException handlingReleasing GILReference countingWriting extensions with CythonCython as a source-to-source compilerCython as a languageDownsides of using extensionsAdditional complexityHarder debuggingInterfacing with dynamic libraries without extensionsThe ctypes moduleLoading librariesCalling C functions using ctypesPassing Python functions as C callbacksCFFISummary

Technical requirementsThe principles of test-driven developmentWriting tests with pytestTest parameterizationpytest's fixturesUsing fakesMocks and the unittest.mock moduleQuality automationTest coverageStyle fixers and code lintersStatic type analysisMutation testingUseful testing utilitiesFaking realistic data valuesFaking time valuesSummary
Technical requirementsPackaging and distributing librariesThe anatomy of a Python packagesetup.pysetup.cfgMANIFEST.inEssential package metadataTrove classifiersTypes of package distributionssdist distributionsbdist and wheel distributionsRegistering and publishing packagesPackage versioning and dependency managementThe SemVer standard for semantic versioningCalVer for calendar versioningInstalling your own packagesInstalling packages directly from sourcesInstalling packages in editable modeNamespace packagesPackage scripts and entry pointsPackaging applications and services for the webThe Twelve-Factor App manifestoLeveraging DockerHandling environment variablesThe role of environment variables in application frameworksCreating standalone executablesWhen standalone executables are usefulPopular toolsPyInstallercx_Freezepy2exe and py2appSecurity of Python code in executable packagesSummary
Technical requirementsCapturing errors and logsPython logging essentialsLogging system componentsLogging configurationGood logging practicesDistributed loggingCapturing errors for later reviewInstrumenting code with custom metricsUsing PrometheusDistributed application tracingDistributed tracing with JaegerSummary
Technical requirementsCommon culprits for bad performanceCode complexityCyclomatic complexityThe big O notationExcessive resource allocation and leaksExcessive I/O and blocking operationsCode profilingProfiling CPU usageMacro-profilingMicro-profilingProfiling memory usageUsing the objgraph moduleC code memory leaksReducing complexity by choosing appropriate data structuresSearching in a listUsing setsUsing the collections moduledequedefaultdictnamedtupleLeveraging architectural trade-offsUsing heuristics and approximation algorithmsUsing task queues and delayed processingUsing probabilistic data structuresCachingDeterministic cachingNon-deterministic cachingSummary

Content preview from Expert Python Programming - Fourth Edition

13 Code Optimization

Code optimization is the process of making an application work more efficiently, usually without modifying its functionality and accuracy. Code optimization is usually concerned with the speed of processing (CPU time optimization), but can also be used to minimize the usage of different resources, such as memory, disk space, or network bandwidth.

In the previous chapter, we learned about various observability techniques that can be used to identify performance hotspots in applications. Logging, metric monitoring, and tracing can be used to create a general performance overview and prioritize optimization efforts. Moreover, operations teams often use custom alerts of performance and resource usage metrics or log messages ...