book

Advanced Python Programming - Second Edition

by Quan Nguyen

March 2022

Intermediate to advanced

606 pages

12h 25m

English

Packt Publishing

Read now

Unlock full access

Second EditionContributorsAbout the authorAbout the reviewers
Who this book is forWhat this book coversTo get the most out of this bookDownload the example code filesDownload the color imagesConventions usedGet in touchShare Your Thoughts
Technical requirementsDesigning your applicationBuilding a particle simulatorVisualizing the simulationWriting tests and benchmarksTiming your benchmarkWriting better tests and benchmarks with pytest-benchmarkFinding bottlenecks with cProfileGraphically analyzing profiling resultsProfiling line by line with line_profilerOptimizing our codeUsing the dis moduleProfiling memory usage with memory_profilerSummaryQuestions Further reading
Technical requirementsUsing the right algorithms and data structuresLists and dequesDictionariesHash mapSetsHeapsTriesImproved efficiency with caching and memoizationJoblibEfficient iteration with comprehensions and generatorsSummaryQuestionsFurther reading
Technical requirementGetting started with NumPyCreating arraysAccessing arraysIndexing and slicingFancy indexingBroadcastingMathematical operationsCalculating the normRewriting the particle simulator in NumPyReaching optimal performance with numexprWorking with database-style data with pandaspandas fundamentalsIndexing Series and DataFrame objectsDatabase-style operations with pandasMappingGrouping, aggregations, and transformsJoiningHigh-performance labeled data with xarrayAnalyzing concentrationThe xarray libraryImproved performancePlotting with xarraySummaryQuestionsFurther reading
Technical requirementsCompiling Cython extensionsAdding static typesDeclaring variablesDeclaring functionsDeclaring classesSharing declarationsWorking with arraysC arrays and pointersWorking with NumPy arraysWorking with typed memoryviewsUsing a particle simulator in CythonProfiling CythonUsing Cython with JupyterSummaryQuestions
Technical requirementsGetting started with NumbaUsing Numba decoratorsType specializationsObject mode versus native modeNumba and NumPyUniversal functions with NumbaGeneralized universal functionsJIT classesLimitations in NumbaThe PyPy projectSetting up PyPyRunning a particle simulator in PyPyOther interesting projectsSummaryQuestionsFurther reading
A crash course in machine learningModel parametersLoss functionLoss minimizationGetting JAX up and runningInstalling JAXUsing Google ColabAutomatic differentiation for loss minimizationMaking the datasetBuilding a linear modelGradient descent with automatic differentiationJust-In-Time compilation for improved efficiencyAutomatic vectorization for efficient kernelsData that is not linearly separableThe kernel method in machine learningAutomatic vectorization for kernelized modelsSummaryQuestionsFurther reading

Technical requirementsAsynchronous programmingWaiting for input/outputConcurrencyCallbacksFuturesEvent loopsThe asyncio frameworkCoroutinesConverting blocking code into non-blocking codeReactive programmingObservablesUseful operatorsHot and cold observablesBuilding a CPU monitorSummaryQuestionsFurther reading
Technical requirementsIntroduction to parallel programmingGPUsUsing multiple processesThe Process and Pool classesThe Executor interfaceMonte Carlo approximation of piSynchronization and locksParallel Cython with OpenMPAutomatic parallelismGetting started with TheanoProfiling TheanoTensorFlowRunning code on a GPUSummaryQuestions
The basics of web requestsHTMLHTTP requestsHTTP status codeThe requests moduleMaking a request in PythonRunning a ping testConcurrent web requestsSpawning multiple threadsRefactoring request logicThe problem with timeoutsSupport from httpstat.us and simulation in PythonTimeout specificationsGood practices in making web requestsConsider the terms of service and data-collecting policiesError handlingUpdate your program regularlyAvoid making a large number of requestsSummaryQuestionsFurther reading
Technical requirementsImage processing fundamentalsPython as an image processing toolInstalling OpenCV and NumPyComputer image basicsRGB valuesPixels and image filesCoordinates inside an imageOpenCV APIImage processing techniquesGrayscalingThresholdingApplying concurrency to image processingGood concurrent image processing practicesChoosing the correct way (out of many)Spawning an appropriate number of processesProcessing input/output concurrentlySummaryQuestionsFurther reading
Technical requirementsThe ecosystem of communication channelsCommunication protocol layersAsynchronous programming for communication channelsTransports and protocols in asyncioThe big picture of asyncio's server clientGetting started with Python and TelnetStarting a serverInstalling TelnetSimulating a connection channelSending messages back to clientsClosing transportsClient-side communication with aiohttpInstalling aiohttp and aiofilesFetching a website's HTML codeWriting files asynchronouslySummaryQuestionsFurther reading
Technical requirementsThe concept of deadlocksThe dining philosophers problemA deadlock in a concurrent systemPython simulationApproaches to deadlock situationsImplementing ranking among resourcesIgnoring locks and sharing resourcesAn additional note about locksConcluding note on deadlock solutionsThe concept of livelocksSummaryQuestionsFurther reading
Technical requirementsUnderstanding starvationWhat is starvation?SchedulingCauses of starvationStarvation's relationship to deadlockApproaching the readers-writers problemProblem statementThe first readers-writers problemThe second readers-writers problemThe third readers-writers problemSolutions to starvationSummaryQuestionsFurther reading
Technical requirementsThe concept of race conditionsCritical sectionsHow race conditions occurSimulating race conditions in PythonLocks as a solution to race conditionsThe effectiveness of locksImplementation in PythonThe downside of locksTurning a concurrent program into a sequential programLocks do not lock anythingRace conditions in real lifeSecurityOperating systemsNetworkingSummaryQuestionsFurther reading
Technical requirementsIntroducing the GILAnalyzing memory management in PythonThe problem that the GIL addressesProblems raised by the GILThe potential removal of the GIL from PythonWorking with the GILImplementing multiprocessing, rather than multithreading Getting around the GIL with native extensionsUtilizing a different Python interpreterSummaryQuestionsFurther reading
Technical requirementsUnderstanding design patternsImplementing the factory methodReal-world examples Use casesImplementing the factory methodApplying the abstract factoryReal-world examplesUse casesImplementing the abstract factory patternSummaryQuestions
Technical requirementsUnderstanding the builder patternReal-world examplesUse casesImplementing an ordering applicationSummaryQuestions
Technical requirementsImplementing the prototype patternReal-world examplesUse casesImplementationImplementing the singleton patternReal-world examplesUse casesImplementationSummaryQuestionsFurther reading
Technical requirementsUnderstanding the adapter patternReal-world examplesUse casesImplementationSummary
Technical requirementsIntroducing the decorator patternReal-world examplesUse casesImplementationSummaryQuestions
Technical requirementsReal-world examplesUse casesImplementationSummaryQuestions
Technical requirementsUnderstanding the façade patternReal-world examplesUse casesImplementationSummaryQuestionsFurther reading
Technical requirementsImplementing the flyweight patternReal-world examplesUse casesImplementationImplementing the model-view-controller patternReal-world examplesUse casesImplementationApplying the proxy patternReal-world examplesUse casesImplementationSummaryQuestions
Technical requirementsUnderstanding the Chain of Responsibility patternReal-world examplesUse casesImplementationSummaryQuestions
Technical requirementsUnderstanding the command patternReal-world examplesUse casesImplementationSummaryQuestions
Technical requirementsUnderstanding the observer patternReal-world examplesUse casesImplementationSummaryQuestions
Chapter 1Chapter 2Chapter 3Chapter 4Chapter 5Chapter 6Chapter 7Chapter 8Chapter 9Chapter 10Chapter 11Chapter 12Chapter 13Chapter 14Chapter 15Chapter 16Chapter 17Chapter 18Chapter 20Chapter 21Chapter 22Chapter 23Chapter 24Chapter 25Chapter 26
Other Books You May EnjoyPackt is searching for authors like youShare Your Thoughts

Content preview from Advanced Python Programming - Second Edition

Chapter 8: Parallel Processing

With parallel processing using multiple cores, you can increase the number of calculations your program can do in a given time frame without needing a faster processor. The main idea is to divide a problem into independent subunits and use multiple cores to solve those subunits in parallel.

Parallel processing is necessary to tackle large-scale problems. Every day, companies produce massive quantities of data that needs to be stored in multiple computers and analyzed. Scientists and engineers run parallel code on supercomputers to simulate massive systems.

Parallel processing allows you to take advantage of multicore central processing units (CPUs) as well as graphics processing units (GPUs) that work extremely ...