book

Deep Dive Python: Techniques and Best Practices for Developers

Name: Deep Dive Python: Techniques and Best Practices for Developers
Author: Adarsh Divakaran
ISBN: 9798868812613

by Adarsh Divakaran

August 2025

Beginner to intermediate

769 pages

12h 51m

English

Apress

Read now

Unlock full access

Deep Dive Python
Introduction
Acknowledgments
Table of Contents
About the Author
About the Technical Reviewer
1. Lists
Initializing Empty ListsIndexingSlicingDynamic SlicesExample from scikit-learnSnippet from IPythonFull SliceExample from PandasList ComprehensionsWith Conditional FilteringWith Tuple UnpackingNested ComprehensionsNested LoopsMultidimensional ListsCopying ListsShallow CopyDeep CopySorting ListsBuilt-In Key FunctionsThe attrgetter FunctionThe itemgetter FunctionThe methodcaller FunctionThe cmp_to_key FunctionStability of sortCustom List ClassesInheriting from the list ClassSQLAlchemyInheriting from UserListAndroguardOpenMMLab mmrazorMutableSequence from collections.abcChoosing Between list, UserList, and MutableSequenceConclusion
2. Tuples
Initializing TuplesImmutabilityLists vs. TuplesTuple Packing and UnpackingTuple PackingTuple UnpackingVariable-Length AssignmentUnpacking in LoopsIgnoring ValuesComparisons and SortingVersion CheckingAs Sort KeysSorting TuplesTuples as Dict Keysnamedtuple (collections.namedtuple)typing.NamedTupleConclusion
3. Set and Dictionary
Set in PythonDict in PythonComprehensionsDictionary ComprehensionsSet ComprehensionsHashabilityHashability in SetsHashability in DictionariesHash CollisionsCustom Hashable ObjectsMore Set OperationsSymmetric DifferenceSubset and Superset ChecksDisjoint Set CheckClearing a SetThe pop MethodUpdate MethodsThe update MethodIntersection UpdateDifference UpdateSymmetric Difference UpdateMore Dictionary OperationsThe setdefault() MethodThe popitem() MethodThe update() MethodThe fromkeys() MethodDictionary View ObjectsDictionary Packing and UnpackingPacking **kwargs into a DictionaryDictionary UnpackingDictionary Merging and UnpackingSet Variant: frozensetDictionary: Variantscollections.defaultdictcollections.OrderedDictcollections.Countercollections.ChainMapCustom Dictionary ClassesFrom the Built-In dictFrom MutableMappingFrom UserDictCustom Set Class: MutableSetTyping Sets and DictsTyping DictionariesTyping SetsTypedDictConclusion
4. Logging
ConfigurationUsing logging.basicConfigDictConfigFileConfigLog LevelsUsing logging.error vs. logging.exceptionLoggersFormattersHandlersSMTPHandlerStreamHandlerLog Rotation and Timed File HandlersRotatingFileHandlerTimedRotatingFileHandlerCustom HandlersLogRecordFiltersLoggerAdapterLog PropagationThe Logging FlowLogger FlowHandler FlowLogger PropagationLogging in LibrariesConclusion

5. Exceptions
Exceptions in PythonBuilt-In ExceptionsTracebacksException Base Classes and Exception HierarchyThe BaseException and Exception ClassesBaseExceptionExceptionBuilt-In Exception HierarchyHandling Exceptions: The try BlockException ChainingThe __cause__, __context__, and __suppress_context__ Attributes__cause____context____suppress_context__Raise from ExceptionRaise from NoneExceptions in Threads and ProcessesExceptions in ThreadsHandling Exceptions in ThreadPoolExecutorExceptions in Multiprocessing and ProcessPoolExecutorCustom Exception ClassesThe traceback Moduletraceback.print_stacktraceback.print_exctraceback.format_exctraceback.format_exception_onlysys module: excepthook and exc_infosys.exc_infosys.excepthookZero-Cost ExceptionsConclusion
6. Functions and Functools
Positional and Keyword ArgumentsPositional-Only ArgumentsKeyword-Only ArgumentsArgument-Passing Model in PythonPassing Immutable ObjectsPassing Mutable ObjectsArgument Unpacking and PackingArgument UnpackingArgument PackingRecursionDefault ArgumentsNested Functions/ClosuresThe nonlocal KeywordHigher-Order FunctionsmapfilterFunction IntrospectionAccessing Function SignatureAccessing AnnotationsFunction Attributes and DocstringThe functools Modulepartialpartialmethodlru_cachecached_propertytotal_orderingreducesingledispatchsingledispatchmethodConclusion
7. Classes and Object-Oriented Programming
Class MethodsStatic MethodNamespace DictionaryPropertyPrivate Attributes and Name ManglingName ManglingSlotted ClassesAttribute Access Using __getattr__ and __getattribute__DataclassWhy a DataclassFrozen DataclassDataclass FieldPost InitInheritanceInheritance in PythonMultiple InheritanceMultilevel InheritanceMethod Resolution OrderThe super Built-InUsing super with Multiple InheritanceThe object Built-InAlternatives to InheritanceMixinsWhen Not to Use ClassesConclusion
8. Dunder Methods
__str__ and __repr__ Dunder Methods__add__ and __sub__ : Addition and Subtraction Operations__eq__ and __lt__: Equality Check and Comparison__int__ and __float__: Conversion to Numbers__len__: Length Calculation__call__: Callable Objects__hash__: Hashability and Hash Calculation__contains__, __getitem__, and __setitem__: Membership Checks and Item Access__getattr__ and __getattribute__: Attribute Access__setattr__: Setting Attributes__missing__: Handle Missing Keys__copy__ and __deepcopy__: Copying Objects__instancecheck__ and __subclasscheck__: Instance and Subclass ChecksConclusion
9. Decorators
ClosuresDecorators: Syntactic SugarUsage of functools.wrapsPractical DecoratorsNesting DecoratorsDecorators with ArgumentsDecorators with Optional Arguments OnlyDecorating ClassesUsing Classes as DecoratorsDecorators with Asyncio SupportContext Management DecoratorsDecorator Use CasesCachingTimingRegisteringValidationConditional ExecutionLoggingException HandlingConclusion
10. Metaclasses
Class Creation Processtype in PythonCalling with a Single ArgumentCalling with Three ArgumentsDefining MetaclassesMetamethodsDefining a Custom MetaclassMethod PropagationInheritance BehaviorMetaclass Determination OrderMetaclass Usage PatternsRegistrationValidation or EnforcementModifying Instance CreationModifying Instance FeaturesUsage in EnumMetaclass AlternativesPEP 487Class DecoratorsConclusion
11. Typing
Type CheckingAnnotating VariablesProtocolsTypeVarStatic Type Checkingurllib3Runtime Type CheckingNebulyLibraries Leveraging Type HintsPydanticSQLModelType StubsAdopting Type Annotations in Your ProjectConclusion
12. Generators and Iterators
IteratorsThe Iterator Protocolnext() and __iter__ MethodsThe iter() Function and __iter__() Dunder MethodThe next() Function and __next__() Dunder MethodConverting Functions to Iterators Using iter()Async IteratorsThe Async Iterator ProtocolGeneratorsGenerator vs. IteratorsYield and Yield FromYieldYield FromAsync GeneratorsGenerator Methods: send(), throw(), and close()send()throw()close()Generator ComprehensionConclusion
13. Itertools
countcyclerepeatproductpermutationscombinationsaccumulatebatchedchaincompressdropwhilefilterfalsegroupbyislicepairwisestarmaptakewhileteezip_longestConclusion
14. Multithreading
Thread Class for Creating ThreadsImplications of the GIL on ThreadingDaemon ThreadsThread-Local Datathreading.TimerThreadPoolExecutorHandling Exceptions in ThreadsCustom Thread ClassesThread SafetyRace ConditionsLocks: Mutual ExclusionLockDeadlockRLockSemaphores: Limiting AccessEvents: SignalingConditions: Conditional WaitingConclusion
15. Multiprocessing
Processes vs. ThreadsBypassing the Global Interpreter Lock (GIL)When to Use MultiprocessingCreating and Managing ProcessesThe multiprocessing.Process ClassStarting Processes with start()Waiting for Processes to Finish with join()Custom Processes with a ClassDaemon ProcessesChecking Process InformationInter-process Communication (IPC) and Data SharingWhy Processes Don’t Share MemoryShared Memory with multiprocessing.Value and multiprocessing.ArrayExample: Using Value to Share a CounterExample: Using Array to Share a List of NumbersMessage Passing with Pipes (multiprocessing.Pipe)Example: Sending a Message from Child to ParentMessage Passing with Queues (multiprocessing.Queue and multiprocessing.SimpleQueue)Example: Multiple Processes Sending Results to a Shared QueueManagers for Shared ObjectsExample: Using a Manager to Share a DictionaryProcess Pools for Task DistributionIntroduction to multiprocessing.PoolExample: Using pool.map() to Compute Squares in ParallelSubmitting Tasks to the Pool: apply_async() and map_async()apply_async(func, args=(), kwds={}, callback=None, error_callback=None)map_async(func, iterable, chunksize=None, callback=None, error_callback=None)Example: Using apply_async() and map_async()concurrent.futures.ProcessPoolExecutor: A Modern AlternativeExample: Using ProcessPoolExecutor to Compute SquaresHandling Results and Exceptions in PoolsExample: Handling Exceptions from Worker FunctionsComparing Pool and Executor with ThreadsProcess Start Methods: Choosing the Right ApproachExplanation of Start Methods: spawn, fork, forkserverspawnforkforkserverWhen to Use Each Start Method and Platform DependenciesWhen to Choose spawnWhen to Choose forkWhen to Choose forkserverSetting the Start Method in CodeSynchronization Between ProcessesNeed for Synchronization Primitives in MultiprocessingOverview and Usage of Synchronization PrimitivesLocks and RLocks (multiprocessing.Lock, multiprocessing.RLock)Example: Protecting a Shared CounterSemaphores (multiprocessing.Semaphore, multiprocessing.BoundedSemaphore)Events (multiprocessing.Event)Example: Starting All Workers at the Same TimeConditions (multiprocessing.Condition)Example: Signaling a Consumer When a Queue Is Non-emptyBarrier (multiprocessing.Barrier)Example: Synchronizing Three ProcessesReal-World Use: Shared Locks with ManagersExample: Locking Access to a Shared DictionaryUsing Synchronization Primitives with Context ManagersShared State vs. Message PassingConclusion
16. Asyncio
What Is Asynchronous Programming?Event Loop and Cooperative Multitaskingasync and await KeywordsWhen to Use Asyncio (I/O Bound vs. CPU Bound)Benefits of AsyncioContext Switching in AsyncioCreating and Running Asyncio Programsasync def for Coroutinesasyncio.run() to Start the Event LoopBasic Example: Concurrent Network RequestsWorking with TasksCreating Tasks with asyncio.create_task()Task CancellationSynchronization PrimitivesAsyncio LockAsyncio SemaphoreAsyncio EventAsyncio ConditionContext VariablesThe Need for Context Variables in AsyncioCreating and Using Context VariablesContext Variables and Task HierarchiesDefault Values and Token ManagementConcurrency Patterns in AsyncioFan-Out/Fan-InProducer–ConsumerWorker PoolError Handling in AsyncioTry–Except Blocks in CoroutinesExceptions in TasksHandling asyncio.CancelledErrorBest PracticesMultithreading vs. Multiprocessing vs. AsyncioMultithreadingMultiprocessingAsyncioSummary TableConclusion
17. Data Serialization and Persistence
JSONjson.dumps()json.loads()json.dump() and json.load()JSONEncoder ClassJSONDecoder ClassPicklePickle Module Functionspickle.dumppickle.loadpickle.loads() and pickle.dumps()Pickler and Unpickler ClassesPicklabilityPickling LimitationsPickling Class RestrictionsClass Constructor BehaviorPickling Function RestrictionsPickling Stateful ObjectsThe __getstate__() MethodThe __setstate__() MethodPickle vs. JSONHuman ReadabilityInteroperabilityType SupportSecurityPerformanceShelveKey MethodsShelf Object MethodsShelve vs. PickleDifferences Between shelve and pickleMarshalCSVThe reader() and writer() FunctionsThe DictReader() and DictWriter() ClassesCSV Dialects and SnifferXMLxml.etree.ElementTree (etree)xml.domxml.saxThe etree ModuleConclusion
18. Context Managers and Contextlib
Context Managers in the Standard LibraryFile HandlingLocks for Thread SafetyManaging Thread PoolsSQLite Database Connection ManagementDefining Custom Context Managers Using DundersContext Manager Base ClassesAbstractAsyncContextManagerDecorators for Defining a Context Manager@contextmanager@asynccontextmanagerContext Manager as a Decoratorcontextlib.ExitStack()AsyncExitStackBuilt-In Context Managers from contextlib.suppressclosingredirect_stdoutUse CasesTimerPatchingConclusion
19. Abstract Base Classes
abc.ABC and abc.ABCMetaThe abc.ABC ClassThe abc.ABCMeta MetaclassABC Method Decorators@abstractmethod DecoratorDecorating Class MethodsDecorating Static MethodsDecorating Property MethodsSubclassing ABCsUse of isinstance() and issubclass() with ABCsMultiple Inheritance with ABCsThe register() Method: Registering Virtual SubclassesThe __subclasshook__() MethodThe collections.abc Modulecollections.abc.MutableSequencecollections.abc.Iterablecollections.abc.Iteratorcollections.abc.CoroutineConclusionWhen to Use Abstract Base ClassesWhen to Avoid Abstract Base ClassesABCs vs. Protocols
20. Packaging
PyPIHistoryEgg FilesDistutilsWheelpyproject.tomlToolsSetuptoolsFlitPoetryHatchTwineBest Practices__init__.py Files in Python PackagesExample: __init__.py in ActionBest Practices for __init__.pyCode Organization: Structuring Packages for Clarity and MaintainabilityManaging Private Python Packages for Internal UseHosting Options for Private Packagespyproject.toml and Private PackagesClassifiers and the Special “Private :: Do Not Upload” ClassifierWhat Classifiers DoOptional Dependencies with “Extras”Defining Extras in pyproject.tomlInstalling Packages with ExtrasAutomating Package Deployments with CI/CDExample: GitHub Action Workflow for PyPI ReleaseVersion ManagementSemantic VersioningAlternative Versioning SchemesTools for Automated VersioningTesting PackagesTypes of Testing for Python PackagesEnvironment Testing with toxSecurity ConsiderationsAuditing Dependencies with SafetyBest Practices for Secure PackagingHandling Security VulnerabilitiesConclusion
21. Dependency Management
Virtual EnvironmentsManaging Python VersionsDeterministic BuildsDependency SpecifiersLocking DependenciesToolspippip-toolspipenvpdmuvDependency ConflictsSecurity in Dependency ManagementTyposquatting AttacksSupply Chain AttacksAuditing DependenciesPip-auditSafetyHash-Checking Mode to Secure DependenciesConclusion
Index

Content preview from Deep Dive Python: Techniques and Best Practices for Developers

A. DivakaranDeep Dive Pythonhttps://doi.org/10.1007/979-8-8688-1261-3_4

4. Logging

Adarsh Divakaran¹

(1)

Kannur, Kerala, India

In the last few chapters, we explored Python’s core data structures like lists, tuples, sets, and dictionaries—tools that help us organize and manipulate data. Now, we shift focus from data structures to understanding how we can monitor what our programs are doing in real time using logging.

Logging is essential in software development for monitoring, debugging, and ensuring the reliability of applications. Logging involves recording the events and data during a program execution, thus providing a history of application events. Python’s ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Effective Python: 90 Specific Ways to Write Better Python, 2nd Edition

Publisher Resources

ISBN: 9798868812613Purchase Link Publisher Website

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Deep Dive Python: Techniques and Best Practices for Developers

by Adarsh Divakaran

4. Logging

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.