book

Fluent Python, 2nd Edition

by Luciano Ramalho

April 2022

Intermediate to advanced

1014 pages

23h 59m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Includes

Has Sandbox

Who This Book Is ForWho This Book Is Not ForFive Books in OneHow the Book Is OrganizedHands-On ApproachSoapbox: My Personal PerspectiveCompanion Website: fluentpython.comConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgmentsAcknowledgments for the First Edition
What’s New in This ChapterA Pythonic Card DeckHow Special Methods Are UsedEmulating Numeric TypesString RepresentationBoolean Value of a Custom TypeCollection APIOverview of Special MethodsWhy len Is Not a MethodChapter SummaryFurther Reading
What’s New in This ChapterOverview of Built-In SequencesList Comprehensions and Generator ExpressionsList Comprehensions and ReadabilityListcomps Versus map and filterCartesian ProductsGenerator ExpressionsTuples Are Not Just Immutable ListsTuples as RecordsTuples as Immutable ListsComparing Tuple and List MethodsUnpacking Sequences and IterablesUsing * to Grab Excess ItemsUnpacking with * in Function Calls and Sequence LiteralsNested UnpackingPattern Matching with SequencesPattern Matching Sequences in an InterpreterSlicingWhy Slices and Ranges Exclude the Last ItemSlice ObjectsMultidimensional Slicing and EllipsisAssigning to SlicesUsing + and * with SequencesBuilding Lists of ListsAugmented Assignment with SequencesA += Assignment Puzzlerlist.sort Versus the sorted Built-InWhen a List Is Not the AnswerArraysMemory ViewsNumPyDeques and Other QueuesChapter SummaryFurther Reading
What’s New in This ChapterModern dict Syntaxdict ComprehensionsUnpacking MappingsMerging Mappings with |Pattern Matching with MappingsStandard API of Mapping TypesWhat Is HashableOverview of Common Mapping MethodsInserting or Updating Mutable ValuesAutomatic Handling of Missing Keysdefaultdict: Another Take on Missing KeysThe __missing__ MethodInconsistent Usage of __missing__ in the Standard LibraryVariations of dictcollections.OrderedDictcollections.ChainMapcollections.Countershelve.ShelfSubclassing UserDict Instead of dictImmutable MappingsDictionary ViewsPractical Consequences of How dict WorksSet TheorySet LiteralsSet ComprehensionsPractical Consequences of How Sets WorkSet OperationsSet Operations on dict ViewsChapter SummaryFurther Reading
What’s New in This ChapterCharacter IssuesByte EssentialsBasic Encoders/DecodersUnderstanding Encode/Decode ProblemsCoping with UnicodeEncodeErrorCoping with UnicodeDecodeErrorSyntaxError When Loading Modules with Unexpected EncodingHow to Discover the Encoding of a Byte SequenceBOM: A Useful GremlinHandling Text FilesBeware of Encoding DefaultsNormalizing Unicode for Reliable ComparisonsCase FoldingUtility Functions for Normalized Text MatchingExtreme “Normalization”: Taking Out DiacriticsSorting Unicode TextSorting with the Unicode Collation AlgorithmThe Unicode DatabaseFinding Characters by NameNumeric Meaning of CharactersDual-Mode str and bytes APIsstr Versus bytes in Regular Expressionsstr Versus bytes in os FunctionsChapter SummaryFurther Reading
What’s New in This ChapterOverview of Data Class BuildersMain FeaturesClassic Named TuplesTyped Named TuplesType Hints 101No Runtime EffectVariable Annotation SyntaxThe Meaning of Variable AnnotationsMore About @dataclassField OptionsPost-init ProcessingTyped Class AttributesInitialization Variables That Are Not Fields@dataclass Example: Dublin Core Resource RecordData Class as a Code SmellData Class as ScaffoldingData Class as Intermediate RepresentationPattern Matching Class InstancesSimple Class PatternsKeyword Class PatternsPositional Class PatternsChapter SummaryFurther Reading
What’s New in This ChapterVariables Are Not BoxesIdentity, Equality, and AliasesChoosing Between == and isThe Relative Immutability of TuplesCopies Are Shallow by DefaultDeep and Shallow Copies of Arbitrary ObjectsFunction Parameters as ReferencesMutable Types as Parameter Defaults: Bad IdeaDefensive Programming with Mutable Parametersdel and Garbage CollectionTricks Python Plays with ImmutablesChapter SummaryFurther Reading
What’s New in This ChapterTreating a Function Like an ObjectHigher-Order FunctionsModern Replacements for map, filter, and reduceAnonymous FunctionsThe Nine Flavors of Callable ObjectsUser-Defined Callable TypesFrom Positional to Keyword-Only ParametersPositional-Only ParametersPackages for Functional ProgrammingThe operator ModuleFreezing Arguments with functools.partialChapter SummaryFurther Reading

What’s New in This ChapterAbout Gradual TypingGradual Typing in PracticeStarting with MypyMaking Mypy More StrictA Default Parameter ValueUsing None as a DefaultTypes Are Defined by Supported OperationsTypes Usable in AnnotationsThe Any TypeSimple Types and ClassesOptional and Union TypesGeneric CollectionsTuple TypesGeneric MappingsAbstract Base ClassesIterableParameterized Generics and TypeVarStatic ProtocolsCallableNoReturnAnnotating Positional Only and Variadic ParametersImperfect Typing and Strong TestingChapter SummaryFurther Reading
What’s New in This ChapterDecorators 101When Python Executes DecoratorsRegistration DecoratorsVariable Scope RulesClosuresThe nonlocal DeclarationVariable Lookup LogicImplementing a Simple DecoratorHow It WorksDecorators in the Standard LibraryMemoization with functools.cacheUsing lru_cacheSingle Dispatch Generic FunctionsParameterized DecoratorsA Parameterized Registration DecoratorThe Parameterized Clock DecoratorA Class-Based Clock DecoratorChapter SummaryFurther Reading
What’s New in This ChapterCase Study: Refactoring StrategyClassic StrategyFunction-Oriented StrategyChoosing the Best Strategy: Simple ApproachFinding Strategies in a ModuleDecorator-Enhanced Strategy PatternThe Command PatternChapter SummaryFurther Reading
What’s New in This ChapterObject RepresentationsVector Class ReduxAn Alternative Constructorclassmethod Versus staticmethodFormatted DisplaysA Hashable Vector2dSupporting Positional Pattern MatchingComplete Listing of Vector2d, Version 3Private and “Protected” Attributes in PythonSaving Memory with __slots__Simple Measure of __slot__ SavingsSummarizing the Issues with __slots__Overriding Class AttributesChapter SummaryFurther Reading
What’s New in This ChapterVector: A User-Defined Sequence TypeVector Take #1: Vector2d CompatibleProtocols and Duck TypingVector Take #2: A Sliceable SequenceHow Slicing WorksA Slice-Aware __getitem__Vector Take #3: Dynamic Attribute AccessVector Take #4: Hashing and a Faster ==Vector Take #5: FormattingChapter SummaryFurther Reading
The Typing MapWhat’s New in This ChapterTwo Kinds of ProtocolsProgramming DucksPython Digs SequencesMonkey Patching: Implementing a Protocol at RuntimeDefensive Programming and “Fail Fast”Goose TypingSubclassing an ABCABCs in the Standard LibraryDefining and Using an ABCABC Syntax DetailsSubclassing an ABCA Virtual Subclass of an ABCUsage of register in PracticeStructural Typing with ABCsStatic ProtocolsThe Typed double FunctionRuntime Checkable Static ProtocolsLimitations of Runtime Protocol ChecksSupporting a Static ProtocolDesigning a Static ProtocolBest Practices for Protocol DesignExtending a ProtocolThe numbers ABCs and Numeric ProtocolsChapter SummaryFurther Reading
What’s New in This ChapterThe super() FunctionSubclassing Built-In Types Is TrickyMultiple Inheritance and Method Resolution OrderMixin ClassesCase-Insensitive MappingsMultiple Inheritance in the Real WorldABCs Are Mixins TooThreadingMixIn and ForkingMixInDjango Generic Views MixinsMultiple Inheritance in TkinterCoping with InheritanceFavor Object Composition over Class InheritanceUnderstand Why Inheritance Is Used in Each CaseMake Interfaces Explicit with ABCsUse Explicit Mixins for Code ReuseProvide Aggregate Classes to UsersSubclass Only Classes Designed for SubclassingAvoid Subclassing from Concrete ClassesTkinter: The Good, the Bad, and the UglyChapter SummaryFurther Reading
What’s New in This ChapterOverloaded SignaturesMax OverloadTakeaways from Overloading maxTypedDictType CastingReading Type Hints at RuntimeProblems with Annotations at RuntimeDealing with the ProblemImplementing a Generic ClassBasic Jargon for Generic TypesVarianceAn Invariant DispenserA Covariant DispenserA Contravariant Trash CanVariance ReviewImplementing a Generic Static ProtocolChapter SummaryFurther Reading
What’s New in This ChapterOperator Overloading 101Unary OperatorsOverloading + for Vector AdditionOverloading * for Scalar MultiplicationUsing @ as an Infix OperatorWrapping-Up Arithmetic OperatorsRich Comparison OperatorsAugmented Assignment OperatorsChapter SummaryFurther Reading
What’s New in This ChapterA Sequence of WordsWhy Sequences Are Iterable: The iter FunctionUsing iter with a CallableIterables Versus IteratorsSentence Classes with __iter__Sentence Take #2: A Classic IteratorDon’t Make the Iterable an Iterator for ItselfSentence Take #3: A Generator FunctionHow a Generator WorksLazy SentencesSentence Take #4: Lazy GeneratorSentence Take #5: Lazy Generator ExpressionWhen to Use Generator ExpressionsAn Arithmetic Progression GeneratorArithmetic Progression with itertoolsGenerator Functions in the Standard LibraryIterable Reducing FunctionsSubgenerators with yield fromReinventing chainTraversing a TreeGeneric Iterable TypesClassic CoroutinesExample: Coroutine to Compute a Running AverageReturning a Value from a CoroutineGeneric Type Hints for Classic CoroutinesChapter SummaryFurther Reading
What’s New in This ChapterContext Managers and with BlocksThe contextlib UtilitiesUsing @contextmanagerPattern Matching in lis.py: A Case StudyScheme SyntaxImports and TypesThe ParserThe EnvironmentThe REPLThe EvaluatorProcedure: A Class Implementing a ClosureUsing OR-patternsDo This, Then That: else Blocks Beyond ifChapter SummaryFurther Reading
What’s New in This ChapterThe Big PictureA Bit of JargonProcesses, Threads, and Python’s Infamous GILA Concurrent Hello WorldSpinner with ThreadsSpinner with ProcessesSpinner with CoroutinesSupervisors Side-by-SideThe Real Impact of the GILQuick QuizA Homegrown Process PoolProcess-Based SolutionUnderstanding the Elapsed TimesCode for the Multicore Prime CheckerExperimenting with More or Fewer ProcessesThread-Based NonsolutionPython in the Multicore WorldSystem AdministrationData ScienceServer-Side Web/Mobile DevelopmentWSGI Application ServersDistributed Task QueuesChapter SummaryFurther ReadingConcurrency with Threads and ProcessesThe GILConcurrency Beyond the Standard LibraryConcurrency and Scalability Beyond Python
What’s New in This ChapterConcurrent Web DownloadsA Sequential Download ScriptDownloading with concurrent.futuresWhere Are the Futures?Launching Processes with concurrent.futuresMulticore Prime Checker ReduxExperimenting with Executor.mapDownloads with Progress Display and Error HandlingError Handling in the flags2 ExamplesUsing futures.as_completedChapter SummaryFurther Reading
What’s New in This ChapterA Few DefinitionsAn asyncio Example: Probing DomainsGuido’s Trick to Read Asynchronous CodeNew Concept: AwaitableDownloading with asyncio and HTTPXThe Secret of Native Coroutines: Humble GeneratorsThe All-or-Nothing ProblemAsynchronous Context ManagersEnhancing the asyncio DownloaderUsing asyncio.as_completed and a ThreadThrottling Requests with a SemaphoreMaking Multiple Requests for Each DownloadDelegating Tasks to ExecutorsWriting asyncio ServersA FastAPI Web ServiceAn asyncio TCP ServerAsynchronous Iteration and Asynchronous IterablesAsynchronous Generator FunctionsAsync Comprehensions and Async Generator Expressionsasync Beyond asyncio: CurioType Hinting Asynchronous ObjectsHow Async Works and How It Doesn’tRunning Circles Around Blocking CallsThe Myth of I/O-Bound SystemsAvoiding CPU-Bound TrapsChapter SummaryFurther Reading
What’s New in This ChapterData Wrangling with Dynamic AttributesExploring JSON-Like Data with Dynamic AttributesThe Invalid Attribute Name ProblemFlexible Object Creation with __new__Computed PropertiesStep 1: Data-Driven Attribute CreationStep 2: Property to Retrieve a Linked RecordStep 3: Property Overriding an Existing AttributeStep 4: Bespoke Property CacheStep 5: Caching Properties with functoolsUsing a Property for Attribute ValidationLineItem Take #1: Class for an Item in an OrderLineItem Take #2: A Validating PropertyA Proper Look at PropertiesProperties Override Instance AttributesProperty DocumentationCoding a Property FactoryHandling Attribute DeletionEssential Attributes and Functions for Attribute HandlingSpecial Attributes that Affect Attribute HandlingBuilt-In Functions for Attribute HandlingSpecial Methods for Attribute HandlingChapter SummaryFurther Reading
What’s New in This ChapterDescriptor Example: Attribute ValidationLineItem Take #3: A Simple DescriptorLineItem Take #4: Automatic Naming of Storage AttributesLineItem Take #5: A New Descriptor TypeOverriding Versus Nonoverriding DescriptorsOverriding DescriptorsOverriding Descriptor Without __get__Nonoverriding DescriptorOverwriting a Descriptor in the ClassMethods Are DescriptorsDescriptor Usage TipsDescriptor Docstring and Overriding DeletionChapter SummaryFurther Reading
What’s New in This ChapterClasses as Objectstype: The Built-In Class FactoryA Class Factory FunctionIntroducing __init_subclass__Why __init_subclass__ Cannot Configure __slots__Enhancing Classes with a Class DecoratorWhat Happens When: Import Time Versus RuntimeEvaluation Time ExperimentsMetaclasses 101How a Metaclass Customizes a ClassA Nice Metaclass ExampleMetaclass Evaluation Time ExperimentA Metaclass Solution for CheckedMetaclasses in the Real WorldModern Features Simplify or Replace MetaclassesMetaclasses Are Stable Language FeaturesA Class Can Only Have One MetaclassMetaclasses Should Be Implementation DetailsA Metaclass Hack with __prepare__Wrapping UpChapter SummaryFurther Reading
Further Reading

Content preview from Fluent Python, 2nd Edition

Chapter 7. Functions as First-Class Objects

I have never considered Python to be heavily influenced by functional languages, no matter what people say or think. I was much more familiar with imperative languages such as C and Algol 68 and although I had made functions first-class objects, I didn’t view Python as a functional programming language.

Guido van Rossum, Python BDFL¹

Functions in Python are first-class objects. Programming language researchers define a “first-class object” as a program entity that can be:

Created at runtime
Assigned to a variable or element in a data structure
Passed as an argument to a function
Returned as the result of a function

Integers, strings, and dictionaries are other examples of first-class objects in Python—nothing fancy here. Having functions as first-class objects is an essential feature of functional languages, such as Clojure, Elixir, and Haskell. However, first-class functions are so useful that they’ve been adopted by popular languages like JavaScript, Go, and Java (since JDK 8), none of which claim to be “functional languages.”

This chapter and most of Part III explore the practical applications of treating functions as objects.

Tip

The term “first-class functions” is widely used as shorthand for “functions as first-class objects.” It’s not ideal because it implies an “elite” among functions. In Python, all functions are first-class.