book

Test-Driven Development with Python, 3rd Edition

Name: Test-Driven Development with Python, 3rd Edition
Author: Harry Percival
ISBN: 9781098148713

by Harry Percival

October 2025

Intermediate to advanced

712 pages

14h 24m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Preface
Why I Wrote a Book About Test-Driven DevelopmentAims of This BookOutlineConventions Used in This BookSubmitting ErrataUsing Code ExamplesO’Reilly Online LearningHow to Contact UsCompanion VideoLicense for the Free Edition
Preface to the Third Edition: TDD in the Age of AI
AI Is Both Insanely Impressive and Incredibly UnreliableMitigations for AI’s Shortcomings Sure Look a Lot Like TDDLeaky Abstractions and the Importance of ExperienceMy Own Experiences with AIThe AI-Enabled Workflow of the Future
Prerequisites and Assumptions
Python 3 and ProgrammingHow HTML WorksDjangoJavaScriptRequired Software InstallationsInstalling FirefoxSetting Up Your VirtualenvActivating and Deactivating the VirtualenvInstalling Django and SeleniumSome Error Messages You’re Likely to See When You Inevitably Fail to Activate Your Virtualenv
Acknowledgments
Additional Thanks for the Second EditionAdditional Thanks for the Third Edition
I. The Basics of TDD and Django
1. Getting Django Set Up Using a Functional Test
Obey the Testing Goat! Do Nothing Until You Have a TestGetting Django Up and RunningStarting a Git Repository
2. Extending Our Functional Test Using the unittest Module
Using a Functional Test to Scope Out a Minimum Viable AppThe Python Standard Library’s unittest ModuleCommit
3. Testing a Simple Home Page with Unit Tests
Our First Django App and Our First Unit TestUnit Tests, and How They Differ from Functional TestsUnit Testing in DjangoDjango’s MVC, URLs, and View FunctionsUnit Testing a ViewAt Last! We Actually Write Some Application Code!The Unit-Test/Code CycleOur Functional Tests Tell Us We’re Not Quite Done YetReading Tracebacksurls.py
4. What Are We Doing with All These Tests? (And, Refactoring)
Programming Is Like Pulling a Bucket of Water Up from a WellUsing Selenium to Test User InteractionsThe “Don’t Test Constants” Rule, and Templates to the RescueRefactoring to Use a TemplateRevisiting Our Unit TestsTest Behaviour, Not ImplementationOn RefactoringA Little More of Our Front PageRecap: The TDD ProcessDouble-Loop TDD
5. Saving User Input: Testing the Database
Wiring Up Our Form to Send a POST RequestTesting the Contract Between Frontend and BackendDebugging Functional TestsDebugging with time.sleepProcessing a POST Request on the ServerPassing Python Variables to Be Rendered in the TemplateAn Unexpected FailureImproving Error Messages in TestsThree Strikes and RefactorThe Django ORM and Our First ModelOur First Database MigrationThe Test Gets Surprisingly FarA New Field Means a New MigrationSaving the POST to the DatabaseRedirect After a POSTBetter Unit Testing Practice: Each Test Should Test One ThingRendering Items in the TemplateCreating Our Production Database with migrateRecap

6. Improving Functional Tests: Ensuring Isolation and Removing Magic Sleeps
Ensuring Test Isolation in Functional TestsRunning Just the Unit TestsOn Implicit and Explicit Waits, and Magic time.sleeps
7. Working Incrementally
Small Design When NecessaryNot Big Design Up FrontYAGNI!REST-ishImplementing the New Design Incrementally Using TDDEnsuring We Have a Regression TestIterating Towards the New DesignTaking a First, Self-Contained Step: One New URLSeparating Out Our Home Page and List View FunctionalityThe FTs Detect a RegressionGetting Back to a Working State as Quickly as PossibleGreen? RefactorAnother Small Step: A Separate Template for Viewing ListsA Third Small Step: A New URL for Adding List ItemsA Test Class for New List CreationA URL and View for New List CreationRemoving Now-Redundant Code and TestsA Regression! Pointing Our Forms at the New URLDebugging in DevToolsBiting the Bullet: Adjusting Our ModelsA Foreign Key RelationshipAdjusting the Rest of the World to Our New ModelsEach List Should Have Its Own URLCapturing Parameters from URLsAdjusting new_list to the New WorldThe Functional Tests Detect Another RegressionOne More URL to Handle Adding Items to an Existing ListThe Last New urls.py EntryThe Last New ViewTesting Template Context DirectlyA Final Refactor Using URL includesCan You Believe It?
8. Prettification: Layout and Styling, and What to Test About It
Testing Layout and StylePrettification: Using a CSS FrameworkDjango Template InheritanceIntegrating BootstrapRows and ColumnsStatic Files in DjangoSwitching to StaticLiveServerTestCaseUsing Bootstrap Components to Improve the Look of the SiteJumbotron!Large InputsTable StylingOptional: Dark ModeA Semi-Decent PageParsing HTML for Less Brittle Tests of Key HTML ContentWhat We Glossed Over: collectstatic and Other Static DirectoriesA Few Things That Didn’t Make It
II. Going to Production
9. Containerization aka Docker
Docker, Containers, and VirtualizationWhy Not Just Use a Virtualenv?Docker and Your CVAs Always, Start with a TestMaking a src FolderInstalling DockerBuilding a Docker Image and Running a Docker ContainerA First Cut of a DockerfileDocker BuildDocker RunInstalling Django in a Virtualenv in Our Container ImageSuccessful RunUsing the FT to Check That Our Container WorksDebugging Container Networking ProblemsDebugging Web Server Connectivity with curlRunning Code “Inside” the Container with docker execDocker Port MappingEssential Googling the Error MessageDatabase MigrationsShould We Run migrate Inside the Dockerfile? No.Mounting Files Inside the Container
10. Making Our App Production-Ready
What We Need to DoSwitching to GunicornThe FTs Catch a Problem with Static FilesServing Static Files with WhiteNoiseUsing requirements.txtUsing Environment Variables to Adjust Settings for ProductionSetting DEBUG=True and SECRET_KEYSetting Environment Variables Inside the DockerfileSetting Environment Variables at the Docker Command LineALLOWED_HOSTS Is Required When Debug Mode Is Turned OffCollectstatic Is Required when Debug Is Turned OffSwitching to a Nonroot UserMaking the Database Filepath ConfigurableUsing UIDs to Set Permissions Across Host/Container MountsConfiguring LoggingProvoking a Deliberate ErrorExercise for the Reader: Using the Django check CommandWrap-Up
11. Getting a Server Ready for Deployment
Manually Provisioning a Server to Host Our SiteChoosing Where to Host Our SiteSpinning Up Our Own ServerGetting a Domain NameConfiguring DNS for Staging and Live DomainsAnsibleAnsible Versus SSH: How We’ll Talk to Our ServerStart by Making Sure We Can SSH InDebugging Issues with SSHInstalling AnsibleChecking Ansible Can Talk to Our Server
12. Infrastructure as Code: Automated Deployments with Ansible
A First Cut of an Ansible Playbook for DeploymentSSHing Into the Server and Viewing Container LogsAllowing Rootless Docker AccessGetting Our Image Onto the ServerTaking a Look Around ManuallySetting Environment Variables and SecretsManually Checking Environment Variables for Running ContainersRunning FTs to Check on Our DeployManual Debugging with curl Against the Staging ServerMounting the Database on the Server and Running MigrationsIt WorkssssDeploying to Prodgit tag the ReleaseTell Everyone!Further Reading
III. Forms and Validation
13. Splitting Our Tests into Multiple Files, and a Generic Wait Helper
Start on a Validation FT: Preventing Blank ItemsSkipping a TestSplitting Functional Tests Out into Many FilesRunning a Single Test FileA New FT Tool: A Generic Explicit Wait HelperFinishing Off the FTRefactoring Unit Tests into Several Files
14. Validation at the Database Layer
Model-Layer ValidationThe self.assertRaises Context ManagerDjango Model Constraints and Their Interaction with DatabasesInspecting Our Constraints at the Database LevelTesting Django Model ValidationA Django Quirk: Model Save Doesn’t Run ValidationSurfacing Model Validation Errors in the ViewChecking That Invalid Input Isn’t Saved to the DatabaseAdding an Early Return to Our FT to Let Us Refactor Against GreenDjango Pattern: Processing POST Requests in the Same View That Renders the FormRefactor: Transferring the new_item Functionality into view_listEnforcing Model Validation in view_listRefactor: Removing Hardcoded URLsThe {% url %} Template TagUsing get_absolute_url for Redirects
15. A Simple Form
Moving Validation Logic Into a FormExploring the Forms API with a Unit TestSwitching to a Django ModelFormTesting and Customising Form ValidationAttempting to Use the Form in Our ViewsUsing the Form in a View with a GET RequestThe Trade-offs of Django ModelForms: The Frontend Is Coupled to the DatabaseA Big Find-and-ReplaceBacking Out Our Changes and Getting to a Working StateRenaming the name AttributeRenaming the id AttributeA Second Attempt at Using the Form in Our ViewsUsing the Form in a View That Takes POST RequestsUsing the Form to Display Errors in the TemplateGet Back to a Working StateA Helper Method for Several Short TestsUsing the Form in the Existing Lists ViewUsing the Form to Pass Errors to the TemplateRefactoring the View to Use the Form FullyAn Unexpected Benefit: Free Client-Side Validation from HTML5A Pat on the BackBut Have We Wasted a Lot of Time?Using the ModelForm’s Own Save Method
16. More Advanced Forms
Another FT for Duplicate ItemsPreventing Duplicates at the Model LayerRewriting the Old Model TestIntegrity Errors That Show Up on SaveExperimenting with Duplicate Item Validation at the Views LayerA More Complex Form to Handle Uniqueness ValidationUsing the Existing List Item Form in the List ViewCustomising the Save Method on Our New FormThe FTs Pick Up an Issue with Bootstrap ClassesConditionally Customising CSS Classes for Invalid FormsA Little Digression on Queryset Ordering and String RepresentationsOn the Trade-offs of Django ModelForms, and Frameworks in GeneralMoving Presentation Logic Back into the TemplateTidying Up the FormsSwitching Back to Simple FormsWrapping Up: What We’ve Learned About Testing Django
IV. More Advanced Topics in Testing
17. A Gentle Excursion into JavaScript
Starting with an FTA Quick SpikeA Simple Inline ScriptUsing the Browser DevToolsChoosing a Basic JavaScript Test RunnerAn Overview of JasmineSetting Up Our JavaScript Test EnvironmentOur First Smoke Test: Describe, It, ExpectRunning the Tests via the BrowserTesting with Some DOM ContentBuilding a JavaScript Unit Test for Our Desired FunctionalityFixtures, Execution Order, and Global State: Key Challenges of JavaScript Testingconsole.log for Debug PrintingUsing an Initialize Function for More Control Over Execution TimeDeliberately Breaking Our Code to Force Ourselves to Write More TestsRed/Green/Refactor: Removing Hardcoded SelectorsDoes it Work?Testing Integration with CSS and BootstrapColumbo Says: Wait for OnloadJavaScript Testing in the TDD Cycle
18. Deploying Our New Code
The Deployment ChecklistA Full Test Run LocallyQuick Test Run Against DockerStaging Deploy and Test RunProduction DeployWhat to Do If You See a Database ErrorHow to Delete the Database on the Staging ServerWrap-Up: git tag the New Release
19. User Authentication, Spiking, and De-Spiking
Passwordless Auth with “Magic Links”A Somewhat Larger SpikeStarting a Branch for the SpikeFrontend Login UISending Emails from DjangoEmail Server Config for DjangoAnother Secret, Another Environment VariableStoring Tokens in the DatabaseCustom Authentication ModelsFinishing the Custom Django AuthDe-SpikingMaking a PlanWring an FT Against the Spiked CodeReverting Our Spiked CodeA Minimal Custom User ModelTests as DocumentationA Token Model to Link Emails with a Unique ID
20. Using Mocks to Test External Dependencies
Before We Start: Getting the Basic Plumbing InMocking Manually—aka MonkeypatchingThe Python Mock LibraryUsing unittest.patchGetting the FT a Little Further AlongTesting the Django Messages FrameworkAdding Messages to Our HTMLStarting on the Login URLChecking That We Send the User a Link with a TokenDe-Spiking Our Custom Authentication BackendOne if = One More TestThe get_user Method
21. Using Mocks for Test Isolation
Using Our Auth Backend in the Login ViewStraightforward Non-Mocky Test for Our ViewCombinatorial ExplosionThe Car Factory ExampleUsing Mocks to Test Parts of Our System in IsolationMocks Can Also Let You Test the Implementation, When It MattersStarting Again: Test-Driving Our Implementation with MocksUsing mock.return_valueUsing .return_value During Test SetupUnDONTifyingDeciding Which Tests to KeepThe Moment of Truth: Will the FT Pass?It Works in Theory! Does It Work in Practice?Using Our New Environment Variable, and Saving It to .envFinishing Off Our FT: Testing Logout
22. Test Fixtures and a Decorator for Explicit Waits
Skipping the Login Process by Pre-creating a SessionChecking That It WorksOur Final Explicit Wait Helper: A Wait Decorator
23. Debugging and Testing Server Issues
The Proof Is in the Pudding: Using Docker to Catch Final BugsInspecting the Docker Container LogsAnother Environment Variable in Dockermail.outbox Won’t Work Outside Django’s Test EnvironmentDeciding How to Test “Real” Email SendingAn Alternative Method for Setting Secret Environment Variables on the ServerDebugging with SQLManaging Fixtures in Real DatabasesA Django Management Command to Create SessionsGetting the FT to Run the Management Command on the ServerRunning Commands Using Docker Exec and (Optionally) SSHRecap: Creating Sessions Locally Versus StagingTesting the Management CommandTest Database CleanupWrap-Up
24. Finishing “My Lists”: Outside-In TDD
The Alternative: Inside-OutWhy Prefer “Outside-In”?The FT for “My Lists”The Outside Layer: Presentation and TemplatesMoving Down One Layer to View Functions (the Controller)Another Pass, Outside-InA Quick Restructure of Our Template CompositionAn Early Return So We’re Refactoring Against GreenFactoring Out Two Template includesDesigning Our API Using the TemplateMoving Down to the Next Layer: What the View Passes to the TemplateThe Next “Requirement” from the Views Layer: New Lists Should Record OwnerA Decision Point: Whether to Proceed to the Next Layer with a Failing TestMoving Down to the Model LayerFinal Step: Feeding Through the .name API from the Template
25. CI: Continuous Integration
CI in Modern Development WorkflowsChoosing a CI ServiceGetting Our Code into GitLabSigning UpStarting a ProjectPushing Our Code Up Using Git PushSetting Up a First Cut of a CI PipelineFirst Build! (and First Failure)Trying to Reproduce a CI Error LocallyEnabling Debug Logs for Selenium/Firefox/WebdriverEnabling Headless Mode for FirefoxA Common Bugbear: Flaky TestsTaking ScreenshotsSaving Build Outputs (or Debug Files) as ArtifactsIf in Doubt, Try Bumping the Timeout!A Successful Python Test RunRunning Our JavaScript Tests in CIInstalling Node.jsInstalling and Configuring the Jasmine Browser RunnerAdding a Build Step for JavaScriptTests Now PassSome Things We Didn’t CoverDefining a Docker Image for CICachingAutomated Deployment, aka Continuous Delivery (CD)
26. The Token Social Bit, the Page Pattern, and an Exercise for the Reader
An FT with Multiple Users, and addCleanupThe Page PatternExtend the FT to a Second User, and the “My Lists” PageAn Exercise for the ReaderStep-by-Step Guide
27. Fast Tests, Slow Tests, and Hot Lava
Why Do We Test? Our Desiderata for Effective TestsConfidence and Correctness (Preventing Regression)A Productive WorkflowDriving Better DesignWere Our Unit Tests Integration Tests All Along? What Is That Warm Glow Coming from the Database?We’ve Been in the “Sweet Spot”What Is a “True” Unit Test? Does it Matter?Integration and Functional Tests Get Slower Over TimeWe’re Not Getting the Full Potential Benefits of TestingThe Ideal of the Test PyramidAvoiding Mock HellThe Actual Solutions Are ArchitecturalPorts and Adapters/Hexagonal/Onion/Clean ArchitectureFunctional Core, Imperative ShellThe Central Conceit: These Architectures Are “Better”The Hardest Part: Knowing When to Make the SwitchWrap-UpFurther Reading
Obey the Testing Goat!
Testing Is HardKeep Your CI Builds GreenTake Pride in Your Tests, as You Do in Your CodeRemember to Tip the Bar StaffDon’t Be a Stranger!
Bibliography
A. Cheat Sheet
Initial Project SetupThe Basic TDD Workflow: Red/Green/RefactorMoving Beyond Dev-Only TestingGeneral Testing Best PracticesSelenium/Functional Testing Best PracticesOutside-InThe Test Pyramid
B. What to Do Next
Switch to PostgresRun Your Tests Against Different BrowsersThe Django Admin SiteWrite Some Security TestsTest for Graceful DegradationCaching and Performance TestingJavaScript FrameworksAsync and WebsocketsSwitch to Using pytestCheck Out coverage.pyClient-Side EncryptionYour Suggestion Here
C. Source Code Examples
Full List of Links for Each ChapterUsing Git to Check Your ProgressDownloading a ZIP File for a ChapterDon’t Let it Become a Crutch!
Index
About the Author

Content preview from Test-Driven Development with Python, 3rd Edition

Chapter 12. Infrastructure as Code: Automated Deployments with Ansible

Automate, automate, automate.

Cay S. Horstmann

Now that our server is up and running, we want to install our app on it, using our Docker image and container.

We could do this manually, but a key insight of modern software engineering is that small, frequent deployments are a must.

Note

This insight about the importance of frequent deployments we owe to Nicole Forsgren and the State of DevOps reports. They are some of the only really firm science we have in the field of software engineering.

Frequent deployments rely on automation,¹ so we’ll use Ansible.

Automation is also key to making sure our tests give us true confidence over our deployments. If we go to the trouble of building a staging server,² we want to make sure that it’s as similar as possible to the production environment. By automating the way we deploy, and using the same automation for staging and prod, we give ourselves much more confidence.

The buzzword for automating your deployments these days is “infrastructure as code” (IaC).

Note

Why not ping me a note once your site is live on the web, and send me the URL? It always gives me a warm and fuzzy feeling…Email me at obeythetestinggoat@gmail.com.

A First Cut of an Ansible Playbook for Deployment

Let’s start using Ansible a little more seriously. We’re not going to jump all the way to the end though! Baby steps, as always. Let’s see if we can get it to run a simple “hello world” Docker ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Effective Python: 125 Specific Ways to Write Better Python, 3rd Edition

Publisher Resources

ISBN: 9781098148706Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Test-Driven Development with Python, 3rd Edition

by Harry Percival

Chapter 12. Infrastructure as Code: Automated Deployments with Ansible

Note

Note

A First Cut of an Ansible Playbook for Deployment

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.