book

Python Real-World Projects

Name: Python Real-World Projects
Author: Steven F. Lott
ISBN: 9781803246765

by Steven F. Lott

September 2023

Beginner to intermediate

478 pages

9h 31m

English

Packt Publishing

Read now

Unlock full access

Preface
Who this book is forWhat this book coversA note on skills requiredTo get the most out of this bookComplete the extrasDownload the example code filesConventions usedGet in touchShare your thoughtsDownload a free PDF copy of this book
Chapter 1: Project Zero: A Template for Other Projects
1.1 On quality1.1.1 More Reading on Quality1.2 Suggested project sprints1.2.1 Inception1.2.2 Elaboration, part 1: define done1.2.3 Elaboration, part 2: define components and tests1.2.4 Construction1.2.5 Transition1.3 List of deliverables1.4 Development tool installation1.5 Project 0 – Hello World with test cases1.5.1 Description1.5.2 Approach1.5.3 Deliverables1.5.4 Definition of done1.6 Summary1.7 Extras1.7.1 Static analysis - mypy, flake81.7.2 CLI features1.7.3 Logging1.7.4 Cookiecutter
Chapter 2: Overview of the Projects
2.1 General data acquisition2.2 Acquisition via Extract2.3 Inspection2.4 Clean, validate, standardize, and persist2.5 Summarize and analyze2.6 Statistical modeling2.7 Data contracts2.8 Summary
Chapter 3: Project 1.1: Data Acquisition Base Application
3.1 Description3.1.1 User experience3.1.2 About the source data3.1.3 About the output data3.2 Architectural approach3.2.1 Class design3.2.2 Design principles3.2.3 Functional design3.3 Deliverables3.3.1 Acceptance tests3.3.2 Additional acceptance scenarios3.3.3 Unit tests3.4 Summary3.5 Extras3.5.1 Logging enhancements3.5.2 Configuration extensions3.5.3 Data subsets3.5.4 Another example data source
Chapter 4: Data Acquisition Features: Web APIs and Scraping
4.1 Project 1.2: Acquire data from a web service4.1.1 Description4.1.2 Approach4.1.3 Deliverables4.2 Project 1.3: Scrape data from a web page4.2.1 Description4.2.2 About the source data4.2.3 Approach4.2.4 Deliverables4.3 Summary4.4 Extras4.4.1 Locate more JSON-format data4.4.2 Other data sets to extract4.4.3 Handling schema variations4.4.4 CLI enhancements4.4.5 Logging
Chapter 5: Data Acquisition Features: SQL Database
5.1 Project 1.4: A local SQL database5.1.1 Description5.1.2 Approach5.1.3 Deliverables5.2 Project 1.5: Acquire data from a SQL extract5.2.1 Description5.2.2 The Object-Relational Mapping (ORM) problem5.2.3 About the source data5.2.4 Approach5.2.5 Deliverables5.3 Summary5.4 Extras5.4.1 Consider using another database5.4.2 Consider using a NoSQL database5.4.3 Consider using SQLAlchemy to define an ORM layer
Chapter 6: Project 2.1: Data Inspection Notebook
6.1 Description6.1.1 About the source data6.2 Approach6.2.1 Notebook test cases for the functions6.2.2 Common code in a separate module6.3 Deliverables6.3.1 Notebook .ipynb file6.3.2 Executing a notebook’s test suite6.4 Summary6.5 Extras6.5.1 Use pandas to examine data
Chapter 7: Data Inspection Features
7.1 Project 2.2: Validating cardinal domains — measures, counts, and durations7.1.1 Description7.1.2 Approach7.1.3 Deliverables7.2 Project 2.3: Validating text and codes — nominal data and ordinal numbers7.2.1 Description7.2.2 Approach7.2.3 Deliverables7.3 Project 2.4: Finding reference domains7.3.1 Description7.3.2 Approach7.3.3 Deliverables7.4 Summary7.5 Extras7.5.1 Markdown cells with dates and data source information7.5.2 Presentation materials7.5.3 JupyterBook or Quarto for even more sophisticated output
Chapter 8: Project 2.5: Schema and Metadata
8.1 Description8.2 Approach8.2.1 Define Pydantic classes and emit the JSON Schema8.2.2 Define expected data domains in JSON Schema notation8.2.3 Use JSON Schema to validate intermediate files8.3 Deliverables8.3.1 Schema acceptance tests8.3.2 Extended acceptance testing8.4 Summary8.5 Extras8.5.1 Revise all previous chapter models to use Pydantic8.5.2 Use the ORM layer
Chapter 9: Project 3.1: Data Cleaning Base Application
9.1 Description9.1.1 User experience9.1.2 Source data9.1.3 Result data9.1.4 Conversions and processing9.1.5 Error reports9.2 Approach9.2.1 Model module refactoring9.2.2 Pydantic V2 validation9.2.3 Validation function design9.2.4 Incremental design9.2.5 CLI application9.3 Deliverables9.3.1 Acceptance tests9.3.2 Unit tests for the model features9.3.3 Application to clean data and create an NDJSON interim file9.4 Summary9.5 Extras9.5.1 Create an output file with rejected samples

Chapter 10: Data Cleaning Features
10.1 Project 3.2: Validate and convert source fields10.1.1 Description10.1.2 Approach10.1.3 Deliverables10.2 Project 3.3: Validate text fields (and numeric coded fields)10.2.1 Description10.2.2 Approach10.2.3 Deliverables10.3 Project 3.4: Validate references among separate data sources10.3.1 Description10.3.2 Approach10.3.3 Deliverables10.4 Project 3.5: Standardize data to common codes and ranges10.4.1 Description10.4.2 Approach10.4.3 Deliverables10.5 Project 3.6: Integration to create an acquisition pipeline10.5.1 Description10.5.2 Approach10.5.3 Deliverables10.6 Summary10.7 Extras10.7.1 Hypothesis testing10.7.2 Rejecting bad data via filtering (instead of logging)10.7.3 Disjoint subentities10.7.4 Create a fan-out cleaning pipeline
Chapter 11: Project 3.7: Interim Data Persistence
11.1 Description11.2 Overall approach11.2.1 Designing idempotent operations11.3 Deliverables11.3.1 Unit test11.3.2 Acceptance test11.3.3 Cleaned up re-runnable application design11.4 Summary11.5 Extras11.5.1 Using a SQL database11.5.2 Persistence with NoSQL databases
Chapter 12: Project 3.8: Integrated Data Acquisition Web Service
12.1 Description12.1.1 The data series resources12.1.2 Creating data for download12.2 Overall approach12.2.1 OpenAPI 3 specification12.2.2 RESTful API to be queried from a notebook12.2.3 A POST request starts processing12.2.4 The GET request for processing status12.2.5 The GET request for the results12.2.6 Security considerations12.3 Deliverables12.3.1 Acceptance test cases12.3.2 RESTful API app12.3.3 Unit test cases12.4 Summary12.5 Extras12.5.1 Add filtering criteria to the POST request12.5.3 Use Celery instead of concurrent.futures12.5.4 Call external processing directly instead of running a subprocess
Chapter 13: Project 4.1: Visual Analysis Techniques
13.1 Description13.2 Overall approach13.2.1 General notebook organization13.2.2 Python modules for summarizing13.2.3 PyPlot graphics13.2.4 Iteration and evolution13.3 Deliverables13.3.1 Unit test13.3.2 Acceptance test13.4 Summary13.5 Extras13.5.1 Use Seaborn for plotting13.5.2 Adjust color palettes to emphasize key points about the data
Chapter 14: Project 4.2: Creating Reports
14.1 Description14.1.1 Slide decks and presentations14.1.2 Reports14.2 Overall approach14.2.1 Preparing slides14.2.2 Preparing a report14.2.3 Creating technical diagrams14.3 Deliverables14.4 Summary14.5 Extras14.5.1 Written reports with UML diagrams
Chapter 15: Project 5.1: Modeling Base Application
15.1 Description15.2 Approach15.2.1 Designing a summary app15.2.2 Describing the distribution15.2.3 Use cleaned data model15.2.4 Rethink the data inspection functions15.2.5 Create new results model15.3 Deliverables15.3.1 Acceptance testing15.3.2 Unit testing15.3.3 Application secondary feature15.4 Summary15.5 Extras15.5.1 Measures of shape15.5.2 Creating PDF reports15.5.3 Serving the HTML report from the data API
Chapter 16: Project 5.2: Simple Multivariate Statistics
16.1 Description16.1.1 Correlation coefficient16.1.2 Linear regression16.1.3 Diagrams16.2 Approach16.2.1 Statistical computations16.2.2 Analysis diagrams16.2.3 Including diagrams in the final document16.3 Deliverables16.3.1 Acceptance tests16.3.2 Unit tests16.4 Summary16.5 Extras16.5.1 Use pandas to compute basic statistics16.5.2 Use the dask version of pandas16.5.3 Use numpy for statistics16.5.4 Use scikit-learn for modeling16.5.5 Compute the correlation and regression using functional programming
Chapter 17: Next Steps
17.1 Overall data wrangling17.2 The concept of “decision support”17.3 Concept of metadata and provenance17.4 Next steps toward machine learning
Why subscribe?
Other Books You Might EnjoyPackt is searching for authors like youShare your thoughtsDownload a free PDF copy of this book
Index

Content preview from Python Real-World Projects

Chapter 11 Project 3.7: Interim Data Persistence

Our goal is to create files of clean, converted data we can then use for further analysis. To an extent, the goal of creating a file of clean data has been a part of all of the previous chapters. We’ve avoided looking deeply at the interim results of acquisition and cleaning. This chapter formalizes some of the processing that was quietly assumed in those earlier chapters. In this chapter, we’ll look more closely at two topics:

File formats and data persistence
The architecture of applications

11.1 Description

In the previous chapters, particularly those starting with Chapter 9, Project 3.1: Data Cleaning Base Application, the question of ”persistence” was dealt with casually. The previous ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Python in Practice - 15 Projects to Master Python

Publisher Resources

ISBN: 9781803246765

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Python Real-World Projects

by Steven F. Lott

Chapter 11 Project 3.7: Interim Data Persistence

11.1 Description

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.