Automate the Boring Stuff with Python

Table of contents

1. Automate the Boring Stuff with Python: Practical Programming for Total Beginners
2. Dedication
3. About the Author
4. About the Tech Reviewer
5. Acknowledgments
6. Introduction
   1. Whom Is This Book For?
   2. Conventions
   3. What Is Programming?
      1. What Is Python?
      2. Programmers Don’t Need to Know Much Math
      3. Programming Is a Creative Activity
   4. About This Book
   5. Downloading and Installing Python
   6. Starting IDLE
      1. The Interactive Shell
   7. How to Find Help
   8. Asking Smart Programming Questions
   9. Summary
7. I. Python Programming Basics
   1. 1. Python Basics
      1. Entering Expressions into the Interactive Shell
      2. The Integer, Floating-Point, and String Data Types
      3. String Concatenation and Replication
      4. Storing Values in Variables
         1. Assignment Statements
         2. Variable Names
      5. Your First Program
      6. Dissecting Your Program
         1. Comments
         2. The print() Function
         3. The input() Function
         4. Printing the User’s Name
         5. The len() Function
         6. The str(), int(), and float() Functions
      7. Summary
      8. Practice Questions
   2. 2. Flow Control
      1. Boolean Values
      2. Comparison Operators
      3. Boolean Operators
         1. Binary Boolean Operators
         2. The not Operator
      4. Mixing Boolean and Comparison Operators
      5. Elements of Flow Control
         1. Conditions
         2. Blocks of Code
      6. Program Execution
      7. Flow Control Statements
         1. if Statements
         2. else Statements
         3. elif Statements
         4. while Loop Statements
            1. An Annoying while Loop
         5. break Statements
         6. continue Statements
         7. for Loops and the range() Function
            1. An Equivalent while Loop
            2. The Starting, Stopping, and Stepping Arguments to range()
      8. Importing Modules
         1. from import Statements
      9. Ending a Program Early with sys.exit()
      10. Summary
      11. Practice Questions
   3. 3. Functions
      1. def Statements with Parameters
      2. Return Values and return Statements
      3. The None Value
      4. Keyword Arguments and print()
      5. Local and Global Scope
         1. Local Variables Cannot Be Used in the Global Scope
         2. Local Scopes Cannot Use Variables in Other Local Scopes
         3. Global Variables Can Be Read from a Local Scope
         4. Local and Global Variables with the Same Name
      6. The global Statement
      7. Exception Handling
      8. A Short Program: Guess the Number
      9. Summary
      10. Practice Questions
      11. Practice Projects
          1. The Collatz Sequence
          2. Input Validation
   4. 4. Lists
      1. The List Data Type
         1. Getting Individual Values in a List with Indexes
         2. Negative Indexes
         3. Getting Sublists with Slices
         4. Getting a List’s Length with len()
         5. Changing Values in a List with Indexes
         6. List Concatenation and List Replication
         7. Removing Values from Lists with del Statements
      2. Working with Lists
         1. Using for Loops with Lists
         2. The in and not in Operators
         3. The Multiple Assignment Trick
      3. Augmented Assignment Operators
      4. Methods
         1. Finding a Value in a List with the index() Method
         2. Adding Values to Lists with the append() and insert() Methods
         3. Removing Values from Lists with remove()
         4. Sorting the Values in a List with the sort() Method
      5. Example Program: Magic 8 Ball with a List
      6. List-like Types: Strings and Tuples
         1. Mutable and Immutable Data Types
         2. The Tuple Data Type
         3. Converting Types with the list() and tuple() Functions
      7. References
         1. Passing References
         2. The copy Module’s copy() and deepcopy() Functions
      8. Summary
      9. Practice Questions
      10. Practice Projects
          1. Comma Code
          2. Character Picture Grid
   5. 5. Dictionaries and Structuring Data
      1. The Dictionary Data Type
         1. Dictionaries vs. Lists
         2. The keys(), values(), and items() Methods
         3. Checking Whether a Key or Value Exists in a Dictionary
         4. The get() Method
         5. The setdefault() Method
      2. Pretty Printing
      3. Using Data Structures to Model Real-World Things
         1. A Tic-Tac-Toe Board
         2. Nested Dictionaries and Lists
      4. Summary
      5. Practice Questions
      6. Practice Projects
         1. Fantasy Game Inventory
         2. List to Dictionary Function for Fantasy Game Inventory
   6. 6. Manipulating Strings
      1. Working with Strings
         1. String Literals
            1. Double Quotes
            2. Escape Characters
            3. Raw Strings
            4. Multiline Strings with Triple Quotes
            5. Multiline Comments
         2. Indexing and Slicing Strings
         3. The in and not in Operators with Strings
      2. Useful String Methods
         1. The upper(), lower(), isupper(), and islower() String Methods
         2. The isX String Methods
         3. The startswith() and endswith() String Methods
         4. The join() and split() String Methods
         5. Justifying Text with rjust(), ljust(), and center()
         6. Removing Whitespace with strip(), rstrip(), and lstrip()
         7. Copying and Pasting Strings with the pyperclip Module
      3. Project: Password Locker
         1. Step 1: Program Design and Data Structures
         2. Step 2: Handle Command Line Arguments
         3. Step 3: Copy the Right Password
      4. Project: Adding Bullets to Wiki Markup
         1. Step 1: Copy and Paste from the Clipboard
         2. Step 2: Separate the Lines of Text and Add the Star
         3. Step 3: Join the Modified Lines
      5. Summary
      6. Practice Questions
      7. Practice Project
         1. Table Printer
8. II. Automating Tasks
   1. 7. Pattern Matching with Regular Expressions
      1. Finding Patterns of Text Without Regular Expressions
      2. Finding Patterns of Text with Regular Expressions
         1. Creating Regex Objects
         2. Matching Regex Objects
         3. Review of Regular Expression Matching
      3. More Pattern Matching with Regular Expressions
         1. Grouping with Parentheses
         2. Matching Multiple Groups with the Pipe
         3. Optional Matching with the Question Mark
         4. Matching Zero or More with the Star
         5. Matching One or More with the Plus
         6. Matching Specific Repetitions with Curly Brackets
      4. Greedy and Nongreedy Matching
      5. The findall() Method
      6. Character Classes
      7. Making Your Own Character Classes
      8. The Caret and Dollar Sign Characters
      9. The Wildcard Character
         1. Matching Everything with Dot-Star
         2. Matching Newlines with the Dot Character
      10. Review of Regex Symbols
      11. Case-Insensitive Matching
      12. Substituting Strings with the sub() Method
      13. Managing Complex Regexes
      14. Combining re.IGNORECASE, re.DOTALL, and re.VERBOSE
      15. Project: Phone Number and Email Address Extractor
          1. Step 1: Create a Regex for Phone Numbers
          2. Step 2: Create a Regex for Email Addresses
          3. Step 3: Find All Matches in the Clipboard Text
          4. Step 4: Join the Matches into a String for the Clipboard
          5. Running the Program
          6. Ideas for Similar Programs
      16. Summary
      17. Practice Questions
      18. Practice Projects
          1. Strong Password Detection
          2. Regex Version of strip()
   2. 8. Reading and Writing Files
      1. Files and File Paths
         1. Backslash on Windows and Forward Slash on OS X and Linux
         2. The Current Working Directory
         3. Absolute vs. Relative Paths
         4. Creating New Folders with os.makedirs()
      2. The os.path Module
         1. Handling Absolute and Relative Paths
         2. Finding File Sizes and Folder Contents
         3. Checking Path Validity
      3. The File Reading/Writing Process
         1. Opening Files with the open() Function
         2. Reading the Contents of Files
         3. Writing to Files
      4. Saving Variables with the shelve Module
      5. Saving Variables with the pprint.pformat() Function
      6. Project: Generating Random Quiz Files
         1. Step 1: Store the Quiz Data in a Dictionary
         2. Step 2: Create the Quiz File and Shuffle the Question Order
         3. Step 3: Create the Answer Options
         4. Step 4: Write Content to the Quiz and Answer Key Files
      7. Project: Multiclipboard
         1. Step 1: Comments and Shelf Setup
         2. Step 2: Save Clipboard Content with a Keyword
         3. Step 3: List Keywords and Load a Keyword’s Content
      8. Summary
      9. Practice Questions
      10. Practice Projects
          1. Extending the Multiclipboard
          2. Mad Libs
          3. Regex Search
   3. 9. Organizing Files
      1. The shutil Module
         1. Copying Files and Folders
         2. Moving and Renaming Files and Folders
         3. Permanently Deleting Files and Folders
         4. Safe Deletes with the send2trash Module
      2. Walking a Directory Tree
      3. Compressing Files with the zipfile Module
         1. Reading ZIP Files
         2. Extracting from ZIP Files
         3. Creating and Adding to ZIP Files
      4. Project: Renaming Files with American-Style Dates to European-Style Dates
         1. Step 1: Create a Regex for American-Style Dates
         2. Step 2: Identify the Date Parts from the Filenames
         3. Step 3: Form the New Filename and Rename the Files
         4. Ideas for Similar Programs
      5. Project: Backing Up a Folder into a ZIP File
         1. Step 1: Figure Out the ZIP File’s Name
         2. Step 2: Create the New ZIP File
         3. Step 3: Walk the Directory Tree and Add to the ZIP File
         4. Ideas for Similar Programs
      6. Summary
      7. Practice Questions
      8. Practice Projects
         1. Selective Copy
         2. Deleting Unneeded Files
         3. Filling in the Gaps
   4. 10. Debugging
      1. Raising Exceptions
      2. Getting the Traceback as a String
      3. Assertions
         1. Using an Assertion in a Traffic Light Simulation
         2. Disabling Assertions
      4. Logging
         1. Using the logging Module
         2. Don’t Debug with print()
         3. Logging Levels
         4. Disabling Logging
         5. Logging to a File
      5. IDLE’s Debugger
         1. Go
         2. Step
         3. Over
         4. Out
         5. Quit
         6. Debugging a Number Adding Program
         7. Breakpoints
      6. Summary
      7. Practice Questions
      8. Practice Project
         1. Debugging Coin Toss
   5. 11. Web Scraping
      1. Project: mapit.py with the webbrowser Module
         1. Step 1: Figure Out the URL
         2. Step 2: Handle the Command Line Arguments
         3. Step 3: Handle the Clipboard Content and Launch the Browser
         4. Ideas for Similar Programs
      2. Downloading Files from the Web with the requests Module
         1. Downloading a Web Page with the requests.get() Function
         2. Checking for Errors
      3. Saving Downloaded Files to the Hard Drive
      4. HTML
         1. Resources for Learning HTML
         2. A Quick Refresher
         3. Viewing the Source HTML of a Web Page
         4. Opening Your Browser’s Developer Tools
         5. Using the Developer Tools to Find HTML Elements
      5. Parsing HTML with the BeautifulSoup Module
         1. Creating a BeautifulSoup Object from HTML
         2. Finding an Element with the select() Method
         3. Getting Data from an Element’s Attributes
      6. Project: “I’m Feeling Lucky” Google Search
         1. Step 1: Get the Command Line Arguments and Request the Search Page
         2. Step 2: Find All the Results
         3. Step 3: Open Web Browsers for Each Result
         4. Ideas for Similar Programs
      7. Project: Downloading All XKCD Comics
         1. Step 1: Design the Program
         2. Step 2: Download the Web Page
         3. Step 3: Find and Download the Comic Image
         4. Step 4: Save the Image and Find the Previous Comic
         5. Ideas for Similar Programs
      8. Controlling the Browser with the selenium Module
         1. Starting a Selenium-Controlled Browser
         2. Finding Elements on the Page
         3. Clicking the Page
         4. Filling Out and Submitting Forms
         5. Sending Special Keys
         6. Clicking Browser Buttons
         7. More Information on Selenium
      9. Summary
      10. Practice Questions
      11. Practice Projects
          1. Command Line Emailer
          2. Image Site Downloader
          3. 2048
          4. Link Verification
   6. 12. Working with Excel Spreadsheets
      1. Excel Documents
      2. Installing the openpyxl Module
      3. Reading Excel Documents
         1. Opening Excel Documents with OpenPyXL
         2. Getting Sheets from the Workbook
         3. Getting Cells from the Sheets
         4. Converting Between Column Letters and Numbers
         5. Getting Rows and Columns from the Sheets
         6. Workbooks, Sheets, Cells
      4. Project: Reading Data from a Spreadsheet
         1. Step 1: Read the Spreadsheet Data
         2. Step 2: Populate the Data Structure
         3. Step 3: Write the Results to a File
         4. Ideas for Similar Programs
      5. Writing Excel Documents
         1. Creating and Saving Excel Documents
         2. Creating and Removing Sheets
         3. Writing Values to Cells
      6. Project: Updating a Spreadsheet
         1. Step 1: Set Up a Data Structure with the Update Information
         2. Step 2: Check All Rows and Update Incorrect Prices
         3. Ideas for Similar Programs
      7. Setting the Font Style of Cells
      8. Font Objects
      9. Formulas
      10. Adjusting Rows and Columns
          1. Setting Row Height and Column Width
          2. Merging and Unmerging Cells
          3. Freeze Panes
      11. Charts
      12. Summary
      13. Practice Questions
      14. Practice Projects
          1. Multiplication Table Maker
          2. Blank Row Inserter
          3. Spreadsheet Cell Inverter
          4. Text Files to Spreadsheet
          5. Spreadsheet to Text Files
   7. 13. Working with PDF and word Documents
      1. PDF Documents
         1. Extracting Text from PDFs
         2. Decrypting PDFs
         3. Creating PDFs
            1. Copying Pages
            2. Rotating Pages
            3. Overlaying Pages
            4. Encrypting PDFs
      2. Project: Combining Select Pages from Many PDFs
         1. Step 1: Find All PDF Files
         2. Step 2: Open Each PDF
         3. Step 3: Add Each Page
         4. Step 4: Save the Results
         5. Ideas for Similar Programs
      3. Word Documents
         1. Reading Word Documents
         2. Getting the Full Text from a .docx File
         3. Styling Paragraph and Run Objects
         4. Creating Word Documents with Nondefault Styles
         5. Run Attributes
         6. Writing Word Documents
         7. Adding Headings
         8. Adding Line and Page Breaks
         9. Adding Pictures
      4. Summary
      5. Practice Questions
      6. Practice Projects
         1. PDF Paranoia
         2. Custom Invitations as Word Documents
         3. Brute-Force PDF Password Breaker
   8. 14. Working with CSV Files and JSON Data
      1. The CSV Module
         1. Reader Objects
         2. Reading Data from Reader Objects in a for Loop
         3. Writer Objects
         4. The delimiter and lineterminator Keyword Arguments
      2. Project: Removing the Header from CSV Files
         1. Step 1: Loop Through Each CSV File
         2. Step 2: Read in the CSV File
         3. Step 3: Write Out the CSV File Without the First Row
         4. Ideas for Similar Programs
      3. JSON and APIs
      4. The JSON Module
         1. Reading JSON with the loads() Function
         2. Writing JSON with the dumps() Function
      5. Project: Fetching Current Weather Data
         1. Step 1: Get Location from the Command Line Argument
         2. Step 2: Download the JSON Data
         3. Step 3: Load JSON Data and Print Weather
         4. Ideas for Similar Programs
      6. Summary
      7. Practice Questions
      8. Practice Project
         1. Excel-to-CSV Converter
   9. 15. Keeping Time, Scheduling Tasks, and Launching Programs
      1. The time Module
         1. The time.time() Function
         2. The time.sleep() Function
      2. Rounding Numbers
      3. Project: Super Stopwatch
         1. Step 1: Set Up the Program to Track Times
         2. Step 2: Track and Print Lap Times
         3. Ideas for Similar Programs
      4. The datetime Module
         1. The timedelta Data Type
         2. Pausing Until a Specific Date
         3. Converting datetime Objects into Strings
         4. Converting Strings into datetime Objects
      5. Review of Python’s Time Functions
      6. Multithreading
         1. Passing Arguments to the Thread’s Target Function
         2. Concurrency Issues
      7. Project: Multithreaded XKCD Downloader
         1. Step 1: Modify the Program to Use a Function
         2. Step 2: Create and Start Threads
         3. Step 3: Wait for All Threads to End
      8. Launching Other Programs from Python
         1. Passing Command Line Arguments to Popen()
         2. Task Scheduler, launchd, and cron
         3. Opening Websites with Python
         4. Running Other Python Scripts
         5. Opening Files with Default Applications
      9. Project: Simple Countdown Program
         1. Step 1: Count Down
         2. Step 2: Play the Sound File
         3. Ideas for Similar Programs
      10. Summary
      11. Practice Questions
      12. Practice Projects
          1. Prettified Stopwatch
          2. Scheduled Web Comic Downloader
   10. 16. Sending Email and Text Messages
       1. SMTP
       2. Sending Email
          1. Connecting to an SMTP Server
          2. Sending the SMTP “Hello” Message
          3. Starting TLS Encryption
          4. Logging in to the SMTP Server
          5. Sending an Email
          6. Disconnecting from the SMTP Server
       3. IMAP
       4. Retrieving and Deleting Emails with IMAP
          1. Connecting to an IMAP Server
          2. Logging in to the IMAP Server
          3. Searching for Email
             1. Selecting a Folder
             2. Performing the Search
             3. Size Limits
          4. Fetching an Email and Marking It As Read
          5. Getting Email Addresses from a Raw Message
          6. Getting the Body from a Raw Message
          7. Deleting Emails
          8. Disconnecting from the IMAP Server
       5. Project: Sending Member Dues Reminder Emails
          1. Step 1: Open the Excel File
          2. Step 2: Find All Unpaid Members
          3. Step 3: Send Customized Email Reminders
       6. Sending Text Messages with Twilio
          1. Signing Up for a Twilio Account
          2. Sending Text Messages
       7. Project: “Just Text Me” Module
       8. Summary
       9. Practice Questions
       10. Practice Projects
           1. Random Chore Assignment Emailer
           2. Umbrella Reminder
           3. Auto Unsubscriber
           4. Controlling Your Computer Through Email
   11. 17. Manipulating Images
       1. Computer Image Fundamentals
          1. Colors and RGBA Values
          2. Coordinates and Box Tuples
       2. Manipulating Images with Pillow
          1. Working with the Image Data Type
          2. Cropping Images
          3. Copying and Pasting Images onto Other Images
          4. Resizing an Image
          5. Rotating and Flipping Images
          6. Changing Individual Pixels
       3. Project: Adding a Logo
          1. Step 1: Open the Logo Image
          2. Step 2: Loop Over All Files and Open Images
          3. Step 3: Resize the Images
          4. Step 4: Add the Logo and Save the Changes
          5. Ideas for Similar Programs
       4. Drawing on Images
          1. Drawing Shapes
             1. Points
             2. Lines
             3. Rectangles
             4. Ellipses
             5. Polygons
             6. Drawing Example
          2. Drawing Text
       5. Summary
       6. Practice Questions
       7. Practice Projects
          1. Extending and Fixing the Chapter Project Programs
          2. Identifying Photo Folders on the Hard Drive
          3. Custom Seating Cards
   12. 18. Controlling the Keyboard and Mouse with GUI Automation
       1. Installing the pyautogui Module
       2. Staying on Track
          1. Shutting Down Everything by Logging Out
          2. Pauses and Fail-Safes
       3. Controlling Mouse Movement
          1. Moving the Mouse
          2. Getting the Mouse Position
       4. Project: “Where Is the Mouse Right Now?”
          1. Step 1: Import the Module
          2. Step 2: Set Up the Quit Code and Infinite Loop
          3. Step 3: Get and Print the Mouse Coordinates
       5. Controlling Mouse Interaction
          1. Clicking the Mouse
          2. Dragging the Mouse
          3. Scrolling the Mouse
       6. Working with the Screen
          1. Getting a Screenshot
          2. Analyzing the Screenshot
       7. Project: Extending the mouseNow Program
       8. Image Recognition
       9. Controlling the Keyboard
          1. Sending a String from the Keyboard
          2. Key Names
          3. Pressing and Releasing the Keyboard
          4. Hotkey Combinations
       10. Review of the PyAutoGUI Functions
       11. Project: Automatic Form Filler
           1. Step 1: Figure Out the Steps
           2. Step 2: Set Up Coordinates
           3. Step 3: Start Typing Data
           4. Step 4: Handle Select Lists and Radio Buttons
           5. Step 5: Submit the Form and Wait
       12. Summary
       13. Practice Questions
       14. Practice Projects
           1. Looking Busy
           2. Instant Messenger Bot
           3. Game-Playing Bot Tutorial
9. A. Installing Third-Party Modules
   1. The pip Tool
   2. Installing Third-Party Modules
10. B. Running Programs
    1. Shebang Line
    2. Running Python Programs on Windows
    3. Running Python Programs on OS X and Linux
11. C. Answers to the Practice Questions
    1. Chapter 1
    2. Chapter 2
    3. Chapter 3
    4. Chapter 4
    5. Chapter 5
    6. Chapter 6
    7. Chapter 7
    8. Chapter 8
    9. Chapter 9
    10. Chapter 10
    11. Chapter 11
    12. Chapter 12
    13. Chapter 13
    14. Chapter 14
    15. Chapter 15
    16. Chapter 16
    17. Chapter 17
    18. Chapter 18
12. D. Resources
13. Index
14. Copyright