book

Automate the Boring Stuff with Python

Name: Automate the Boring Stuff with Python
Author: Al Sweigart
ISBN: 9781593275990

by Al Sweigart

April 2015

Intermediate to advanced

504 pages

12h 52m

English

No Starch Press

Read now

Unlock full access

Automate the Boring Stuff with Python: Practical Programming for Total Beginners
Dedication
About the Author
About the Tech Reviewer
Acknowledgments
Introduction
Whom Is This Book For?ConventionsWhat Is Programming?What Is Python?Programmers Don’t Need to Know Much MathProgramming Is a Creative ActivityAbout This BookDownloading and Installing PythonStarting IDLEThe Interactive ShellHow to Find HelpAsking Smart Programming QuestionsSummary
I. Python Programming Basics
1. Python Basics
Entering Expressions into the Interactive ShellThe Integer, Floating-Point, and String Data TypesString Concatenation and ReplicationStoring Values in VariablesAssignment StatementsVariable NamesYour First ProgramDissecting Your ProgramCommentsThe print() FunctionThe input() FunctionPrinting the User’s NameThe len() FunctionThe str(), int(), and float() FunctionsSummaryPractice Questions
2. Flow Control
Boolean ValuesComparison OperatorsBoolean OperatorsBinary Boolean OperatorsThe not OperatorMixing Boolean and Comparison OperatorsElements of Flow ControlConditionsBlocks of CodeProgram ExecutionFlow Control Statementsif Statementselse Statementselif Statementswhile Loop StatementsAn Annoying while Loopbreak Statementscontinue Statementsfor Loops and the range() FunctionAn Equivalent while LoopThe Starting, Stopping, and Stepping Arguments to range()Importing Modulesfrom import StatementsEnding a Program Early with sys.exit()SummaryPractice Questions
3. Functions
def Statements with ParametersReturn Values and return StatementsThe None ValueKeyword Arguments and print()Local and Global ScopeLocal Variables Cannot Be Used in the Global ScopeLocal Scopes Cannot Use Variables in Other Local ScopesGlobal Variables Can Be Read from a Local ScopeLocal and Global Variables with the Same NameThe global StatementException HandlingA Short Program: Guess the NumberSummaryPractice QuestionsPractice ProjectsThe Collatz SequenceInput Validation

4. Lists
The List Data TypeGetting Individual Values in a List with IndexesNegative IndexesGetting Sublists with SlicesGetting a List’s Length with len()Changing Values in a List with IndexesList Concatenation and List ReplicationRemoving Values from Lists with del StatementsWorking with ListsUsing for Loops with ListsThe in and not in OperatorsThe Multiple Assignment TrickAugmented Assignment OperatorsMethodsFinding a Value in a List with the index() MethodAdding Values to Lists with the append() and insert() MethodsRemoving Values from Lists with remove()Sorting the Values in a List with the sort() MethodExample Program: Magic 8 Ball with a ListList-like Types: Strings and TuplesMutable and Immutable Data TypesThe Tuple Data TypeConverting Types with the list() and tuple() FunctionsReferencesPassing ReferencesThe copy Module’s copy() and deepcopy() FunctionsSummaryPractice QuestionsPractice ProjectsComma CodeCharacter Picture Grid
5. Dictionaries and Structuring Data
The Dictionary Data TypeDictionaries vs. ListsThe keys(), values(), and items() MethodsChecking Whether a Key or Value Exists in a DictionaryThe get() MethodThe setdefault() MethodPretty PrintingUsing Data Structures to Model Real-World ThingsA Tic-Tac-Toe BoardNested Dictionaries and ListsSummaryPractice QuestionsPractice ProjectsFantasy Game InventoryList to Dictionary Function for Fantasy Game Inventory
6. Manipulating Strings
Working with StringsString LiteralsDouble QuotesEscape CharactersRaw StringsMultiline Strings with Triple QuotesMultiline CommentsIndexing and Slicing StringsThe in and not in Operators with StringsUseful String MethodsThe upper(), lower(), isupper(), and islower() String MethodsThe isX String MethodsThe startswith() and endswith() String MethodsThe join() and split() String MethodsJustifying Text with rjust(), ljust(), and center()Removing Whitespace with strip(), rstrip(), and lstrip()Copying and Pasting Strings with the pyperclip ModuleProject: Password LockerStep 1: Program Design and Data StructuresStep 2: Handle Command Line ArgumentsStep 3: Copy the Right PasswordProject: Adding Bullets to Wiki MarkupStep 1: Copy and Paste from the ClipboardStep 2: Separate the Lines of Text and Add the StarStep 3: Join the Modified LinesSummaryPractice QuestionsPractice ProjectTable Printer
II. Automating Tasks
7. Pattern Matching with Regular Expressions
Finding Patterns of Text Without Regular ExpressionsFinding Patterns of Text with Regular ExpressionsCreating Regex ObjectsMatching Regex ObjectsReview of Regular Expression MatchingMore Pattern Matching with Regular ExpressionsGrouping with ParenthesesMatching Multiple Groups with the PipeOptional Matching with the Question MarkMatching Zero or More with the StarMatching One or More with the PlusMatching Specific Repetitions with Curly BracketsGreedy and Nongreedy MatchingThe findall() MethodCharacter ClassesMaking Your Own Character ClassesThe Caret and Dollar Sign CharactersThe Wildcard CharacterMatching Everything with Dot-StarMatching Newlines with the Dot CharacterReview of Regex SymbolsCase-Insensitive MatchingSubstituting Strings with the sub() MethodManaging Complex RegexesCombining re.IGNORECASE, re.DOTALL, and re.VERBOSEProject: Phone Number and Email Address ExtractorStep 1: Create a Regex for Phone NumbersStep 2: Create a Regex for Email AddressesStep 3: Find All Matches in the Clipboard TextStep 4: Join the Matches into a String for the ClipboardRunning the ProgramIdeas for Similar ProgramsSummaryPractice QuestionsPractice ProjectsStrong Password DetectionRegex Version of strip()
8. Reading and Writing Files
Files and File PathsBackslash on Windows and Forward Slash on OS X and LinuxThe Current Working DirectoryAbsolute vs. Relative PathsCreating New Folders with os.makedirs()The os.path ModuleHandling Absolute and Relative PathsFinding File Sizes and Folder ContentsChecking Path ValidityThe File Reading/Writing ProcessOpening Files with the open() FunctionReading the Contents of FilesWriting to FilesSaving Variables with the shelve ModuleSaving Variables with the pprint.pformat() FunctionProject: Generating Random Quiz FilesStep 1: Store the Quiz Data in a DictionaryStep 2: Create the Quiz File and Shuffle the Question OrderStep 3: Create the Answer OptionsStep 4: Write Content to the Quiz and Answer Key FilesProject: MulticlipboardStep 1: Comments and Shelf SetupStep 2: Save Clipboard Content with a KeywordStep 3: List Keywords and Load a Keyword’s ContentSummaryPractice QuestionsPractice ProjectsExtending the MulticlipboardMad LibsRegex Search
9. Organizing Files
The shutil ModuleCopying Files and FoldersMoving and Renaming Files and FoldersPermanently Deleting Files and FoldersSafe Deletes with the send2trash ModuleWalking a Directory TreeCompressing Files with the zipfile ModuleReading ZIP FilesExtracting from ZIP FilesCreating and Adding to ZIP FilesProject: Renaming Files with American-Style Dates to European-Style DatesStep 1: Create a Regex for American-Style DatesStep 2: Identify the Date Parts from the FilenamesStep 3: Form the New Filename and Rename the FilesIdeas for Similar ProgramsProject: Backing Up a Folder into a ZIP FileStep 1: Figure Out the ZIP File’s NameStep 2: Create the New ZIP FileStep 3: Walk the Directory Tree and Add to the ZIP FileIdeas for Similar ProgramsSummaryPractice QuestionsPractice ProjectsSelective CopyDeleting Unneeded FilesFilling in the Gaps
10. Debugging
Raising ExceptionsGetting the Traceback as a StringAssertionsUsing an Assertion in a Traffic Light SimulationDisabling AssertionsLoggingUsing the logging ModuleDon’t Debug with print()Logging LevelsDisabling LoggingLogging to a FileIDLE’s DebuggerGoStepOverOutQuitDebugging a Number Adding ProgramBreakpointsSummaryPractice QuestionsPractice ProjectDebugging Coin Toss
11. Web Scraping
Project: mapit.py with the webbrowser ModuleStep 1: Figure Out the URLStep 2: Handle the Command Line ArgumentsStep 3: Handle the Clipboard Content and Launch the BrowserIdeas for Similar ProgramsDownloading Files from the Web with the requests ModuleDownloading a Web Page with the requests.get() FunctionChecking for ErrorsSaving Downloaded Files to the Hard DriveHTMLResources for Learning HTMLA Quick RefresherViewing the Source HTML of a Web PageOpening Your Browser’s Developer ToolsUsing the Developer Tools to Find HTML ElementsParsing HTML with the BeautifulSoup ModuleCreating a BeautifulSoup Object from HTMLFinding an Element with the select() MethodGetting Data from an Element’s AttributesProject: “I’m Feeling Lucky” Google SearchStep 1: Get the Command Line Arguments and Request the Search PageStep 2: Find All the ResultsStep 3: Open Web Browsers for Each ResultIdeas for Similar ProgramsProject: Downloading All XKCD ComicsStep 1: Design the ProgramStep 2: Download the Web PageStep 3: Find and Download the Comic ImageStep 4: Save the Image and Find the Previous ComicIdeas for Similar ProgramsControlling the Browser with the selenium ModuleStarting a Selenium-Controlled BrowserFinding Elements on the PageClicking the PageFilling Out and Submitting FormsSending Special KeysClicking Browser ButtonsMore Information on SeleniumSummaryPractice QuestionsPractice ProjectsCommand Line EmailerImage Site Downloader2048Link Verification
12. Working with Excel Spreadsheets
Excel DocumentsInstalling the openpyxl ModuleReading Excel DocumentsOpening Excel Documents with OpenPyXLGetting Sheets from the WorkbookGetting Cells from the SheetsConverting Between Column Letters and NumbersGetting Rows and Columns from the SheetsWorkbooks, Sheets, CellsProject: Reading Data from a SpreadsheetStep 1: Read the Spreadsheet DataStep 2: Populate the Data StructureStep 3: Write the Results to a FileIdeas for Similar ProgramsWriting Excel DocumentsCreating and Saving Excel DocumentsCreating and Removing SheetsWriting Values to CellsProject: Updating a SpreadsheetStep 1: Set Up a Data Structure with the Update InformationStep 2: Check All Rows and Update Incorrect PricesIdeas for Similar ProgramsSetting the Font Style of CellsFont ObjectsFormulasAdjusting Rows and ColumnsSetting Row Height and Column WidthMerging and Unmerging CellsFreeze PanesChartsSummaryPractice QuestionsPractice ProjectsMultiplication Table MakerBlank Row InserterSpreadsheet Cell InverterText Files to SpreadsheetSpreadsheet to Text Files
13. Working with PDF and word Documents
PDF DocumentsExtracting Text from PDFsDecrypting PDFsCreating PDFsCopying PagesRotating PagesOverlaying PagesEncrypting PDFsProject: Combining Select Pages from Many PDFsStep 1: Find All PDF FilesStep 2: Open Each PDFStep 3: Add Each PageStep 4: Save the ResultsIdeas for Similar ProgramsWord DocumentsReading Word DocumentsGetting the Full Text from a .docx FileStyling Paragraph and Run ObjectsCreating Word Documents with Nondefault StylesRun AttributesWriting Word DocumentsAdding HeadingsAdding Line and Page BreaksAdding PicturesSummaryPractice QuestionsPractice ProjectsPDF ParanoiaCustom Invitations as Word DocumentsBrute-Force PDF Password Breaker
14. Working with CSV Files and JSON Data
The CSV ModuleReader ObjectsReading Data from Reader Objects in a for LoopWriter ObjectsThe delimiter and lineterminator Keyword ArgumentsProject: Removing the Header from CSV FilesStep 1: Loop Through Each CSV FileStep 2: Read in the CSV FileStep 3: Write Out the CSV File Without the First RowIdeas for Similar ProgramsJSON and APIsThe JSON ModuleReading JSON with the loads() FunctionWriting JSON with the dumps() FunctionProject: Fetching Current Weather DataStep 1: Get Location from the Command Line ArgumentStep 2: Download the JSON DataStep 3: Load JSON Data and Print WeatherIdeas for Similar ProgramsSummaryPractice QuestionsPractice ProjectExcel-to-CSV Converter
15. Keeping Time, Scheduling Tasks, and Launching Programs
The time ModuleThe time.time() FunctionThe time.sleep() FunctionRounding NumbersProject: Super StopwatchStep 1: Set Up the Program to Track TimesStep 2: Track and Print Lap TimesIdeas for Similar ProgramsThe datetime ModuleThe timedelta Data TypePausing Until a Specific DateConverting datetime Objects into StringsConverting Strings into datetime ObjectsReview of Python’s Time FunctionsMultithreadingPassing Arguments to the Thread’s Target FunctionConcurrency IssuesProject: Multithreaded XKCD DownloaderStep 1: Modify the Program to Use a FunctionStep 2: Create and Start ThreadsStep 3: Wait for All Threads to EndLaunching Other Programs from PythonPassing Command Line Arguments to Popen()Task Scheduler, launchd, and cronOpening Websites with PythonRunning Other Python ScriptsOpening Files with Default ApplicationsProject: Simple Countdown ProgramStep 1: Count DownStep 2: Play the Sound FileIdeas for Similar ProgramsSummaryPractice QuestionsPractice ProjectsPrettified StopwatchScheduled Web Comic Downloader
16. Sending Email and Text Messages
SMTPSending EmailConnecting to an SMTP ServerSending the SMTP “Hello” MessageStarting TLS EncryptionLogging in to the SMTP ServerSending an EmailDisconnecting from the SMTP ServerIMAPRetrieving and Deleting Emails with IMAPConnecting to an IMAP ServerLogging in to the IMAP ServerSearching for EmailSelecting a FolderPerforming the SearchSize LimitsFetching an Email and Marking It As ReadGetting Email Addresses from a Raw MessageGetting the Body from a Raw MessageDeleting EmailsDisconnecting from the IMAP ServerProject: Sending Member Dues Reminder EmailsStep 1: Open the Excel FileStep 2: Find All Unpaid MembersStep 3: Send Customized Email RemindersSending Text Messages with TwilioSigning Up for a Twilio AccountSending Text MessagesProject: “Just Text Me” ModuleSummaryPractice QuestionsPractice ProjectsRandom Chore Assignment EmailerUmbrella ReminderAuto UnsubscriberControlling Your Computer Through Email
17. Manipulating Images
Computer Image FundamentalsColors and RGBA ValuesCoordinates and Box TuplesManipulating Images with PillowWorking with the Image Data TypeCropping ImagesCopying and Pasting Images onto Other ImagesResizing an ImageRotating and Flipping ImagesChanging Individual PixelsProject: Adding a LogoStep 1: Open the Logo ImageStep 2: Loop Over All Files and Open ImagesStep 3: Resize the ImagesStep 4: Add the Logo and Save the ChangesIdeas for Similar ProgramsDrawing on ImagesDrawing ShapesPointsLinesRectanglesEllipsesPolygonsDrawing ExampleDrawing TextSummaryPractice QuestionsPractice ProjectsExtending and Fixing the Chapter Project ProgramsIdentifying Photo Folders on the Hard DriveCustom Seating Cards
18. Controlling the Keyboard and Mouse with GUI Automation
Installing the pyautogui ModuleStaying on TrackShutting Down Everything by Logging OutPauses and Fail-SafesControlling Mouse MovementMoving the MouseGetting the Mouse PositionProject: “Where Is the Mouse Right Now?”Step 1: Import the ModuleStep 2: Set Up the Quit Code and Infinite LoopStep 3: Get and Print the Mouse CoordinatesControlling Mouse InteractionClicking the MouseDragging the MouseScrolling the MouseWorking with the ScreenGetting a ScreenshotAnalyzing the ScreenshotProject: Extending the mouseNow ProgramImage RecognitionControlling the KeyboardSending a String from the KeyboardKey NamesPressing and Releasing the KeyboardHotkey CombinationsReview of the PyAutoGUI FunctionsProject: Automatic Form FillerStep 1: Figure Out the StepsStep 2: Set Up CoordinatesStep 3: Start Typing DataStep 4: Handle Select Lists and Radio ButtonsStep 5: Submit the Form and WaitSummaryPractice QuestionsPractice ProjectsLooking BusyInstant Messenger BotGame-Playing Bot Tutorial
A. Installing Third-Party Modules
The pip ToolInstalling Third-Party Modules
B. Running Programs
Shebang LineRunning Python Programs on WindowsRunning Python Programs on OS X and LinuxRunning Python Programs with Assertions Disabled
C. Answers to the Practice Questions
Chapter 1Chapter 2Chapter 3Chapter 4Chapter 5Chapter 6Chapter 7Chapter 8Chapter 9Chapter 10Chapter 11Chapter 12Chapter 13Chapter 14Chapter 15Chapter 16Chapter 17Chapter 18
D. Resources
Index
Copyright

Content preview from Automate the Boring Stuff with Python

Chapter 11. Web Scraping

In those rare, terrifying moments when I’m without Wi-Fi, I realize just how much of what I do on the computer is really what I do on the Internet. Out of sheer habit I’ll find myself trying to check email, read friends’ Twitter feeds, or answer the question, “Did Kurtwood Smith have any major roles before he was in the original 1987 RoboCop?”^[2]

Since so much work on a computer involves going on the Internet, it’d be great if your programs could get online. Web scraping is the term for using a program to download and process content from the Web. For example, Google runs many web scraping programs to index web pages for its search engine. In this chapter, you will learn about several modules that make it easy to scrape ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Automate the Boring Stuff with Python, 2nd Edition

Publisher Resources

ISBN: 9781457189906Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Automate the Boring Stuff with Python

by Al Sweigart

Chapter 11. Web Scraping

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.