book

Automate the Boring Stuff with Python

by Al Sweigart

April 2015

Intermediate to advanced

504 pages

12h 52m

English

No Starch Press

Read now

Unlock full access

Whom Is This Book For?ConventionsWhat Is Programming?What Is Python?Programmers Don’t Need to Know Much MathProgramming Is a Creative ActivityAbout This BookDownloading and Installing PythonStarting IDLEThe Interactive ShellHow to Find HelpAsking Smart Programming QuestionsSummary
Entering Expressions into the Interactive ShellThe Integer, Floating-Point, and String Data TypesString Concatenation and ReplicationStoring Values in VariablesAssignment StatementsVariable NamesYour First ProgramDissecting Your ProgramCommentsThe print() FunctionThe input() FunctionPrinting the User’s NameThe len() FunctionThe str(), int(), and float() FunctionsSummaryPractice Questions
Boolean ValuesComparison OperatorsBoolean OperatorsBinary Boolean OperatorsThe not OperatorMixing Boolean and Comparison OperatorsElements of Flow ControlConditionsBlocks of CodeProgram ExecutionFlow Control Statementsif Statementselse Statementselif Statementswhile Loop StatementsAn Annoying while Loopbreak Statementscontinue Statementsfor Loops and the range() FunctionAn Equivalent while LoopThe Starting, Stopping, and Stepping Arguments to range()Importing Modulesfrom import StatementsEnding a Program Early with sys.exit()SummaryPractice Questions
def Statements with ParametersReturn Values and return StatementsThe None ValueKeyword Arguments and print()Local and Global ScopeLocal Variables Cannot Be Used in the Global ScopeLocal Scopes Cannot Use Variables in Other Local ScopesGlobal Variables Can Be Read from a Local ScopeLocal and Global Variables with the Same NameThe global StatementException HandlingA Short Program: Guess the NumberSummaryPractice QuestionsPractice ProjectsThe Collatz SequenceInput Validation

The List Data TypeGetting Individual Values in a List with IndexesNegative IndexesGetting Sublists with SlicesGetting a List’s Length with len()Changing Values in a List with IndexesList Concatenation and List ReplicationRemoving Values from Lists with del StatementsWorking with ListsUsing for Loops with ListsThe in and not in OperatorsThe Multiple Assignment TrickAugmented Assignment OperatorsMethodsFinding a Value in a List with the index() MethodAdding Values to Lists with the append() and insert() MethodsRemoving Values from Lists with remove()Sorting the Values in a List with the sort() MethodExample Program: Magic 8 Ball with a ListList-like Types: Strings and TuplesMutable and Immutable Data TypesThe Tuple Data TypeConverting Types with the list() and tuple() FunctionsReferencesPassing ReferencesThe copy Module’s copy() and deepcopy() FunctionsSummaryPractice QuestionsPractice ProjectsComma CodeCharacter Picture Grid
The Dictionary Data TypeDictionaries vs. ListsThe keys(), values(), and items() MethodsChecking Whether a Key or Value Exists in a DictionaryThe get() MethodThe setdefault() MethodPretty PrintingUsing Data Structures to Model Real-World ThingsA Tic-Tac-Toe BoardNested Dictionaries and ListsSummaryPractice QuestionsPractice ProjectsFantasy Game InventoryList to Dictionary Function for Fantasy Game Inventory
Working with StringsString LiteralsDouble QuotesEscape CharactersRaw StringsMultiline Strings with Triple QuotesMultiline CommentsIndexing and Slicing StringsThe in and not in Operators with StringsUseful String MethodsThe upper(), lower(), isupper(), and islower() String MethodsThe isX String MethodsThe startswith() and endswith() String MethodsThe join() and split() String MethodsJustifying Text with rjust(), ljust(), and center()Removing Whitespace with strip(), rstrip(), and lstrip()Copying and Pasting Strings with the pyperclip ModuleProject: Password LockerStep 1: Program Design and Data StructuresStep 2: Handle Command Line ArgumentsStep 3: Copy the Right PasswordProject: Adding Bullets to Wiki MarkupStep 1: Copy and Paste from the ClipboardStep 2: Separate the Lines of Text and Add the StarStep 3: Join the Modified LinesSummaryPractice QuestionsPractice ProjectTable Printer
Finding Patterns of Text Without Regular ExpressionsFinding Patterns of Text with Regular ExpressionsCreating Regex ObjectsMatching Regex ObjectsReview of Regular Expression MatchingMore Pattern Matching with Regular ExpressionsGrouping with ParenthesesMatching Multiple Groups with the PipeOptional Matching with the Question MarkMatching Zero or More with the StarMatching One or More with the PlusMatching Specific Repetitions with Curly BracketsGreedy and Nongreedy MatchingThe findall() MethodCharacter ClassesMaking Your Own Character ClassesThe Caret and Dollar Sign CharactersThe Wildcard CharacterMatching Everything with Dot-StarMatching Newlines with the Dot CharacterReview of Regex SymbolsCase-Insensitive MatchingSubstituting Strings with the sub() MethodManaging Complex RegexesCombining re.IGNORECASE, re.DOTALL, and re.VERBOSEProject: Phone Number and Email Address ExtractorStep 1: Create a Regex for Phone NumbersStep 2: Create a Regex for Email AddressesStep 3: Find All Matches in the Clipboard TextStep 4: Join the Matches into a String for the ClipboardRunning the ProgramIdeas for Similar ProgramsSummaryPractice QuestionsPractice ProjectsStrong Password DetectionRegex Version of strip()
Files and File PathsBackslash on Windows and Forward Slash on OS X and LinuxThe Current Working DirectoryAbsolute vs. Relative PathsCreating New Folders with os.makedirs()The os.path ModuleHandling Absolute and Relative PathsFinding File Sizes and Folder ContentsChecking Path ValidityThe File Reading/Writing ProcessOpening Files with the open() FunctionReading the Contents of FilesWriting to FilesSaving Variables with the shelve ModuleSaving Variables with the pprint.pformat() FunctionProject: Generating Random Quiz FilesStep 1: Store the Quiz Data in a DictionaryStep 2: Create the Quiz File and Shuffle the Question OrderStep 3: Create the Answer OptionsStep 4: Write Content to the Quiz and Answer Key FilesProject: MulticlipboardStep 1: Comments and Shelf SetupStep 2: Save Clipboard Content with a KeywordStep 3: List Keywords and Load a Keyword’s ContentSummaryPractice QuestionsPractice ProjectsExtending the MulticlipboardMad LibsRegex Search
The shutil ModuleCopying Files and FoldersMoving and Renaming Files and FoldersPermanently Deleting Files and FoldersSafe Deletes with the send2trash ModuleWalking a Directory TreeCompressing Files with the zipfile ModuleReading ZIP FilesExtracting from ZIP FilesCreating and Adding to ZIP FilesProject: Renaming Files with American-Style Dates to European-Style DatesStep 1: Create a Regex for American-Style DatesStep 2: Identify the Date Parts from the FilenamesStep 3: Form the New Filename and Rename the FilesIdeas for Similar ProgramsProject: Backing Up a Folder into a ZIP FileStep 1: Figure Out the ZIP File’s NameStep 2: Create the New ZIP FileStep 3: Walk the Directory Tree and Add to the ZIP FileIdeas for Similar ProgramsSummaryPractice QuestionsPractice ProjectsSelective CopyDeleting Unneeded FilesFilling in the Gaps
Raising ExceptionsGetting the Traceback as a StringAssertionsUsing an Assertion in a Traffic Light SimulationDisabling AssertionsLoggingUsing the logging ModuleDon’t Debug with print()Logging LevelsDisabling LoggingLogging to a FileIDLE’s DebuggerGoStepOverOutQuitDebugging a Number Adding ProgramBreakpointsSummaryPractice QuestionsPractice ProjectDebugging Coin Toss
Project: mapit.py with the webbrowser ModuleStep 1: Figure Out the URLStep 2: Handle the Command Line ArgumentsStep 3: Handle the Clipboard Content and Launch the BrowserIdeas for Similar ProgramsDownloading Files from the Web with the requests ModuleDownloading a Web Page with the requests.get() FunctionChecking for ErrorsSaving Downloaded Files to the Hard DriveHTMLResources for Learning HTMLA Quick RefresherViewing the Source HTML of a Web PageOpening Your Browser’s Developer ToolsUsing the Developer Tools to Find HTML ElementsParsing HTML with the BeautifulSoup ModuleCreating a BeautifulSoup Object from HTMLFinding an Element with the select() MethodGetting Data from an Element’s AttributesProject: “I’m Feeling Lucky” Google SearchStep 1: Get the Command Line Arguments and Request the Search PageStep 2: Find All the ResultsStep 3: Open Web Browsers for Each ResultIdeas for Similar ProgramsProject: Downloading All XKCD ComicsStep 1: Design the ProgramStep 2: Download the Web PageStep 3: Find and Download the Comic ImageStep 4: Save the Image and Find the Previous ComicIdeas for Similar ProgramsControlling the Browser with the selenium ModuleStarting a Selenium-Controlled BrowserFinding Elements on the PageClicking the PageFilling Out and Submitting FormsSending Special KeysClicking Browser ButtonsMore Information on SeleniumSummaryPractice QuestionsPractice ProjectsCommand Line EmailerImage Site Downloader2048Link Verification
Excel DocumentsInstalling the openpyxl ModuleReading Excel DocumentsOpening Excel Documents with OpenPyXLGetting Sheets from the WorkbookGetting Cells from the SheetsConverting Between Column Letters and NumbersGetting Rows and Columns from the SheetsWorkbooks, Sheets, CellsProject: Reading Data from a SpreadsheetStep 1: Read the Spreadsheet DataStep 2: Populate the Data StructureStep 3: Write the Results to a FileIdeas for Similar ProgramsWriting Excel DocumentsCreating and Saving Excel DocumentsCreating and Removing SheetsWriting Values to CellsProject: Updating a SpreadsheetStep 1: Set Up a Data Structure with the Update InformationStep 2: Check All Rows and Update Incorrect PricesIdeas for Similar ProgramsSetting the Font Style of CellsFont ObjectsFormulasAdjusting Rows and ColumnsSetting Row Height and Column WidthMerging and Unmerging CellsFreeze PanesChartsSummaryPractice QuestionsPractice ProjectsMultiplication Table MakerBlank Row InserterSpreadsheet Cell InverterText Files to SpreadsheetSpreadsheet to Text Files
PDF DocumentsExtracting Text from PDFsDecrypting PDFsCreating PDFsCopying PagesRotating PagesOverlaying PagesEncrypting PDFsProject: Combining Select Pages from Many PDFsStep 1: Find All PDF FilesStep 2: Open Each PDFStep 3: Add Each PageStep 4: Save the ResultsIdeas for Similar ProgramsWord DocumentsReading Word DocumentsGetting the Full Text from a .docx FileStyling Paragraph and Run ObjectsCreating Word Documents with Nondefault StylesRun AttributesWriting Word DocumentsAdding HeadingsAdding Line and Page BreaksAdding PicturesSummaryPractice QuestionsPractice ProjectsPDF ParanoiaCustom Invitations as Word DocumentsBrute-Force PDF Password Breaker
The CSV ModuleReader ObjectsReading Data from Reader Objects in a for LoopWriter ObjectsThe delimiter and lineterminator Keyword ArgumentsProject: Removing the Header from CSV FilesStep 1: Loop Through Each CSV FileStep 2: Read in the CSV FileStep 3: Write Out the CSV File Without the First RowIdeas for Similar ProgramsJSON and APIsThe JSON ModuleReading JSON with the loads() FunctionWriting JSON with the dumps() FunctionProject: Fetching Current Weather DataStep 1: Get Location from the Command Line ArgumentStep 2: Download the JSON DataStep 3: Load JSON Data and Print WeatherIdeas for Similar ProgramsSummaryPractice QuestionsPractice ProjectExcel-to-CSV Converter
The time ModuleThe time.time() FunctionThe time.sleep() FunctionRounding NumbersProject: Super StopwatchStep 1: Set Up the Program to Track TimesStep 2: Track and Print Lap TimesIdeas for Similar ProgramsThe datetime ModuleThe timedelta Data TypePausing Until a Specific DateConverting datetime Objects into StringsConverting Strings into datetime ObjectsReview of Python’s Time FunctionsMultithreadingPassing Arguments to the Thread’s Target FunctionConcurrency IssuesProject: Multithreaded XKCD DownloaderStep 1: Modify the Program to Use a FunctionStep 2: Create and Start ThreadsStep 3: Wait for All Threads to EndLaunching Other Programs from PythonPassing Command Line Arguments to Popen()Task Scheduler, launchd, and cronOpening Websites with PythonRunning Other Python ScriptsOpening Files with Default ApplicationsProject: Simple Countdown ProgramStep 1: Count DownStep 2: Play the Sound FileIdeas for Similar ProgramsSummaryPractice QuestionsPractice ProjectsPrettified StopwatchScheduled Web Comic Downloader
SMTPSending EmailConnecting to an SMTP ServerSending the SMTP “Hello” MessageStarting TLS EncryptionLogging in to the SMTP ServerSending an EmailDisconnecting from the SMTP ServerIMAPRetrieving and Deleting Emails with IMAPConnecting to an IMAP ServerLogging in to the IMAP ServerSearching for EmailSelecting a FolderPerforming the SearchSize LimitsFetching an Email and Marking It As ReadGetting Email Addresses from a Raw MessageGetting the Body from a Raw MessageDeleting EmailsDisconnecting from the IMAP ServerProject: Sending Member Dues Reminder EmailsStep 1: Open the Excel FileStep 2: Find All Unpaid MembersStep 3: Send Customized Email RemindersSending Text Messages with TwilioSigning Up for a Twilio AccountSending Text MessagesProject: “Just Text Me” ModuleSummaryPractice QuestionsPractice ProjectsRandom Chore Assignment EmailerUmbrella ReminderAuto UnsubscriberControlling Your Computer Through Email
Computer Image FundamentalsColors and RGBA ValuesCoordinates and Box TuplesManipulating Images with PillowWorking with the Image Data TypeCropping ImagesCopying and Pasting Images onto Other ImagesResizing an ImageRotating and Flipping ImagesChanging Individual PixelsProject: Adding a LogoStep 1: Open the Logo ImageStep 2: Loop Over All Files and Open ImagesStep 3: Resize the ImagesStep 4: Add the Logo and Save the ChangesIdeas for Similar ProgramsDrawing on ImagesDrawing ShapesPointsLinesRectanglesEllipsesPolygonsDrawing ExampleDrawing TextSummaryPractice QuestionsPractice ProjectsExtending and Fixing the Chapter Project ProgramsIdentifying Photo Folders on the Hard DriveCustom Seating Cards
Installing the pyautogui ModuleStaying on TrackShutting Down Everything by Logging OutPauses and Fail-SafesControlling Mouse MovementMoving the MouseGetting the Mouse PositionProject: “Where Is the Mouse Right Now?”Step 1: Import the ModuleStep 2: Set Up the Quit Code and Infinite LoopStep 3: Get and Print the Mouse CoordinatesControlling Mouse InteractionClicking the MouseDragging the MouseScrolling the MouseWorking with the ScreenGetting a ScreenshotAnalyzing the ScreenshotProject: Extending the mouseNow ProgramImage RecognitionControlling the KeyboardSending a String from the KeyboardKey NamesPressing and Releasing the KeyboardHotkey CombinationsReview of the PyAutoGUI FunctionsProject: Automatic Form FillerStep 1: Figure Out the StepsStep 2: Set Up CoordinatesStep 3: Start Typing DataStep 4: Handle Select Lists and Radio ButtonsStep 5: Submit the Form and WaitSummaryPractice QuestionsPractice ProjectsLooking BusyInstant Messenger BotGame-Playing Bot Tutorial
The pip ToolInstalling Third-Party Modules
Shebang LineRunning Python Programs on WindowsRunning Python Programs on OS X and LinuxRunning Python Programs with Assertions Disabled
Chapter 1Chapter 2Chapter 3Chapter 4Chapter 5Chapter 6Chapter 7Chapter 8Chapter 9Chapter 10Chapter 11Chapter 12Chapter 13Chapter 14Chapter 15Chapter 16Chapter 17Chapter 18

Content preview from Automate the Boring Stuff with Python

Chapter 11. Web Scraping

In those rare, terrifying moments when I’m without Wi-Fi, I realize just how much of what I do on the computer is really what I do on the Internet. Out of sheer habit I’ll find myself trying to check email, read friends’ Twitter feeds, or answer the question, “Did Kurtwood Smith have any major roles before he was in the original 1987 RoboCop?”^[2]

Since so much work on a computer involves going on the Internet, it’d be great if your programs could get online. Web scraping is the term for using a program to download and process content from the Web. For example, Google runs many web scraping programs to index web pages for its search engine. In this chapter, you will learn about several modules that make it easy to scrape ...