book

Scripting

Name: Scripting
ISBN: 9781806024872

by Rheinwerk Publishing, Inc, Michael Kofler

June 2025

Beginner to intermediate

473 pages

13h 30m

English

Packt Publishing

Read now

Unlock full access

Notes on Usage
Table of Contents
Preface
Bash, PowerShell, or Python? About This Book
Part I Scripting Languages
1 Scripting: Doing One Thing
1.1 What Does Scripting Mean?1.1.1 Scripting versus Programming1.1.2 Glue Languages and Glue Code1.1.3 Do One Thing and Do It Well
1.2 Scripting Languages
1.2.1 Bash and Zsh1.2.2 PowerShell1.2.3 Python1.2.4 Many Similarities, Even More Differences
1.3 The Agony of Choice
2 Ten Times Ten Lines
2.1 Markdown Spell Checker (Bash)
2.2 Sorting Images by Date (PowerShell)
2.3 Converting a JSON File to XML Format (Python)

2.4 Daily Server Backups (Bash)
2.5 Web Scraping (Python)
2.6 Logging Weather Data (Python)
2.7 Microsoft Hyper-V Cleanup (PowerShell)
2.8 Statistical Analysis of a Logging File (Bash)
2.9 Uploading Files to the Cloud (PowerShell)
2.10 Cloning Virtual Machines (Bash)
3 Bash and Zsh
3.1 Terminal, Shell, and Bash3.1.1 Determining the Current Shell
3.2 Installation
3.2.1 Using Bash in a Virtual Machine3.2.2 Windows Subsystem for Linux3.2.3 Git Bash3.2.4 Docker3.2.5 Bash Configuration (/etc/profile and .bashrc)
3.3 Running Commands Interactively
3.3.1 Operating the Terminal3.3.2 Online Help3.3.3 Command Reference
3.4 Zsh as an Alternative to Bash
3.4.1 Installation3.4.2 Zsh Configuration (/etc/zshrc and .zshrc)3.4.3 Oh My Zsh
3.5 The First Bash Script
3.5.1 Hash Bang (Shebang)3.5.2 Shebang Variants3.5.3 Making the Script Executable (chmod +x)3.5.4 Hello, World!3.5.5 Elementary Syntax Rules3.5.6 A “Real” Example: Backup Script3.5.7 Running Scripts without ./
3.6 Running Commands
3.6.1 Serial Execution3.6.2 Conditional Command Execution3.6.3 Running Commands in the Background3.6.4 Running Commands in a Subshell
3.7 Standard Input and Standard Output
3.7.1 Redirecting Input and Output3.7.2 The Pipe Operator3.7.3 Redirecting and Displaying Outputs at the Same Time
3.8 Globbing, Brace Extension, and Handling File and Directory Names
3.8.1 Recursive Globbing3.8.2 Access to Important Directories3.8.3 Brace Extension3.8.4 Filenames and Directory Names with Spaces
3.9 Variables
3.9.1 Initializing and Deleting Variables3.9.2 Declaring Variables3.9.3 Performing Calculations in Bash3.9.4 Arrays3.9.5 Predefined Variables3.9.6 Environment Variables
3.10 Strings
3.10.1 Single versus Double Quotation Marks3.10.2 Outputting Strings (echo)3.10.3 Colors3.10.4 Entering or Reading Character Strings (read)3.10.5 Substitution and Expansion Mechanisms3.10.6 Parameter Substitution3.10.7 Heredocs and Herestrings3.10.8 The Backslash
3.11 Branches
3.11.1 if Short Notation with && or ||3.11.2 Conditions3.11.3 Branches with case3.11.4 Parameter Analysis via case
3.12 Loops
3.12.1 Processing Filenames with Spaces3.12.2 while and until3.12.3 Loops over Text Files
3.13 Functions
3.13.1 Local Variables
3.14 Error Protection
3.14.1 Detecting Errors3.14.2 Canceling in Case of Errors3.14.3 exit3.14.4 Responding to Signals (trap)3.14.5 Timeout
4 PowerShell
4.1 Installation4.1.1 Installation on Linux4.1.2 Installation on macOS4.1.3 Limitations of PowerShell on Windows versus Linux/macOS4.1.4 Configuration
4.2 Windows Terminal
4.2.1 Configuration4.2.2 Operation
4.3 Calling cmdlets and Functions
4.3.1 cmdlets4.3.2 The Verb-Noun Terminology4.3.3 Aliases4.3.4 Parameters and Options4.3.5 Splatting4.3.6 Functions4.3.7 Running Conventional Commands4.3.8 Online Help
4.4 Combining Commands
4.4.1 Chains of Commands4.4.2 Multiline Statements4.4.3 Dealing with Special Characters
4.5 The First Script
4.5.1 Hello, World!4.5.2 Trouble with the Execution Policy4.5.3 Setting Up a Custom Script Directory4.5.4 Running Scripts on Linux and macOS4.5.5 Example: Cleaning Up the Downloads Directory4.5.6 Differentiating Between PowerShell Versions
4.6 Variables, Strings, and Objects
4.6.1 Data Types4.6.2 Calculating and Comparing4.6.3 Strings4.6.4 Command Substitution4.6.5 Objects4.6.6 Assembling Custom Objects4.6.7 Predefined Variables4.6.8 Environment Variables
4.7 Arrays and Hash Tables
4.7.1 Creating and Processing Custom Arrays4.7.2 Hash Tables
4.8 Output Redirection
4.8.1 Streams4.8.2 Output Redirection4.8.3 Duplicating Output (Tee-Object)4.8.4 (Not) Formatting the Output4.8.5 No Input Redirection
4.9 Loops
4.9.1 ForEach-Object4.9.2 for, while, do-while, and do-until
4.10 Branches
4.10.1 Case Distinctions Using switch
4.11 Functions and Parameters
4.11.1 Function Results4.11.2 return4.11.3 Parameters4.11.4 Calling Functions4.11.5 Processing Standard Input4.11.6 Syntax Variants
4.12 Modules
4.12.1 Module Directories4.12.2 Modules for Experts
4.13 Error Protection
4.13.1 Set-StrictMode4.13.2 Aborting Scripts on Errors4.13.3 Separating Error Messages from Other Output4.13.4 Error Protection Using try/catch4.13.5 Terminating the Script Execution Prematurely (Exit)
5 Python
5.1 Installing Python5.1.1 Windows5.1.2 macOS
5.2 Getting to Know Python in a Terminal Window
5.3 Programming Custom Scripts
5.3.1 Shebang
5.4 Elementary Syntax Rules
5.4.1 Block Elements5.4.2 Comments
5.5 Numbers
5.5.1 Floats5.5.2 Random Numbers
5.6 Strings
5.6.1 Raw Strings5.6.2 Processing Strings5.6.3 Slicing5.6.4 print and input5.6.5 Formatting Outputs
5.7 Lists
5.7.1 List Comprehension5.7.2 map and filter
5.8 Tuples, Sets, and Dictionaries
5.8.1 Sets5.8.2 Dictionaries
5.9 Variables
5.9.1 Assignments5.9.2 Data Types5.9.3 Mutable or Immutable?
5.10 Operators
5.10.1 Combining Assignment and Evaluation
5.11 Branches (if)
5.11.1 Conditions5.11.2 Short Notation with if
5.12 Loops (for and while)
5.12.1 break, continue, and else5.12.2 Loops over Number Ranges (range)5.12.3 Loops over the Characters of a String5.12.4 Loops over Lists, Tuples, and Sets5.12.5 Loops over Dictionaries5.12.6 Loops over the Parameters of a Script5.12.7 Globbing on Windows5.12.8 Loops over the Lines of a Text File5.12.9 Loops over All Files in a Directory
5.13 Functions
5.13.1 Global and Local Variables5.13.2 Parameters5.13.3 Optional Parameters5.13.4 Variable Number of Parameters5.13.5 Lambda Functions
5.14 Processing Text Files
5.14.1 Example: Analyzing a CSV File5.14.2 Example: Creating SQL Commands
5.15 Error Protection
5.15.1 try/except
5.16 System Functions
5.16.1 Access to the Program Parameters5.16.2 Access to Standard Input and Standard Output5.16.3 Exiting a Program5.16.4 Calling Linux Commands5.16.5 Processing Results5.16.6 Running Commands through the Shell5.16.7 Error during a Command Call5.16.8 Waiting (sleep)
5.17 Modules
5.17.1 import5.17.2 Distributing Custom Scripts across Multiple Files
5.18 Installing Additional Modules Using pip
5.18.1 Installing pip5.18.2 Applying pip5.18.3 pip Problems with Windows5.18.4 pip Problems with Linux5.18.5 requirements.txt5.18.6 pipenv
Part II Work Techniques and Tools
6 Linux Toolbox
6.1 Directories and Files6.1.1 Listing, Copying, Moving, and Deleting Files
6.2 Finding Files
6.2.1 Text Search Using grep
6.3 Compressing and Archiving Files
6.3.1 Archiving Files Using tar6.3.2 ZIP Files
6.4 Using Root Privileges
6.4.1 sudo Privileges for Selected Accounts Only6.4.2 Issues with Output Redirection6.4.3 Access Rights6.4.4 Managing Processes6.4.5 Terminating Processes6.4.6 Background Processes and System Services6.4.7 Logging Files6.4.8 Determining Free Memory6.4.9 Determining Other System Information
6.5 Software Installation
6.5.1 Updating Software6.5.2 Installing Additional Packages
6.6 Other Commands
7 cmdlets for PowerShell
7.1 Directories and Files7.1.1 Listing, Copying, Moving, and Deleting Files7.1.2 Reading and Writing Text7.1.3 Inputting and Outputting Text
7.2 Finding Files
7.2.1 Searching Text Files7.2.2 Text versus Objects
7.3 Compressing and Archiving Files
7.3.1 Compressing Search Results
7.4 Process Management
7.4.1 Launching Other PowerShell Scripts7.4.2 Running Commands in the Background7.4.3 Managing Services7.4.4 Working on Other Computers
7.5 Registration Database and System Information
7.5.1 Determining System Information7.5.2 Evaluating the Logging System
7.6 Processing cmdlet Results
7.6.1 Select-Object and Sort-Object7.6.2 Where-Object7.6.3 Group-Object7.6.4 ForEach-Item7.6.5 Measure-Object7.6.6 Formatting and Exporting cmdlet Results
7.7 Other cmdlets
7.8 Installing Additional Modules
7.8.1 Listing Modules, Commands, and Package Sources7.8.2 NuGet and winget
7.9 Standard Aliases
8 Analyzing Texts with Filters and Pipes
8.1 grep, sort, cut, and uniq8.1.1 grep8.1.2 wc8.1.3 cut8.1.4 sort8.1.5 head and tail8.1.6 uniq8.1.7 tr8.1.8 awk and sed
8.2 Example: Statistical Data Analysis
8.2.1 Data Extraction via Script
8.3 Example: ping Analysis
8.3.1 ping Call via Script
8.4 Example: Apache Log Analysis
8.4.1 Extracting IP Addresses8.4.2 Identifying Popular Pages
8.5 CSV Files
8.5.1 Processing CSV Files Using Python8.5.2 Processing CSV Files in PowerShell
9 Regular Expressions
9.1 Syntax Rules for Regular Expressions9.1.1 Characters
9.2 Groups and Alternatives
9.2.1 Quantifiers9.2.2 On Greed (Greedy versus Lazy)9.2.3 Alpha and Omega9.2.4 Exercise Pages and Cheat Sheets9.2.5 Example: Recognizing a Date9.2.6 Example: Recognizing IPv4 Addresses9.2.7 Example: Recognizing Email Addresses
9.3 Regular Expressions in Bash (grep, sed)
9.3.1 Bash Comparison Operator for Regular Expressions9.3.2 Filtering Text Using grep9.3.3 The "sed" Stream Editor9.3.4 Searching and Replacing Using sed9.3.5 sed Example: Manipulating Paths of Image Files
9.4 Regular Expressions in PowerShell
9.4.1 The match and replace Operators9.4.2 Select-String9.4.3 Example: Input Validation9.4.4 Example: List of Figures
9.5 Regular Expressions in Python (re Module)
9.5.1 Example: Verifying a MAC Address9.5.2 Example: Anonymizing a Logging File
10 JSON, XML, and INI
10.1 JSON in PowerShell10.1.1 Example: Saving Event Log Entries in JSON Format10.1.2 Example: Analyzing Domain Queries10.1.3 Example: Converting between CSV and JSON
10.2 JSON and Python
10.2.1 Example: Collecting Birthdays10.2.2 Example: Determining Holidays
10.3 JSON in Bash
10.3.1 Viewing JSON Files Interactively Using fx
10.4 XML in PowerShell
10.4.1 XML Data Type10.4.2 Select-Xml10.4.3 ConvertTo-Xml, Export-Clixml, and Import-Clixml
10.5 XML and Python
10.5.1 Example: Creating a Dictionary for Country Codes10.5.2 Example: Analyzing an RSS Feed10.5.3 Example: Extracting MAC Addresses from Virtual Machine Files
10.6 XML in Bash
10.6.1 xmllint10.6.2 XMLStarlet
10.7 INI Files
10.7.1 Python10.7.2 PowerShell10.7.3 Bash
11 Running Scripts Automatically
11.1 cron11.1.1 /etc/crontab11.1.2 Personal crontab Files11.1.3 Hourly, Daily, Weekly, and Monthly Directories11.1.4 Alternatives to cron11.1.5 Starting Jobs Automatically on macOS
11.2 Example: Web Server Monitoring
11.2.1 Testing and Troubleshooting11.2.2 Real Monitoring
11.3 Microsoft Windows Task Scheduler
11.3.1 Troubleshooting11.3.2 Setting Up Tasks via cmdlets
11.4 Example: Saving Exchange Rates
11.5 Tracking File System Changes
11.5.1 inotify11.5.2 Alternatives to inotify
12 SSH
12.1 Installing the SSH Client and Server12.1.1 Linux12.1.2 macOS12.1.3 Windows12.1.4 Using an Editor in an SSH Session12.1.5 Securing an SSH Server
12.2 Working with SSH
12.2.1 Host Verification12.2.2 Potential Issues and Their Cause12.2.3 Running Linux and macOS Commands12.2.4 Running Windows Commands12.2.5 SSH Remoting in PowerShell
12.3 scp and rsync
12.3.1 Copy-Item with SSH Remoting12.3.2 rsync
12.4 SSH Authentication with Keys
12.4.1 Generating a Key Pair12.4.2 Storing the Public Component of the Key on the Server (macOS and Linux)12.4.3 Storing the Public Component of the Key on the Server (Windows)
12.5 Example: Uploading Images to a Linux Web Server
12.5.1 Preparatory Measures12.5.2 Bash Script12.5.3 PowerShell Script
12.6 Example: Analyzing Virtual Machines
13 Visual Studio Code
13.1 Introduction13.1.1 Comparing VS Code, VSCodium, and Visual Studio13.1.2 Think in Terms of Directories, Not Files!
13.2 Language-Specific VS Code Extensions
13.2.1 PowerShell Extension13.2.2 Python Extension13.2.3 Bash/Shell Extensions
13.3 Remote – SSH Extension
13.3.1 Applying the Remote – SSH Extension13.3.2 Limitations
14 Git
14.1 Git Crash Course14.1.1 Preparation Tasks14.1.2 The First Repository14.1.3 The First Commit14.1.4 Additional Commits14.1.5 Setting Up the Repository on a Second Computer14.1.6 Git Status14.1.7 Excluding Files from Git (.gitignore)14.1.8 Transferring Existing Code to a New Repository
14.2 Handling Settings and Passwords Correctly
14.2.1 What’s a Better Method?14.2.2 Example
14.3 Git Automation
14.3.1 Controlling GitHub Remotely Using gh
14.4 Git Hooks
14.4.1 Example: Detecting Unversioned Files Before the Commit
Part III Applications and Examples
15 Backups
15.1 Synchronizing Directories to External Storage15.1.1 PowerShell Script with robocopy15.1.2 Ideas for Improvement15.1.3 Bash Script with rsync
15.2 WordPress Backup
15.3 SQL Server Backup
15.3.1 Backing Up and Compressing all Databases via Script
16 Image Processing
16.1 Manipulating Image Files16.1.1 Installing ImageMagick16.1.2 Trying Out ImageMagick16.1.3 Example: convert2eps (Bash Variant)16.1.4 Example: convert2eps (PowerShell Variant)
16.2 Sorting Photos by Date Taken
16.2.1 Installing and Trying Out ExifTool16.2.2 Bash Example16.2.3 PowerShell Example
16.3 Converting Exif Metadata to SQL Commands
16.3.1 PyExifTool16.3.2 EXIF2SQL16.3.3 Options for Enhancement
17 Web Scraping
17.1 Limitations
17.2 Web Scraping, Web Crawling, and Data Mining
17.3 Downloading Websites Using wget
17.3.1 Example 1: Downloading Directly Linked Images17.3.2 Example 2: Downloading all PDF Files Linked on the Website17.3.3 Example 3: Creating a Local Copy of the Website
17.4 Web Scraping with Regular Expressions
17.5 Web Scraping with Beautiful Soup
17.5.1 Beautiful Soup and Requests17.5.2 Hello, Beautiful Soup!17.5.3 Example: Determining a List of Top Titles from Rheinwerk Publishing
17.6 Web Scraping with Requests-HTML
17.6.1 Example: Analyzing the GitHub Status
17.7 Web Scraping with PowerShell
17.7.1 PowerHTML Module
18 Using REST APIs
18.1 Tools
18.2 Sample APIs to Try Out
18.3 Implementing Custom REST APIs
18.4 curl and wget
18.4.1 curl18.4.2 wget18.4.3 Using REST APIs in Bash Scripts
18.5 Using REST APIs in PowerShell
18.5.1 Options
18.6 Example: Determining the Current Weather
18.7 Using REST APIs in Python
18.8 Example: Determining Electricity Prices and Displaying Them Graphically
18.8.1 aWATTar API18.8.2 Analysis of the Data18.8.3 Matplotlib18.8.4 Controlling the Energy Consumption
19 Databases
19.1 Updating and Maintaining Databases19.1.1 PowerShell and sqlcmd
19.2 Creating a New Customer Account
19.2.1 Account Data19.2.2 Structure of the Script
19.3 Storing Exif Metadata in a Database
19.3.1 PyMySQL19.3.2 Saving Exif Metadata
19.4 Importing JSON Data into a Table
20 Scripting in the Cloud
20.1 AWS CLI20.1.1 Installation on Linux and macOS20.1.2 Configuration20.1.3 Getting Started20.1.4 Encrypting Files
20.2 Example: Uploading Encrypted Backup Files to the Cloud
20.3 AWS PowerShell Module
20.3.1 Getting Started20.3.2 Copying Files
20.4 Example: Offloading Large Files from a Website to the Cloud
20.4.1 Preparations20.4.2 Script20.4.3 Limitations
21 Virtual Machines
21.1 Setting Up and Running Virtual Machines (KVMs)21.1.1 Cloning Virtual Machines21.1.2 Starting and Shutting Down Virtual Machines21.1.3 Running Scripts on Multiple Virtual Machines
21.2 Automating the Network Configuration (KVMs)
21.2.1 Starting Point
21.3 Controlling Hyper-V
21.3.1 Cloning Virtual Machines
The Author
Index
Service Pages
Legal Notes

Content preview from Scripting

17.2 Web Scraping, Web Crawling, and Data Mining

Two other terms are often used along with web scraping that have similar meanings. While web scraping extracts data from a website, web crawling tracks the links of a website. Through web crawling, you can create a directory of all pages on a website and, when tracking links to other pages, even a directory of large parts of the internet. Search engines like Google or Bing are based on the concept of web crawling.

Data mining also goes a step further than web scraping: In this context, we are combining and statistically analyzing information from different data sources and over longer periods of time. Many reasons exist for data mining, such as marketing studies or scientific tasks.

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781806024872

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Scripting

by Rheinwerk Publishing, Inc, Michael Kofler

17.2 Web Scraping, Web Crawling, and Data Mining

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

More than 5,000 organizations count on O’Reilly

Julian F.

Addison B.

Amir M.

Mark W.

You might also like

The Book of Batch Scripting

Mastering PowerShell Scripting - Fourth Edition

Mastering PowerShell Scripting - Fifth Edition

Bash Shell Scripting for Pentesters

Publisher Resources