book

Linux Shell Scripting Cookbook, Second Edition - Second Edition

Name: Linux Shell Scripting Cookbook, Second Edition - Second Edition
ISBN: 9781782162742

by Shantanu Tushar, Sarath Lakshman

May 2013

Beginner to intermediate

384 pages

7h 40m

English

Packt Publishing

Read now

Unlock full access

Linux Shell Scripting Cookbook - Second Edition
Table of Contents
Linux Shell Scripting Cookbook - Second Edition
Credits
About the Authors
About the Reviewers
www.PacktPub.com
Support files, eBooks, discount offers and moreWhy Subscribe?Free Access for Packt account holders
Preface
What this book covers
What you need for this book
Who this book is for

Conventions
Reader feedback
Customer support
Downloading the example codeErrataPiracyQuestions
1. Shell Something Out
Introduction
Printing in the terminal
How to do it...How it works...There's more...Escaping newline in echoPrinting a colored output
Playing with variables and environment variables
Getting readyHow to do it...There's more...Finding the length of a stringIdentifying the current shellChecking for super userModifying the Bash prompt string (username@hostname:~$)
Function to prepend to environment variables
How to do it...How it works...
Math with the shell
Getting readyHow to do it...
Playing with file descriptors and redirection
Getting readyHow to do it...How it works...There's more...Redirection from a file to a commandRedirecting from a text block enclosed within a scriptCustom file descriptors
Arrays and associative arrays
Getting readyHow to do it...There's more...Defining associative arraysListing of array indexes
Visiting aliases
How to do it...There's more...Escaping aliases
Grabbing information about the terminal
Getting readyHow to do it...
Getting and setting dates and delays
Getting readyHow to do it...How it works...There's more...Producing delays in a script
Debugging the script
How to do it...How it works...There's more...Shebang hack
Functions and arguments
How to do it...There's more...The recursive functionExporting functionsReading the return value (status) of a commandPassing arguments to commands
Reading the output of a sequence of commands in a variable
Getting readyHow to do it...There's more...Spawning a separate process with subshellSubshell quoting to preserve spacing and the newline character
Reading n characters without pressing the return key
How to do it...
Running a command until it succeeds
How to do it...How it works...There's more...A faster approachAdding a delay
Field separators and iterators
Getting readyHow to do it...
Comparisons and tests
How to do it...
2. Have a Good Command
Introduction
Concatenating with cat
How to do it...How it works…There's more...Getting rid of extra blank linesDisplaying tabs as ^ILine numbers
Recording and playing back of terminal sessions
Getting readyHow to do it...How it works...
Finding files and file listing
Getting readyHow to do it...There's more...Search based on filename or regular expression matchNegating argumentsSearch based on the directory depthSearch based on file typeSearch on file timesSearch based on file sizeDeleting based on the file matchesMatch based on the file permissions and ownershipExecuting commands or actions with findSkipping specified directories when using the find command
Playing with xargs
Getting readyHow to do it...How it works…There's more...Passing formatted arguments to a command by reading stdinUsing xargs with findCounting the number of lines of C code in a source code directoryWhile and subshell trick with stdin
Translating with tr
Getting readyHow to do it...How it works…There's more...Deleting characters using trComplementing character setSqueezing characters with trCharacter classes
Checksum and verification
Getting readyHow to do it...How it works...There's more...Checksum for directories
Cryptographic tools and hashes
How to do it...
Sorting unique and duplicates
Getting readyHow to do it...How it works…There's more...Sorting according to the keys or columnsuniq
Temporary file naming and random numbers
How to do it...How it works…
Splitting files and data
How to do it...There's more…Specifying a filename prefix for the split files
Slicing filenames based on extension
How to do it…How it works…
Renaming and moving files in bulk
Getting readyHow to do it...How it works…
Spell checking and dictionary manipulation
How to do it...How it works...
Automating interactive input
Getting readyHow to do it...How it works…There's more...Automating with expect
Making commands quicker by running parallel processes
How to do it...How it works...
3. File In, File Out
Introduction
Generating files of any size
How to do it...
The intersection and set difference (A-B) on text files
Getting readyHow to do it...How it works...
Finding and deleting duplicate files
Getting readyHow to do it...How it works...
Working with file permissions, ownership, and the sticky bit
How to do it...There's more...Changing ownershipSetting sticky bitApplying permissions recursively to filesApplying ownership recursivelyRunning an executable as a different user (setuid)
Making files immutable
Getting readyHow to do it...
Generating blank files in bulk
Getting readyHow to do it...
Finding symbolic links and their targets
How to do it...How it works...
Enumerating file type statistics
Getting readyHow to do it...How it works...
Using loopback files
How to do it...How it works...There's more...Creating partitions inside loopback imagesQuicker way to mount loopback disk images with partitionsMounting ISO files as loopbackFlush changing immediately with sync
Creating ISO files and hybrid ISO
Getting readyHow to do it...There's more...Hybrid ISO that boots off a flash drive or hard diskBurning an ISO from the command linePlaying with the CD-ROM tray
Finding the difference between files, patching
How to do it...There's more...Generating difference against directories
Using head and tail for printing the last or first 10 lines
How to do it...
Listing only directories – alternative methods
Getting readyHow to do it...How it works...
Fast command-line navigation using pushd and popd
Getting readyHow to do it...There's more...Most frequently used directory switching
Counting the number of lines, words, and characters in a file
How to do it...
Printing the directory tree
Getting readyHow to do it...There's more...HTML output for tree
4. Texting and Driving
Introduction
Using regular expressions
How to do it...How it works...There's more...Treatment of special charactersVisualizing regular expressions
Searching and mining a text inside a file with grep
How to do it...There's more...Recursively search many filesIgnoring case of patterngrep by matching multiple patternsIncluding and excluding files in a grep searchUsing grep with xargs with zero-byte suffixSilent output for grepPrinting lines before and after text matches
Cutting a file column-wise with cut
How to do it...There's moreSpecifying the range of characters or bytes as fields
Using sed to perform text replacement
How to do it…There's more...Removing blank linesPerforming replacement directly in the fileMatched string notation (&)Substring match notation (\1)Combination of multiple expressionsQuoting
Using awk for advanced text processing
Getting ready...How to do it…How it works…There's more…Special variablesPassing an external variable to awkReading a line explicitly using getlineFiltering lines processed by awk with filter patternsSetting delimiter for fieldsReading the command output from awkUsing loop inside awkString manipulation functions in awk
Finding the frequency of words used in a given file
Getting readyHow to do it...How it works...See also
Compressing or decompressing JavaScript
Getting readyHow to do it...How it works...See also
Merging multiple files as columns
How to do it...See also
Printing the nth word or column in a file or line
How to do it...See also
Printing text between line numbers or patterns
Getting readyHow to do it...See also
Printing lines in the reverse order
Getting readyHow to do it...How it works...
Parsing e-mail addresses and URLs from text
How to do it...How it works...See also
Removing a sentence in a file containing a word
Getting readyHow to do it...How it works...See also
Replacing a pattern with text in all the files in a directory
How to do it...How it works...There's more...
Text slicing and parameter operations
How to do it...See also
5. Tangled Web? Not At All!
Introduction
Downloading from a web page
Getting readyHow to do it...How it works...There's more...Restricting the download speedResume downloading and continueCopying a complete website (mirroring)Accessing pages with HTTP or FTP authentication
Downloading a web page as plain text
How to do it...
A primer on cURL
Getting readyHow to do it…How it works...There's more...Continuing and resuming downloadsSetting the referer string with cURLCookies with cURLSetting a user agent string with cURLSpecifying a bandwidth limit on cURLSpecifying the maximum download sizeAuthenticating with cURLPrinting response headers excluding the dataSee also
Accessing Gmail e-mails from the command line
How to do it...How it works...See also
Parsing data from a website
How to do it...How it works...See also
Image crawler and downloader
How to do it...How it works...See also
Web photo album generator
Getting readyHow to do it...How it works...See also
Twitter command-line client
Getting readyHow to do it...How it works...See also
Creating a "define" utility by using the Web backend
Getting readyHow to do it...How it works...See also
Finding broken links in a website
Getting readyHow to do it...How it works...See also
Tracking changes to a website
Getting readyHow to do it...How it works...See also
Posting to a web page and reading the response
Getting readyHow to do it...How it works...See also
6. The Backup Plan
Introduction
Archiving with tar
Getting readyHow to do it...How it works...There's more...Appending files to an archiveExtracting files and folders from an archivestdin and stdout with tarConcatenating two archivesUpdating files in an archive with a timestamp checkComparing files in the archive and file systemDeleting files from the archiveCompression with the tar archiveExcluding a set of files from archivingExcluding version control directoriesPrinting total bytesSee also
Archiving with cpio
How to do it...How it works...
Compressing data with gzip
How to do it...There's more...Gzip with tarballzcat - reading gzipped files without extractingCompression ratioUsing bzip2Using lzmaSee also
Archiving and compressing with zip
How to do it...How it works...
Faster archiving with pbzip2
Getting readyHow to do it...How it works...There's more...Manually specifying the number of CPUsSpecifying the compression ratio
Creating filesystems with compression
Getting readyHow to do it...There's more...Excluding files while creating a squashfs file
Backup snapshots with rsync
How to do it...How it works...There's more...Excluding files while archiving with rsyncDeleting non-existent files while updating rsync backupScheduling backups at intervals
Version control-based backup with Git
Getting readyHow to do it...
Creating entire disk images using fsarchiver
Getting readyHow to do it...How it works...
7. The Old-boy Network
Introduction
Setting up the network
Getting readyHow to do it...There's more...Printing the list of network interfacesDisplaying IP addressesSpoofing the hardware address (MAC address)Name server and DNS (Domain Name Service)DNS lookupShowing routing table informationSee also
Let us ping!
How to do it...There's moreRound trip timeLimiting the number of packets to be sentReturn status of the ping commandTraceroute
Listing all the machines alive on a network
Getting readyHow to do it...How it works...There's more...Parallel pingsUsing fpingSee also
Running commands on a remote host with SSH
Getting readyHow to do it...There's more...SSH with compressionRedirecting data into stdin of remote host shell commandsRunning graphical commands on a remote machineSee also
Transferring files through the network
Getting readyHow to do it...There's more...Automated FTP transferSFTP (Secure FTP)The rsync commandSCP (secure copy program)Recursive copying with SCPSee also
Connecting to a wireless network
Getting readyHow to do it...How it works...See also
Password-less auto-login with SSH
Getting readyHow to do it...
Port forwarding using SSH
How to do it...There's more...Non-interactive port forwardReverse port forwarding
Mounting a remote drive at a local mount point
Getting readyHow to do it...See also
Network traffic and port analysis
Getting readyHow to do it...How it works...There's more...Opened port and services using netstat
Creating arbitrary sockets
Getting readyHow to do it...There's more...Quickly copying files over the network
Sharing an Internet connection
Getting readyHow to do it...
Basic firewall using iptables
How to do it...How it works...There's more...
8. Put on the Monitor's Cap
Introduction
Monitoring disk usage
Getting readyHow to do it...There's more...Displaying disk usage in KB, MB, or BlocksDisplaying the grand total sum of disk usagePrinting files in specified unitsExcluding files from the disk usage calculationFinding the 10 largest size files from a given directoryDisk free information
Calculating the execution time for a command
How to do it...How it works...
Collecting information about logged in users, boot logs, and boot failures
Getting readyHow to do it...
Listing the top 10 CPU consuming processes in an hour
Getting readyHow to do it...How it works...See also
Monitoring command outputs with watch
How to do it...There's moreHighlighting the differences in the watch output
Logging access to files and directories
Getting readyHow to do it...How it works...
Logfile management with logrotate
Getting readyHow to do it...How it works...
Logging with syslog
Getting readyHow to do it...See also
Monitoring user logins to find intruders
Getting readyHow to do it…How it works…
Remote disk usage health monitor
Getting readyHow to do it…How it works…See also
Finding out active user hours on a system
Getting readyHow to do it…How it works…
Measuring and optimizing power usage
Getting readyHow to do it...
Monitoring disk activity
Getting readyHow to do it...
Checking disks and filesystems for errors
Getting readyHow to do it...How it works...
9. Administration Calls
Introduction
Gathering information about processes
Getting readyHow to do it...How it works...There's more...topSorting the ps output with respect to a parameterFinding the process ID when given command namesFilters with ps for real user or ID, effective user or IDTTY filter for psInformation about process threadsSpecifying output width and columns to be displayedShowing environment variables for a processAbout which, whereis, file, whatis, and load averageSee also
Killing processes and send or respond to signals
Getting readyHow to do it...There's more...The kill family of commandsCapturing and responding to signals
Sending messages to user terminals
Getting readyHow to do it...How it works...
Gathering system information
How to do it...
Using /proc for gathering information
How to do it...
Scheduling with cron
Getting readyHow to do it…How it works...There's more…Specifying environment variablesRunning commands at system start up/bootViewing the cron tableRemoving the cron table
Writing and reading the MySQL database from Bash
Getting readyHow to do it…How it works…
User administration script
How to do it…How it works…
Bulk image resizing and format conversion
Getting readyHow to do it..How it works…See also
Taking screenshots from the terminal
Getting readyHow to do it...
Managing multiple terminals from one
Getting readyHow to do it...
Index

Content preview from Linux Shell Scripting Cookbook, Second Edition - Second Edition

Checking disks and filesystems for errors

Data is the most important thing in any computer system. Naturally, it is important to monitor the consistency of data stored on physical media.

Getting ready

We will use the standard tool, fsck to check for errors in the filesystems. This command should be preinstalled on all modern distros. If not, use your package manager to install it.

How to do it...

Let us see how to use fsck with its various options to check filesystems for errors, and optionally fix them.

To check for errors on a partition or filesystem, just pass its path to fsck:

# fsck /dev/sdb3
fsck from util-linux 2.20.1
e2fsck 1.42.5 (29-Jul-2012)
HDD2 has been mounted 26 times without being checked, check forced.
Pass 1: Checking inodes, blocks, ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Learning Linux Shell Scripting - Second Edition

Publisher Resources

ISBN: 9781782162742Other

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills