book

The Ultimate Linux Shell Scripting Guide

Name: The Ultimate Linux Shell Scripting Guide
Author: Donald A. Tevault
ISBN: 9781835463574

by Donald A. Tevault

October 2024

Beginner to intermediate

698 pages

16h 7m

English

Packt Publishing

Read now

Unlock full access

Preface
Who this book is forWhat this book coversTo get the most out of this bookGet in touchFree Benefits with Your BookHow to UnlockStay Sharp in Cloud and DevOps – Join 44,000+ Subscribers of CloudPro
Getting Started with the Shell
Free Benefits with Your BookUnderstanding ShellsFinding Help with Shell CommandsUnderstanding Manual PagesUnderstanding Info PagesGetting to Know the Linux Documentation ProjectUsing Your Favorite Search EngineUsing a Text Editor to Create Shell ScriptsText-mode EditorsGUI Text EditorsUnderstanding Compiled versus Interpreted ProgrammingUnderstanding root and sudo PrivilegesSummaryQuestionsFurther ReadingAnswers
Interpreting Commands
Understanding the Structure of a CommandUsing Command OptionsHands-on Lab – Practice With Command OptionsUsing Command ArgumentsExecuting Multiple Commands at OnceRunning Commands InteractivelyUsing Command SequencesChaining Commands with a Semi-ColonConditional Command Execution with Double AmpersandsConditional Command Execution with Double PipesUsing the find UtilityPerforming Multiple Actions with findHands-on Lab – Using find to Perform Other CommandsRunning Commands RecursivelyHands-on Lab – Using Commands with RecursionUnderstanding the Command HistoryEscaping and QuotingEscaping MetacharactersQuotingSummaryQuestionsFurther ReadingAnswersGet This Book’s PDF Version and Exclusive Extras
Understanding Variables and Pipelines
Understanding Environmental VariablesUnderstanding Programming VariablesUnderstanding PipelinesSummaryQuestionsFurther ReadingAnswersJoin the CloudPro Newsletter with 44000+ Subscribers
Understanding Input/Output Redirection
Introduction to Input/Output RedirectionUnderstanding stdoutPreventing File OverwritesUsing the File DescriptorUnderstanding stdinUnderstanding stderrUnderstanding teeHands-on Lab – Pipes, Redirectors, and findSummaryQuestionsFurther ReadingAnswersGet This Book’s PDF Version and Exclusive Extras
Customizing the Environment
Technical RequirementsReviewing the Environmental VariablesUnderstanding Shell SessionsUnderstanding the Configuration Filesbash Global Configuration Files on FedoraUsers’ Configuration Files on Fedorabash Global Configuration Files on DebianUsers’ Configuration Files on DebianSetting the Default Editor on DebianSetting Shell Options from the Command-lineUnderstanding AliasesSummaryQuestionsFurther ReadingAnswers
Text-Stream Filters – Part 1
Technical RequirementsIntroduction to Text-Stream FiltersUsing catUsing tacUsing cutUsing pasteUsing joinUsing sortSummaryQuestionsFurther ReadingAnswers
Text Stream Filters – Part 2
Technical RequirementsUsing expandUsing unexpandUsing nlUsing headUsing tailUsing Head And Tail TogetherUsing odUsing uniqUsing wcUsing fmtUsing splitUsing trUsing xargsUsing prPrinting from the Command-lineSummaryQuestionsFurther ReadingAnswersJoin the CloudPro Newsletter with 44000+ Subscribers
Basic Shell Script Construction
Technical RequirementsUnderstanding Basic Shell Script ConstructionHands-on Lab – Counting Logged-in UsersPerforming TestsUsing the test KeywordEnclosing a test Condition Within Square BracketsUsing an if. . .then ConstructUsing Other Types of TestsUnderstanding SubshellsHands-on Lab – Testing ConditionsUnderstanding Scripting VariablesCreating and Deleting VariablesUnderstanding Variables and Shell LevelsUnderstanding Case SensitivityUnderstanding Read-Only VariablesUnderstanding Array VariablesHands-on Lab – Using ArraysUnderstanding Variable ExpansionSubstituting a Value for an Unset VariableSubstituting a Value for a Set VariableAssigning a Value to a VariableDisplaying an Error MessageUsing Variable OffsetsMatching PatternsUnderstanding Command SubstitutionUnderstanding Decisions and LoopsThe if. .then ConstructThe do. . while constructThe for..in ConstructThe for ConstructUsing breakUsing continueThe until ConstructThe case ConstructUsing Positional ParametersUnderstanding Exit CodesStandard Shell Exit CodesUser-defined Exit CodesMore Information About echoLooking at Some Real-World ExamplesHands-on Lab: Using if..thenHands-on Lab – Parsing an Apache Access LogHands-on Lab – Beta Testing a Hard DriveSummaryQuestionsFurther ReadingAnswers
Filtering Text with grep, sed, and Regular Expressions
Technical RequirementsUnderstanding Regular ExpressionsLiterals and MetacharactersUnderstanding sedUnderstanding sed Portability IssuesInstalling gsed on FreeBSDInstalling gsed on macOSInstalling gsed on OpenIndianaSubstitution with sedExample 1: Modifying an Office MemoExample 2: Modifying a List of Hollywood ActorsExample 3: Modifying Lists of CarsExample 4: Performing a Whole-Word SubstitutionDeletion with sedExample 1: Deleting Items from a ListExample 2: Deleting Blank LinesAppending and Inserting with sedExample 1: Appending Lines of TextExample 2: Performing Multiple Operations at OnceExample 3: Inserting Lines of TextChanging with sedExample 1: Changing Edsel to StudebakerExample 2: Changing Entire Lines of TextOther Miscellaneous sed tricksExample 1: Using the q CommandExample 2: Using the w CommandExample 3: Using the r CommandUsing sed program filesExample 1: Appending Lines in a Text FileExample 2: Changing Lines in a Text FileExample 3: Substituting TextExample 4: Copying Lines from One File to AnotherCompound Scripts in sed Program FilesUsing sed in Shell ScriptsUnderstanding grepBasic Searches with grepMore Advanced Searches with grepExample 1: Searching for Whole WordsEven More Advanced Searches with grepExample 1: Auditing Source Code FilesExample 2: Searching for Social Security NumbersExample 3: Using the ^ MetacharacterUsing Extended Regular Expressions with grepExample 1: Basic Search with Extended SyntaxExample 2: Searching for Consecutive Duplicate WordsExample 3: Searching for Words that Begin with a Certain LetterExample 4: Searching for Words with DigitsUsing Fixed-strings Regular Expressions with grepUsing RegEx Helper ProgramsRegexBuddy and RegexMagicRegex101Looking at Some Real-World ExamplesModifying Multiple Files at OnceSearching Through Apache Webserver Logs for Cross-site Scripting AttacksAutomating Third-party Repository InstallationsFilling Empty Fields in a .csv FileSummaryQuestionsFurther ReadingAnswersGet This Book’s PDF Version and Exclusive Extras

Understanding Functions
Technical RequirementsIntroduction to FunctionsDefining a FunctionUsing Functions in Shell ScriptsCreating and Calling FunctionsPassing Positional Parameters to FunctionsPassing Values from a FunctionCreating Function LibrariesLooking at Some Real-World ExamplesChecking Network ConnectivityUsing the CoinGecko APIHands-on Lab – Creating the coingecko.sh ScriptSummaryQuestionsFurther ReadingAnswers
Performing Mathematical Operations
Technical RequirementsPerforming Integer Math with ExpressionsUsing the expr CommandUsing echo with Math ExpressionsPerforming Integer Math with Integer VariablesPerforming Floating Point Math with bcUsing bc in Interactive ModeUsing bc Program FilesUsing bc in Shell ScriptsSummaryQuestionsFurther ReadingAnswersJoin the CloudPro Newsletter with 44000+ Subscribers
Automating Scripts with here Documents and expect
Technical RequirementsUsing here DocumentsCreating here Documents with Static DataCreating here documents with Dynamic DataUsing Functions in here DocumentsAutomating Responses with expectSecurity Implications with expectSummaryQuestionsFurther ReadingAnswersGet This Book’s PDF Version and Exclusive Extras
Scripting with ImageMagick
Technical RequirementsConverting Non-standard Filename ExtensionsInstalling ImageMagickDisplaying ImagesViewing Image PropertiesResizing and Customizing ImagesBatch-processing Image FilesUsing Fred’s ImageMagick ScriptsSummaryQuestionsFurther ReadingAnswers
Using awk – Part 1
Introducing awkUnderstanding Patterns and ActionsObtaining Input from Text FilesLooking for Human UsersParsing Webserver Access LogsUsing Regular ExpressionsObtaining Input from CommandsSummaryQuestionsFurther ReadingAnswers
Using awk – Part 2
Technical RequirementsBasic awk Script ConstructionUsing Conditional StatementsUsing a while Construct and Setting VariablesSumming Numbers in a LineFinding the CPU GenerationUsing for loops and ArraysUsing Floating Point Math and printfWorking with Multi-Line RecordsSummaryQuestionsFurther ReadingAnswersJoin the CloudPro Newsletter with 44000+ Subscribers
Creating User Interfaces with yad, dialog, and xdialog
Technical RequirementsCreating a Graphical User Interface with yadThe yad BasicsCreating Data Entry FormsCreating a Drop-down ListUsing the yad File ManagerCreating a File Checksum UtilityCreating a GUI Front-end for ImageMagickProgramming Form ButtonsSome Final Thoughts about yadCreating User Interfaces with dialog and xdialogThe dialog BasicsThe xdialog BasicsAutomatically Choosing Either dialog or xdialogAdding WidgetsCreating an SSH Login InterfaceSummaryQuestionsFurther ReadingAnswersGet This Book’s PDF Version and Exclusive Extras
Using Shell Script Options with getops
Technical RequirementsUnderstanding the Need for getoptsUnderstanding getopt versus getoptsUsing getoptsLooking at Real-world ExamplesThe Modified Coingecko ScriptThe Tecmint Monitor ScriptSummaryQuestionsFurther ReadingAnswers
Shell Scripting for Security Professionals
Technical RequirementsSimple Scripts for AuditingIdentifying an Operating SystemA Simple Port-scanning ScriptAuditing the root User AccountCreating the root Account Auditing Script for Linux and OpenIndianaModifying the root Account Auditing Script for Use on FreeBSDCreating a User Activity Monitoring ScriptCreating Simple Firewall ScriptsCreating an IP Address Blocking Script for Red Hat DistrosHands-on Lab: Create the Script with an Array and a for loopHands-on Lab: Creating the Script with xargsSearching for Existing Security-related ScriptsSummaryQuestionsFurther ReadingAnswersGet This Book’s PDF Version and Exclusive Extras
Shell Script Portability
Technical RequirementsRunning bash on Non-Linux SystemsUsing env to Set the bash EnvironmentCreating a Symbolic Link to bashUnderstanding POSIX complianceUnderstanding the Differences Between ShellsUnderstanding BashismsUsing Portable TestsMaking Portable ArraysUnderstanding Portability Problems with echoTesting Scripts for POSIX ComplianceCreating Scripts on a POSIX-compliant ShellUsing checkbashismsUsing shellcheckSpecifying a Shell with the -s OptionHands-on Lab – Using -s to Scan Function LibrariesUsing shallSummaryQuestionsFurther ReadingAnswersJoin the CloudPro Newsletter with 44000+ Subscribers
Shell Script Security
Technical RequirementsControlling Access to Your ScriptsAssigning sudo PrivilegesHands-on Lab – Configuring sudoUsing an Access Control ListHands-on Lab – Setting an ACL for Horatio on LinuxHands-on Lab – Setting an ACL for Horatio on FreeBSD 14Hands-on Lab – Setting an ACL for Horatio on OpenIndianaObfuscating Plain-Text ScriptsInstalling shcHands-on Lab – Using shcHands-on Lab – Creating Untraceable ExecutablesDecrypting shc BinariesUnderstanding SUID and SGID ConsiderationsAvoiding Sensitive Data LeakageSecuring Temporary FilesUnderstanding the /tmp/ DirectoryThe Wrong Way to Create Temporary FilesThe Right Way to Create Temporary FilesUsing Passwords in Shell ScriptsHands-on Lab – Encrypting PasswordsUnderstanding Command Injection with evalUsing eval on the Command-lineUsing eval SafelyUsing eval DangerouslyUsing Alternatives to evalUsing Command SubstitutionEvaluating if eval is NecessaryUnderstanding Path SecurityAttack Scenario 1: Compromising the User’s AccountAttack Scenario 2: Social EngineeringSummaryQuestionsFurther ReadingAnswersGet This Book’s PDF Version and Exclusive Extras
Debugging Shell Scripts
Technical RequirementsUnderstanding Common Scripting ErrorsNot Enough QuotingFilenames with Blank SpacesProblems with Unset VariablesCreating a Wild LoopUsing Shell Script Debugging Tools and TechniquesUsing echo StatementsUsing xtrace for DebuggingChecking for Undefined VariablesChecking for Errors with the -e OptionUnderstanding the Problems with set -e and -eUsing bash DebuggerInstalling bashdb on LinuxInstalling bashdb on FreeBSDInstalling on macOSDebugging a Script with bashdbGetting Help with bashdbSummaryQuestionsFurther ReadingAnswers
Introduction to Z Shell Scripting
Technical RequirementsIntroducing zshInstalling zshUnderstanding the Unique Features of zsh ScriptingDifferences in Variable ExpansionSubstituting ValuesSubstituting SubstringsTranslating Between Upper and Lower CaseExtended File GlobbingUnderstanding zsh ArraysEnhanced Math CapabilitiesUsing zsh ModulesUsing the mathfunc ModuleThe datetime ModuleSummaryQuestionsFurther ReadingAnswers
Using PowerShell on Linux
Technical RequirementsInstalling PowerShell on Linux and macOSInstalling PowerShell on Linux via a snap PackageInstalling PowerShell on FedoraInstalling PowerShell on macOSInvoking PowerShellReasons for Linux and Mac Admins to Learn PowerShellWorking with Mixed Operating System EnvironmentsPowerShell Commands Can Be SimplerEnhanced Builtin Math CapabilitiesDifferences Between PowerShell Scripting and Traditional Linux/Unix ScriptingUsing Filename Extensions and the Executable PermissionPowerShell is Object-orientedPowerShell Uses CmdletsUsing Aliases on PowerShellViewing the Available PowerShell CommandsGetting Help with PowerShell CommandsReal-World Cross-Platform PowerShell ScriptsThe write-marquee.ps1 ScriptThe check-cpu.ps1 ScriptSummaryFurther Reading
Unlock Your Exclusive Benefits
Unlock this Book’s Free Benefits in 3 Easy StepsStep 1Step 2Step 3Need help?
Other Books You May Enjoy
Stay Sharp in Cloud and DevOps – Join 44,000+ Subscribers of CloudPro
Index

Content preview from The Ultimate Linux Shell Scripting Guide

Index

Symbols

/tmp/ Directory 543, 544

Access Control List (ACL) 521

setting, for Horatio on FreeBSD 14 524-526

setting, for Horatio on Linux 521-524

setting, for Horatio on OpenIndiana 526-528

using 521

action 371, 372

aliases 81-86, 633-636

American Telephone and Telegraph (AT&T) 490

apache access log

parsing 226-228

Apache webserver logs

searching, for cross-site scripting 281-283

Application Programming Interface (API) 303

arrays

using 191-194, 407, 408

array variables 190, 191

associative array 379

awk command 369, 370

busybox 370

gawk 370

mawk 370

nawk 370

awk programming variables

using 401

awk script construction 397, 398

bash 1

env command, using 488, 489

running, on Non-Linux Systems 488

Symbolic Link, creating 489, 490

bashdb ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Learn Linux Shell Scripting - Fundamentals of Bash 4.4

Publisher Resources

ISBN: 9781835463574

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

The Ultimate Linux Shell Scripting Guide

by Donald A. Tevault

Index

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.