book

Shell Scripting Recipes: A Problem-Solution Approach

Name: Shell Scripting Recipes: A Problem-Solution Approach
Author: Michael Young
ISBN: 9781590594711

by Michael Young

May 2005

Intermediate to advanced

446 pages

8h 9m

English

Apress

Read now

Unlock full access

Copyright
About the Author
About the Technical Reviewer
Acknowledgments
Introduction
Why Shell Scripts?Which Shells Will Run These Scripts?Looking Through the BookThinking Outside the Book
1. The POSIX Shell and Command-Line Utilities
1.1. Shell Commands1.1.1. echo1.1.2. printf1.1.3. set1.1.4. shift1.1.5. type1.1.6. getopts1.1.7. case1.1.8. eval1.1.9. local1.2. Parameters and Variables1.2.1. Positional Parameters1.2.2. Special Parameters1.2.3. standard-vars—A Collection of Useful Variables1.3. Patterns1.3.1. Pathname Expansion1.3.2. Regular Expressions1.4. Parameter Expansion1.4.1. The Bourne Shell Expansions1.4.2. POSIX Parameter Expansions1.4.3. Shell-Specific Expansions, bash2, and ksh931.5. Shell Arithmetic1.6. Aliases1.7. Sourcing a File1.8. Functions1.8.1. Functions Are Fast1.8.2. Command Substitution Is Slow1.8.3. Using the Functions in This Book1.9. standard-funcs: A Collection of Useful Commands1.9.1. get_key—Get a Single Keystroke from the User1.9.2. getline—Prompt User to Enter a Line1.9.3. press_any_key—Prompt for a Single Keypress1.9.4. menu1—Print a Menu and Execute a Selected Command1.9.5. arg—Prompt for Required Argument If None Supplied1.9.6. die—Print Error Message and Exit with Error Status1.9.7. show_date—Display Date in D[D] MMM YYYY Format1.9.8. date_vars—Set Date and Time Variables1.9.9. is_num—Is This a Positive Integer?1.9.10. abbrev_num—Abbreviate Large Numbers1.9.11. commas—Add Thousands Separators to a Number1.9.12. pr1—Print Arguments, One to a Line1.9.13. checkdirs—Check for Directories; Create If Necessary1.9.14. checkfiles—Check That a Directory Contains Certain Files1.9.15. zpad—Pad a Number with Leading Zeroes1.9.16. cleanup—Remove Temporary Files and Reset Terminal on Exit1.10. The Unix Utilities1.10.1. cat: Concatenate Files to the Standard Output1.10.2. sed: A Text Stream Editor1.10.3. awk: Pattern Scanning and Processing Language1.10.4. grep: Print Lines Matching a Regular Expression1.10.5. date: Show or Set the System Date1.10.6. tr: A Character Translation Utility1.10.7. wc: Count Characters, Words, and Lines in a File1.10.8. file: Determine the File Type1.10.9. ls: Sort and Provide Details About Files1.10.10. uniq: Remove Consecutive Duplicate Lines1.10.11. sudo: Execute Commands as the Superuser1.10.12. split: Divide a File into Equal-Sized Pieces1.10.13. which: Show the Full Path to a Command1.10.14. gs, gv: Render, Convert, or View PostScript and PDF Files1.11. Summary
2. Playing with Files: Viewing, Manipulating, and Editing Text Files
2.1. End of the Line for OS Differences: Converting Text Files2.1.1. dos2unix—Convert Windows Text Files to Unix2.1.2. unix2dos—Convert a Unix File to Windows2.1.3. mac2unix—Convert Macintosh Files to Unix2.1.4. unix2mac—Convert Unix Files to Mac Format2.1.5. dos2mac—Convert Windows Files to Macintosh2.1.6. mac2dos—Convert Macintosh Files to Windows2.2. Displaying Files2.2.1. prn—Print File with Line Numbers2.2.2. prw—Print One Word per Line2.2.3. wbl—Sort Words by Length2.3. Formatting File Facts2.3.1. finfo—File Information Display2.3.2. wfreq—Word Frequency2.3.3. lfreq—Letter Frequency2.3.4. fed—A Simple Batch File Editor2.4. Summary
3. String Briefs
3.1. Character Actions: The char-funcs Library3.1.1. chr—Convert a Decimal Number to an ASCII Character3.1.2. asc—Convert a Character to Its Decimal Equivalent3.1.3. nxt—Get the Next Character in ASCII Sequence3.1.4. upr—Convert Character(s) to Uppercase3.1.5. lwr—Convert Character(s) to Lowercase3.2. String Cleaning: The string-funcs Library3.2.1. sub—Replace First Occurrence of a Pattern3.2.2. gsub—Globally Replace a Pattern in a String3.2.3. repeat—Build a String of a Specified Length3.2.4. index, rindex—Find Position of One String Within Another3.2.5. substr—Extract a Portion of a String3.2.6. insert_str—Place One String Inside Another3.3. Summary
4. What's in a Word?
4.1. Finding and Massaging Word Lists4.2. wf-funcs: WordFinder Function Library4.2.1. write_config—Write User's Information to the Configuration File4.2.2. do_config—Check For and Source Default Configuration File4.2.3. set_sysdict—Select the Dictionary Directory4.2.4. mkwsig—Sort Letters in a Word4.2.5. wf-clean—Remove Carriage Returns and Accents4.2.6. wf-compounds—Squish Compound Words and Save with Lengths4.2.7. wf-setup—Prepare Word and Anagram Lists4.3. Playing with Matches4.3.1. wf—Find Words That Match a Pattern4.3.2. wfb—Find Words That Begin with a Given Pattern4.3.3. wfe—Find Words That End with a Given Pattern4.3.4. wfc—Find Words That Contain a Given Pattern4.3.5. wfit—Find Words That Fit Together in a Grid4.3.6. anagram—Find Words That Are Anagrams of a Given Word4.3.7. aplus—Find Anagrams of a Word with a Letter Added4.3.8. aminus—Remove Each Letter in Turn and Anagram What's Left4.4. Summary
5. Scripting by Numbers
5.1. The math-funcs Library5.1.1. fpmul—Multiply Decimal Fractions5.1.2. int—Return the Integer Portion of a Decimal Fraction5.1.3. round—Round the Argument to the Nearest Integer5.1.4. pow—Raise a Number to Any Given Power5.1.5. square—Raise a Number to the Second Power5.1.6. cube—Raise a Number to the Third Power5.1.7. calc—A Simple Command-Line Calculator5.2. Adding and Averaging5.2.1. total—Add a List of Numbers5.2.2. mean—Find the Arithmetic Mean of a List of Numbers5.2.3. median—Find the Median of a List of Numbers5.2.4. mode—Find the Number That Appears Most in a List5.2.5. range—Find the Range of a Set of Numbers5.2.6. stdev—Finding the Standard Deviation5.3. Converting Between Unit Systems5.3.1. conversion-funcs—Converting Metric Units5.3.2. conversion—A Menu System for Metric Conversion5.4. Summary

6. Loose Names Sink Scripts: Bringing Sanity to Filenames
6.1. What's in a Name?6.1.1. POSIX Portable Filenames6.1.2. OK Filenames6.2. Functioning Filenames: The filename-funcs Library6.2.1. basename—Extract the Last Element of a Pathname6.2.2. dirname—Return All but the Last Element of a Pathname6.2.3. is_pfname—Check for POSIX Portable Filename6.2.4. is_OKfname—Check Whether a Filename Is Acceptable6.2.5. pfname—Convert Nonportable Characters in Filename6.2.6. OKfname—Make a Filename Acceptable6.2.7. is_whitespc—Does the Filename Contain Whitespace Characters?6.2.8. whitespc—Fix Filenames Containing Whitespace Characters6.2.9. is_dir—Is This a Directory I See Before Me?6.2.10. nodoublechar—Remove Duplicate Characters from a String6.2.11. new_filename—Change Filename to Desired Character Set6.2.12. fix_pwd—Fix All the Filenames in the Current Directory6.2.13. fixfname—Convert Filenames to Sensible Names6.3. Summary
7. Treading a Righteous PATH
7.1. The path-funcs Library7.1.1. path—Display a User's PATH7.1.2. unslash—Remove Extraneous Slashes7.1.3. checkpath—Clean Up the PATH Variable7.1.4. addpath—Add Directories to the PATH Variable7.1.5. rmpath—Remove One or More Directories from $PATH7.2. Summary
8. The Dating Game
8.1. The date-funcs Library8.1.1. split_date—Divide a Date into Day, Month, and Year8.1.2. is_leap_year—Is There an Extra Day This Year?8.1.3. days_in_month—How Many Days Hath September?8.1.4. date2julian—Calculate the Julian Day Number8.1.5. julian2date—Convert Julian Back to Year, Month, and Day8.1.6. dateshift—Add or Subtract a Number of Days8.1.7. diffdate—Find the Number of Days Between Two Dates8.1.8. day_of_week—Find the Day of the Week for Any Date8.1.9. display_date—Show a Date in Text Format8.1.10. parse_date—Decipher Various Forms of Date String8.1.11. valid_date—Where Was I on November 31st?8.2. Summary
9. Good Housekeeping: Monitoring and Tidying Up File Systems
9.1.9.1.1. dfcmp—Notify User of Major Changes in Disk Usage9.1.2. symfix—Remove Broken Symbolic Links9.1.3. sym2file—Converts Symbolic Links to Regular Files9.1.4. zrm—Remove Empty Files9.1.5. undup—Remove Duplicate Files9.1.6. lsr—List the Most Recent (or Oldest) Files in a Directory9.2. Summary
10. POP Goes the E-Mail
10.1.10.1.1. popcheck—Display Number of Messages in POP3 Mailbox10.1.2. pop3-funcs—Functions for Managing a POP3 Mailbox10.1.3. pop3list—Examine a POP3 Mailbox10.1.4. pop3filter—Delete and Retrieve Messages from a POP3 Server10.1.5. pflog—Print a Summary of a Day's Activity10.1.6. viewlog—Continuously View the pop3filter Log File10.2. Summary
11. PostScript: More Than an Afterthought
11.1.11.1.1. ps-labels—Producing Mailing Labels from a List11.1.2. ps-envelopes—Print Envelopes from List11.1.3. ps-grid—Convert a Crossword Grid to PostScript11.2. Summary
12. Screenplay: The screen—funcs Library
12.1.12.1.1. screen-vars—Variables for Screen Manipulation12.1.2. set_attr—Set Screen-Printing Attributes12.1.3. set_fg, set_bg, set_fgbg-—Set Colors forPrinting to the Screen12.1.4. cls—Clear the Screen12.1.5. printat—Position Cursor by Row and Column12.1.6. put_block_at—Print Lines in a Column Anywhere on the Screen12.1.7. get_size—Set LINES and COLUMNS Variables12.1.8. max_length—Find the Length of the Longest Argument12.1.9. print_block_atxy—Print a Block of Lines Anywhere on the Screen12.1.10. vbar, hbar—Print a Vertical or Horizontal Bar12.1.11. center—Center a String on N Columns12.1.12. flush_right—Align String with the Right Margin12.1.13. ruler—Draw a Ruler Across the Width and Height of the Window12.1.14. box_block, box_block_at—Print Text Surrounded by a Box12.1.15. clear_area, clear_area_at—Clear an Area of the Screen12.1.16. box_area, box_area_at—Draw a Box Around an Area12.1.17. screen-demo—Saving and Redisplaying Areas of the Screen12.2. Summary
13. Backing Up the Drive
13.1.13.1.1. bu—Backup Directories13.1.2. bin-pack—Fill Fixed-Capacity Directories with Files13.1.3. unpack—Extract All Files from One or More Archives13.2. Summary
14. Aging, Archiving, and Deleting Files
14.1.14.1.1. date-file—Add a Datestamp to a Filename14.1.2. rmold—Remove Old Files14.1.3. keepnewest—Remove All but the Newest or Oldest Files14.2. Summary
15. Covering All Your Databases
15.1.15.1.1. lookup—Find the Corresponding Value for a Key15.2. shdb-funcs: Shell Database Function Library15.2.1. load_db—Import Database into Shell Array15.2.2. split_record—Split a Record into Fields15.2.3. csv_split—Extract Fields from CSV Records15.2.4. put_record—Assemble a Database Record from an Array15.2.5. put_csv—Assemble Fields into a CSV Record15.2.6. db-demo—View and Edit a Password File15.3. PhoneBase: A Simple Phone Number Database15.3.1. ph—Look Up a Phone Number15.3.2. phadd—Add an Entry to PhoneBase15.3.3. phdel—Delete an Entry from PhoneBase15.3.4. phx—Show ph Search Results in an X Window15.4. Summary
16. Home on the Web
16.1. Playing with Hypertext: The html-funcs Library16.1.1. get_element—Extract the First Occurrence of an Element16.1.2. split_tags—Put Each Tag on Its Own Line16.1.3. html-title—Get the Title from an HTML File16.2. HTML on the Fly: The cgi-funcs Library16.2.1. x2d2—Convert a Two-Digit Hexadecimal Number to Decimal16.2.2. dehex—Convert Hex Strings (%XX) to Characters16.2.3. filedate—Find and Format the Modification Date of a File16.3. Creating HTML Files16.3.1. mk-htmlindex—Create an HTML Index16.3.2. pretext—Create a Wrapper Around a Text File16.3.3. text2html—Convert a Text File to HTML16.3.4. demo.cgi—A CGI Script16.4. Summary
17. Taking Care of Business
17.1.17.1.1. prcalc—A Printing Calculator17.1.2. gle—Keeping Records Without a Shoebox17.2. Summary
18. Random Acts of Scripting
18.1. The rand-funcs Library18.1.1. random—Return One or More Random Integers in a Given Range18.1.2. toss—Simulate Tossing a Coin18.1.3. randstr—Select a String at Random18.2. A Random Sampling of Scripts18.2.1. rand-date—Generate Random Dates in ISO Format18.2.2. randsort—Print Lines in Random Order18.2.3. randomword—Generate Random Words According to Format Specifications18.2.4. dice—Roll a Set of Dice18.2.5. throw—Throw a Pair of Dice18.3. Summary
19. A Smorgasbord of Scripts
19.1.19.1.1. topntail—Remove Top and Bottom Lines from a File19.1.2. flocate—Locate Files by Filename Alone19.1.3. sus—Display a POSIX Man Page19.1.4. cwbw—Count Words Beginning With19.1.5. cci—Configure, Compile, and Install from Tarball19.1.6. ipaddr—Find a Computer's Network Address19.1.7. ipaddr.cgi—Print the Remote Address of an HTTP Connection19.1.8. iprev—Reverse the Order of Digits in an IP Address19.1.9. intersperse—Insert a String Between Doubled Characters19.1.10. ll—Use a Pager for a Directory Listing Only If Necessary19.1.11. name-split—Divide a Person's Full Name into First, Last, and Middle Names19.1.12. rot13—Encode or Decode Text19.1.13. showfstab—Show Information from /etc/fstab19.1.14. unique—Remove All Duplicate Lines from a File19.2. Summary
20. Script Development Management
20.1.20.1.1. script-setup—Prepare the Scripting Environment20.1.2. cpsh—Install Script and Make Backup Copy20.1.3. shgrep—Search Scripts for String or Regular Expression20.1.4. shcat—Display a Shell Script20.2. Summary
A. Internet Scripting Resources
A.1. Introductions to Shell ScriptingA.2. Intermediate and Advanced ScriptingA.3. Collections of ScriptsA.4. Home Pages for ShellsA.5. Regular Expressions, sed, and awkA.6. Miscellaneous PagesA.7. History of the Shell

Content preview from Shell Scripting Recipes: A Problem-Solution Approach

Chapter 15. Covering All Your Databases

Adatabase is simply a collection of information. The information is usually divided into records that may be simple or complex. There may be a handful of records, or a few dozen. Or there may be thousands or millions of records.

Each record in a database may be a single word, or it may comprise a number of fields. These fields may be self-contained data, or they may be keys to another database. A record may be one or more lines in a file, or an entire file by itself.

The /etc/passwd file on a Unix system is a database containing one record for each user. Each record contains seven fields, separated by colons. A typical entry for root is

root:x:0:0:root:/root:/bin/sh

The word-finder and anagram scripts in Chapter ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Shell Scripting Recipes: A Problem-Solution Approach, Second Edition

Publisher Resources

ISBN: 9781590594711Purchase book

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Shell Scripting Recipes: A Problem-Solution Approach

by Michael Young

Chapter 15. Covering All Your Databases

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.