Mastering UNIX® Shell Scripting: Bash, Bourne, and Korn Shell Scripting for Programmers, System Administrators, and UNIX Gurus, Second Edition

CHAPTER2

24 Ways to Process a File Line-by-Line

Have you ever created a really slick shell script to process file data and found that you have to wait until after lunch to get the results? The script may be running so slowly because of how you are processing the file. Over mdy many years of UNIX administration I have seen some rather, well, let's call them creative ways to process a file line-by-line. I have listed 24 of these methods in this chapter. Some techniques are very fast, and some make you wait for half a day. Then there are a few that, on first look, should work but do not. The techniques used in this chapter are measurable, and I created a shell script that will time each method so that you can see which method suits your needs.

When processing an ASCII text/data file, we are normally inside a loop of some kind. Then, as we go through the file from the top to the bottom, we process each line of text. A UNIX shell script is really not meant to work on a text file character-by-character (Pearl is better for this), but you can accomplish the task using various techniques. The task for this chapter is to show the line-by-line parsing techniques by parsing each entire line at once, not character by character. We are also going to look at using file descriptors as a method of file input and output redirection for file processing.

Command Syntax

First, as always, we need to go over the command syntax that we are going to use. The commands that we want to concentrate on in ...