bash Cookbook

Processing Files with No Line Breaks

Problem

You have a large file with no line breaks, and you need to process it.

Solution

Pre-process the file and add line breaks in appropriate places. For example, Open- Office.org’s Open Document Format (ODF) files are basically zipped XML files. It is possible to unzip them and grep the XML, which we did a lot while writing this book. See Comparing Two Documents for a more comprehensive treatment of ODF files. In this example, we insert a newline after every closing angle bracket (>). That makes it much easier to process the file using grep or other textutils. Note that we must enter a backslash followed immediately by the Enter key to embed an escaped newline in the sed script:

$ wc -l content.xml
1 content.xml


$ sed -e 's/>/>\
/g' content.xml | wc -l
1687

If you have fixed-length records with no newlines, do this instead, where 48 is the length of the record.

$ cat fixed-length Line_1_ _aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZLine_2_ _ aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZLine_3_ _ aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZLine_4_ _ aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZLine_5_ _ aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZLine_6_ _ aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZLine_7_ _ aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZLine_8_ _ aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZLine_9_ _ aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZLine_10_ aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZLine_11_ aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZLine_12_ aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ ...