Linux and Unix Shell Programming

11.3. Using uniq

The purpose of uniq is to strip or suppress duplicate lines from a text file. Uniq assumes the file has been sorted to get the correct results, though this is not set in stone: you can use any type of unordered text and you can even compare ordinary lines of text if you wish.

You may think that this is a bit like the unique option in the sort command. Well, it is in a way, but with one important difference. The sort’s uniq option gets rid of all duplicates, whereas the uniq command does not get rid of all duplicates. What is a repeated line then? In uniq ’s case it means any line that is continuously repeated without a break. Here’s an example.

						$ pg myfile.txt 
May Day 
May Day 
May Day 
Going Down 
May Day.
					

Uniq sees the first ...