Linux Pocket Guide, 2nd Edition

Name

uniq — stdin  stdout  - file  -- opt  --help  --version

Synopsis

uniq [options] [files]

The uniq command operates on consecutive, duplicate lines of text. For example, if you have a file myfile:

$ cat myfile
a
b
b
c
b

then uniq would detect and process (in whatever way you specify) the two consecutive b’s, but not the third b.

$ uniq myfile
a
b
c
b

uniq is often used after sorting a file:

$ sort myfile | uniq
a
b
c

In this case, only a single b remains because all three were made adjacent by sort, then collapsed to one by uniq. Also, you can count duplicate lines instead of eliminating them:

$ sort myfile | uniq -c
      1 a
      3 b
      1 c

Useful options

-c	Count adjacent duplicate lines.
-i	Case-insensitive operation.
-u	Print unique lines only.
-d	Print duplicate lines only.
-s N	Ignore the first N characters on each line when detecting duplicates.
-f N	Ignore the first N whitespace-separated fields on each line when detecting duplicates.
-w N	Consider only the first N characters on each line when detecting duplicates. If used with -s or -f, sort will ignore the specified number of characters or fields first, then consider the next N characters.