Extracting Unique Elements from a List
Problem
You want to eliminate duplicate values from a list, such as when you build the list from a file or from the output of another command. This recipe is equally applicable to removing duplicates as they occur in input and to removing duplicates from an array you’ve already populated.
Solution
Use a hash to record which items have been seen, then
keys to extract them. You can use Perl’s
idea of truth to shorten and speed up your code.
Straightforward
%seen = ();
@uniq = ();
foreach $item (@list) {
unless ($seen{$item}) {
# if we get here, we have not seen it before
$seen{$item} = 1;
push(@uniq, $item);
}
}Faster
%seen = ();
foreach $item (@list) {
push(@uniq, $item) unless $seen{$item}++;
}Similar but with user function
%seen = ();
foreach $item (@list) {
some_func($item) unless $seen{$item}++;
}Faster but different
%seen = ();
foreach $item (@list) {
$seen{$item}++;
}
@uniq = keys %seen;Faster and even more different
%seen = ();
@uniqu = grep { ! $seen{$_} ++ } @list;Discussion
The question at the heart of the matter is “Have I seen this element before?” Hashes are ideally suited to such lookups. The first technique (Section 4.6.2.1) builds up the array of unique values as we go along, using a hash to record whether something is already in the array.
The second technique (Section 4.6.2.2) is the most natural way to write this sort of thing in Perl. It creates a new entry in the hash every time it sees an element that hasn’t been seen before, ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access