Skip to Main Content
Perl Cookbook
book

Perl Cookbook

by Tom Christiansen, Nathan Torkington
August 1998
Intermediate to advanced content levelIntermediate to advanced
800 pages
39h 20m
English
O'Reilly Media, Inc.
Content preview from Perl Cookbook

Processing Every Word in a File

Problem

You need to do something to every word in a file, similar to the foreach function of csh.

Solution

Either split each line on whitespace:

while (<>) {
    for $chunk (split) {
        # do something with $chunk
    }
}

Or use the m//g operator to pull out one chunk at a time:

while (<>) {
    while ( /(\w[\w'-]*)/g ) {
        # do something with $1
    }
}

Discussion

Decide what you mean by “word.” Sometimes you want anything but whitespace, sometimes you only want program identifiers, and sometimes you want English words. Your definition governs which regular expression to use.

The preceding two approaches work differently. Patterns are used in the first approach to decide what is not a word. In the second, they’re used to decide what is a word.

With these techniques, it’s easy to make a word frequency counter. Use a hash to store how many times each word has been seen:

# Make a word frequency count
%seen = ();
while (<>) {
    while ( /(\w['\w-]*)/g ) {
        $seen{lc $1}++;
    }
}

# output hash in a descending numeric sort of its values
foreach $word ( sort { $seen{$b} <=> $seen{$a} } keys %seen) {
    printf "%5d %s\n", $seen{$word}, $word;
}

To make the example program count line frequency instead of word frequency, omit the second while loop and do $seen{lc $_}++ instead:

# Line frequency count
%seen = ();
while (<>) {
    $seen{lc $_}++;
}
foreach $line ( sort { $seen{$b} <=> $seen{$a} } keys %seen ) {
    printf "%5d %s", $seen{$line}, $line;
}

Odd things that may need to be considered as words include ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Start your free trial

You might also like

Perl in a Nutshell

Perl in a Nutshell

Nathan Patwardhan, Ellen Siever, Stephen Spainhour
Perl Best Practices

Perl Best Practices

Damian Conway
Mastering Perl

Mastering Perl

brian d foy
Perl Cookbook, 2nd Edition

Perl Cookbook, 2nd Edition

Tom Christiansen, Nathan Torkington

Publisher Resources

ISBN: 1565922433Supplemental ContentCatalog PageErrata