O'Reilly logo

Perl & LWP by Sean M. Burke

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Chapter 8. Tokenizing Walkthrough

So far, I’ve been showing examples of data in a particular format, then presenting code for extracting the data out of that format, as an illustration of newly introduced HTML::TokeParser methods. But in real life, you do not proceed tidily from the problem to an immediate and fully formed solution. And ideally, the task of data extraction is simple: identify patterns surrounding the data you’re after and write a program that matches those patterns and extracts the embedded data.

In practice, however, you write programs bit by bit and in fits and starts, and with data extraction specifically; this involves a good amount of trying one pattern, finding that its matching is too narrow or too broad, trying to amend it, possibly having to backtrack and try another pattern, and so on. Moreover, even equally effective patterns are not equal; some patterns are easier to capture in code than others, and some patterns are more temporary than others.

In this section, I’ll try to make these points by walking though the implementation of a data extraction task, with all alternatives considered, and even a misstep or two.

The Problem

As a starting point, consider the task of harvesting a month’s worth of listings and corresponding RealAudio URLs from the web site of the National Public Radio program Fresh Air, at http://freshair.npr.org. Fresh Air is on NPR stations each weekday, and on every show, different guests are interviewed. The show’s web site lists which ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required