Spidering Hacks
by Kevin Hemenway, Tara Calishain

The unconfirmed error reports are from readers.  They have not yet been 
approved or disproved by the author or editor and represent solely the
opinion of the reader.

Here's a key to the markup:
        [page-number]: serious technical mistake
        {page-number}: minor technical mistake
        <page-number>: important language/formatting problem
        (page-number): language change or minor formatting problem
        ?page-number?: reader question or request for clarification

This page was updated Sept. 6, 2006.


UNCONFIRMED  errors and comments from readers: 


(48)  top of script;
The "$|++" should be bold. The text explains that the only meaningful change is in bold. 
Without this change, the progress bar is buffered, and the point of the hack is lost.


{50}  in the foreach loop;
In the progress bar examples, you call ua->head() to get the content-length, then
make a separate call to ua->get() to get the data, passing the content-length to the
progress callback via the lexical variable $total_size.  This is not necessary.  The
$response value passed to the callback contains the Content-Length header, so you can
avoid the call to ua->head() altogether by rewriting callback() as

sub callback {
  my ($data, $response, $protocol) = @_;
  $final_data .= $data;
  print progress_bar(
    length($final_data),
    $response->header('Content-Length'),
    25,
    '=');
}