Mastering Delphi Programming: A Complete Reference Guide

Filter stage

The filter stage creates a TStringList object, which holds the names of all the URLs that have already been processed. For each URL received on the input, it checks whether the URL is already in the list. If so, the URL is thrown away. The code also checks if the URL belongs to the site we are crawling. If not, it is also thrown away. (We certainly don't want to crawl the entire internet!) If all of the tests pass, the URL is added to the list of already processed links and is sent to the output queue.

The Asy_UniqueFilter method implements the filter stage, as follows:

procedure TWebSpider.Asy_UniqueFilter(baseUrl: string;  inQueue, outQueue: TThreadedQueue<string>);var  baseUrl2: string;  url: string; visitedPages: TStringList; ...