November 2019
Beginner to intermediate
674 pages
15h
English
The downloader stage uses a THTTPClient object to download the web page. This task is implemented in the Asy_HttpGet method, as follows:
procedure TWebSpider.Asy_HttpGet( inQueue: TThreadedQueue<string>; outQueue: TThreadedQueue<THttpPage>);var httpClient: THTTPClient; response: IHTTPResponse; url: string;begin TThread.NameThreadForDebugging('Http get'); httpClient := THTTPClient.Create; try while inQueue.PopItem(url) = wrSignaled do begin if inQueue.ShutDown then break; //while try response := httpClient.Get(url); except if TInterlocked.Decrement(FPageCount) = 0 then NotifyFinished; end; if (response.StatusCode div 100) = 2 then outQueue.PushItem(THttpPage.Create(url, response)) else if TInterlocked.Decrement(FPageCount) ...Read now
Unlock full access