14.7. Obtaining the HTML from a URL
Problem
You need to get the HTML returned from a web server in order to examine it for items of interest. For example, you could examine the returned HTML for links to other pages or for headlines from a news site.
Solution
You can use the methods for web communication that were set up in Recipes 14.4 and 14.5 to make the HTTP request and verify the response; then, you can get at the HTML via the ResponseStream property of the HttpWebResponse object:
public static string GetHtmlFromUrl(Uri url)
{
string html = string.Empty;
HttpWebRequest request = GenerateHttpWebRequest(url);
using(HttpWebResponse response = (HttpWebResponse)request.GetResponse( ))
{
if (CategorizeResponse(response) == ResponseCategories.Success)
{
Stream responseStream = response.GetResponseStream( );
using(StreamReader reader =
new StreamReader(responseStream, Encoding.UTF8))
{
html = reader.ReadToEnd( );
}
}
}
return html;
}Discussion
The GetHtmlFromUrl method gets a web page using the GenerateHttpWebRequest and GetResponse methods, verifies the response using the CategorizeResponse method, and then, once it has a valid response, starts looking for the HTML that was returned.
The GetResponseStream method on the HttpWebResponse provides access to the body of the message that was returned in a System.IO.Stream object. In order to read the data, instantiate a StreamReader with the response stream and the UTF8 property of the Encoding class to allow for the UTF8-encoded text data to ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access