Skip to Content
C# 3.0 Cookbook, 3rd Edition
book

C# 3.0 Cookbook, 3rd Edition

by Jay Hilyard, Stephen Teilhet
December 2007
Intermediate to advanced
896 pages
19h 57m
English
O'Reilly Media, Inc.
Content preview from C# 3.0 Cookbook, 3rd Edition

14.7. Obtaining the HTML from a URL

Problem

You need to get the HTML returned from a web server in order to examine it for items of interest. For example, you could examine the returned HTML for links to other pages or for headlines from a news site.

Solution

You can use the methods for web communication that were set up in Recipes 14.4 and 14.5 to make the HTTP request and verify the response; then, you can get at the HTML via the ResponseStream property of the HttpWebResponse object:

	public static string GetHtmlFromUrl(Uri url)
	{
	    string html = string.Empty;
	    HttpWebRequest request = GenerateHttpWebRequest(url);
	    using(HttpWebResponse response = (HttpWebResponse)request.GetResponse( ))
	    {
	        if (CategorizeResponse(response) == ResponseCategories.Success)
	        {
	            Stream responseStream = response.GetResponseStream( );
	            using(StreamReader reader =
	                  new StreamReader(responseStream, Encoding.UTF8))
	            {
	                 html = reader.ReadToEnd( );
	            }
	         }
	     }
	     return html;
	}

Discussion

The GetHtmlFromUrl method gets a web page using the GenerateHttpWebRequest and GetResponse methods, verifies the response using the CategorizeResponse method, and then, once it has a valid response, starts looking for the HTML that was returned.

The GetResponseStream method on the HttpWebResponse provides access to the body of the message that was returned in a System.IO.Stream object. In order to read the data, instantiate a StreamReader with the response stream and the UTF8 property of the Encoding class to allow for the UTF8-encoded text data to ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

AirBnbBlueOriginElectronic ArtsHomeDepotNasdaqRakutenTata Consultancy Services

QuotationMarkO’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.
Julian F.
Head of Cybersecurity
QuotationMarkI wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.
Addison B.
Field Engineer
QuotationMarkI’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.
Amir M.
Data Platform Tech Lead
QuotationMarkI'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
Mark W.
Embedded Software Engineer

You might also like

C# 6.0 Cookbook, 4th Edition

C# 6.0 Cookbook, 4th Edition

Stephen Teilhet, Jay Hilyard
C# Cookbook

C# Cookbook

Stephen Teilhet, Jay Hilyard
C# Cookbook

C# Cookbook

Joe Mayo
C# Cookbook, 2nd Edition

C# Cookbook, 2nd Edition

Jay Hilyard, Stephen Teilhet

Publisher Resources

ISBN: 9780596516109Errata Page