RESTful Web Services

Chapter 11. Ajax Applications as REST Clients

Ajax applications have become very hot during the past couple of years. Significantly hotter, in fact, than even knowing what Ajax applications are. Fortunately, once you understand the themes of this book it’s easy to explain Ajax in those terms. At the risk of seeming old-fashioned, I’d like to present a formal definition of Ajax:

An Ajax application is a web service client that runs inside a web browser.

Does this make sense? Consider two examples widely accepted not to be Ajax applications: a JavaScript form validator and a Flash graphics demo. Both run inside the web browser, but they don’t make programmatic HTTP requests, so they’re not Ajax applications. On the flip side: the standalone clients I wrote in Chapters2 and 3 aren’t Ajax applications because they don’t run inside a web browser.

Now consider Gmail, a site that everyone agrees uses Ajax. If you log into Gmail you can watch your browser make background requests to the web service at mail.google.com, and update the web page you’re seeing with new data. That’s exactly what a web service client does. The Gmail web service has no public-facing name and is not intended for use by clients other than the Gmail web page, but it’s a web service nonetheless. Don’t believe it? There are libraries like libgmail that act as unofficial, non-Ajax clients to the Gmail web service. Remember, if it’s on the Web, it’s a web service.

This chapter covers client programming, and it picks up where Chapter 2 left off. Here I’m focusing on the special powers and needs of web service clients that run in a browser environment. I cover JavaScript’s XMLHttpRequest class and the browser’s DOM, and show how security settings affect which web service clients you can run in a browser.

From AJAX to Ajax

Every introduction to Ajax will tell you that it used to be AJAX, an acronym for Asynchronous JavaScript And XML. The acronym has been decommissioned and now Ajax is just a word. It’s worth spending a little time exploring why this happened. Programmers didn’t suddenly lose interest in acronyms. AJAX had to be abandoned because what it says isn’t necessarily true. Ajax is an architectural style that doesn’t need to involve JavaScript or XML.

The JavaScript in AJAX actually means whatever browser-side language is making the HTTP requests. This is usually JavaScript, but it can be any language the browser knows how to interpret. Other possibilities are ActionScript (running within a Flash application), Java (running within an applet), and browser-specific languages like Internet Explorer’s VBScript.

XML actually means whatever representation format the web service is sending. This can be any format, so long as the browser side can understand it. Again, this is usually XML, because it’s easy for browsers to parse, and because web services tend to serve XML representations. But JSON is also very common, and it can be also be HTML, plain text, or image files: anything the browser can handle or the browser-side script can parse.

So AJAX hackers decided to become Ajax hackers, rather than always having to explain that JavaScript needn’t mean JavaScript and XML might not be XML, or becoming Client-Side Scripting And Representation Format hackers. When I talk about Ajax in this book I mostly talk in terms of JavaScript and XML, but I’m not talking about those technologies: I’m talking about an application architecture.

The Ajax Architecture

The Ajax architecture works something like this:

1. A user, controlling a browser, makes a request for the main URI of an application.

2. The server serves a web page that contains an embedded script.

3. The browser renders the web page and either runs the script, or waits for the user to trigger one of the script’s actions with a keyboard or mouse operation.

4. The script makes an asynchronous HTTP request to some URI on the server. The user can do other things while the request is being made, and is probably not even aware that the request is happening.

5. The script parses the HTTP response and uses the data to modify the user’s view. This might mean using DOM methods to change the tag structure of the original HTML page. It might mean modifying what’s displayed inside a Flash application or Java applet.

   From the user’s point of view, it looks like the GUI just modified itself.

This architecture looks a lot like that of a client-side GUI application. In fact, that’s what this is. The web browser provides the GUI elements (as described in your initial HTML file) and the event loop (through JavaScript events). The user triggers events, which get data from elsewhere and alter the GUI elements to fit. This is why Ajax applications are often praised as working like desktop applications: they have the same architecture.

A standard web application has the same GUI elements but a simpler event loop. Every click or form submission causes a refresh of the entire view. The browser gets a new HTML page and constructs a whole new set of GUI elements. In an Ajax application, the GUI can change a little bit at a time. This saves bandwidth and reduces the psychological effects on the end user. The application appears to change incrementally instead of in sudden jerks.

The downside is that every application state has the same URI: the first one the end user visited. Addressability and statelessness are destroyed. The underlying web service may be addressable and stateless, but the end user can no longer bookmark a particular state, and the browser’s “Back” button stops working the way it should. The application is no longer on the Web, any more than a SOAP+WSDL web service that only exposes a single URI is on the Web. I discuss what to do about this later.

A del.icio.us Example

Back in Chapter 2 I showed clients in various languages for a REST-RPC hybrid service: the API for the del.icio.us social bookmarking application. Though I implemented my own, fully RESTful version of that service in Chapter 7, I’m going to bring the original service out one more time to demonstrate a client written in JavaScript. Like most JavaScript programs, this one runs in a web browser, and since it’s a web service client, that makes it an Ajax application. Although simple, this program brings up almost all of the advantages of and problems with Ajax that I discuss in this chapter.

The first part of the application is the user interface, implemented in plain HTML. This is quite different from my other del.icio.us clients, which ran on the command line and wrote their data to standard output (see Example 11-1).

Example 11-1. An Ajax client to the del.icio.us web service

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0
 Transitional//EN" "http://www.w3.org/TR/REC-html40/transitional.dtd">
<!--delicious-ajax.html-->
<!--An Ajax application that uses the del.icio.us web service. This
     application will probably only work when saved as a local
     file. Even then, your browser's security policy might prevent it
     from running.-->

<html>
 <head>
  <title>JavaScript del.icio.us</title>
 </head> 
 <body>
  <h1>JavaScript del.icio.us example</h1>

  <p>Enter your del.icio.us account information, and I'll fetch and
  display your most recent bookmarks.</p>

  <form onsubmit="callDelicious(); return false;"> 
    Username: <input id="username" type="text" /><br /> 
    Password: <input id="password" type="password" /><br /> 
    <input type="submit" value="Fetch del.icio.us bookmarks"/>
  </form>

  <div id="message"></div>
  
  <ul id="links"></ul>

My user interface is an HTML form that doesn’t point anywhere, and some tags (div and ul) that don’t contain anything. I’m going to manipulate these tags with JavaScript functions. The first is setMessage, which puts a given string into the div tag (see Example 11-2).

Example 11-2. Ajax client continued: definition of setMessage

  <script type="text/javascript">      
   function setMessage(newValue) { 
     message = document.getElementById("message"); 
     message.firstChild.textContent = newValue;
   }

And it’s not quite fair to say that the HTML form doesn’t point anywhere. Sure, it doesn’t have an “action” attribute like a normal HTML form, but it does have an onsubmit event handler. This means the web browser will call the JavaScript function callDelicious whenever the end user clicks the submit button. Instead of going through the page request loop of a web browser, I’m using the GUI-like event loop of a JavaScript program.

The callDelicious function uses the JavaScript library XMLHttpRequest to fetch data from https://api.del.icio.us/v1/posts/recent/. This is the URI used throughout Chapter 2 to fetch a user’s most recent del.icio.us bookmarks. First we need to do some housekeeping: get permission from the browser to send the request, and gather whatever data the user entered into the HTML form (see Example 11-3).

Example 11-3. Ajax client continued: definition of callDelicious

   function callDelicious() { // Get permission from the browser to send the request. 
     try {
       if (netscape.security.PrivilegeManager.enablePrivilege)
         netscape.security.PrivilegeManager.enablePrivilege("UniversalBrowserRead");
     } catch (e) { 
       alert("Sorry, browser security settings won't let this program run."); 
       return; 
     }

     // Fetch the user-entered account information 
     var username = document.getElementById("username").value; 
     var password = document.getElementById("password").value;

     // Remove any old links from the list. 
     var links = document.getElementById("links"); 
     while (links.firstChild) 
       links.removeChild(links.firstChild) 
     setMessage("Please wait...");

Now we’re ready to send the HTTP request, as shown in Example 11-4.

Example 11-4. callDelicious definition continued

     // Send the request. 
     // See "Working Around the Corner Cases" for a cross-browser 
     // "createXMLHttpRequest" implementation. 
     request = new XMLHttpRequest(); 
     request.open("GET", "https://api.del.icio.us/v1/posts/recent", true, 
                  username, password);
     request.onreadystatechange = populateLinkList; 
     request.send(null);

The third JavaScript function I’ll define is populateLinkList. I’ve already referenced this function, in the line request.onreadystatechange = populateLinkList. That line sets up populateLinkList as a callback function. The idea is that while api.del.icio.us is processing the request, the user can go about her business, surfing the web in another browser window. Once the request completes, the browser calls populateLinkList, which handles the response. You can do JavaScript programming without these callback functions, but it’s a bad idea. Without callbacks, the web browser will go nonresponsive while the XMLHttpRequest object is making an HTTP request. Not very asynchronous.

The job of populateLinkList is to parse the XML document from the del.icio.us web service. The representation in Example 11-5 represents a list of bookmarks, and populateLinkList turns each bookmark into a list item of the formerly empty ul list tag.

Example 11-5. Ajax client concluded: definition of populateLinkList

     // Called when the HTTP request has completed. 
     function populateLinkList() { 
       if (request.readyState != 4) // Request has not yet completed
         return;

      setMessage("Request complete."); 
      if (netscape.security.PrivilegeManager.enablePrivilege) 
        netscape.security.PrivilegeManager.enablePrivilege("UniversalBrowserRead");
      // Find the "post" tags in the representation
      posts = request.responseXML.getElementsByTagName("post");

      setMessage(posts.length + " link(s) found:"); 
      // For every "post" tag in the XML document...
      for (var i = 0; i < posts.length; i++) { 
        post = posts[i]; 
        // ...create a link that links to the appropriate URI.
        var link = document.createElement("a"); 
        var description = post.getAttribute('description'); 
        link.setAttribute("href", post.getAttribute('href'));
        link.appendChild(document.createTextNode(description));

        // Stick the link in an "li" tag...
        var listItem = document.createElement("li");
        // ...and make the "li" tag a child of the "ul" tag.
        listItem.appendChild(link); links.appendChild(listItem)
      }
    }
  }
  </script> 
 </body>
</html>

The Advantages of Ajax

If you try out the del.icio.us client you’ll notice some nice features that come from the web browser environment. Most obviously: unlike the examples in Chapter 2, this application has a GUI. And as GUI programming goes, this is pretty easy. Method calls that seem to do nothing but manipulate a mysterious document data structure, actually change the end user’s view of the application. The document is just the thing the user sees rendered in the browser. Since the browser knows how to turn changes to the document into GUI layout changes, there’s no widget creation and layout specification, as you’d see in conventional GUI programs.

This client also never explicitly parses the XML response from the del.icio.us web service. A web browser has an XML parser built in, and XMLHttpRequest automatically parses into a DOM object any XML document that comes in on a web service response. You access the DOM object through the XMLHttpRequest.responseXML member. The DOM standard for web browsers defines the API for this object: you can iterate over its children, search it with methods like getElementsByTagName, or hit it with XPath expressions.

More subtly: try loading this HTML file and clicking the submit button without providing a username and password. You’ll get a dialog box asking you for a del.icio.us username and password: the same dialog box you get whenever your browser accesses a page that requires HTTP basic auth. This is exactly what you’re doing: visiting https://api.del.icio.us/v1/posts/recent, which requires HTTP basic auth, in your web browser. But now you’re doing it by triggering an action in an Ajax application, rather than clicking on a link to the URI.

Web browsers are by far the most popular HTTP clients out there, and they’ve been written to handle the corner cases of HTTP. You could remove both text fields from the HTML form in Example 11-1, and the Ajax application would still work, because real web browsers have their own user interface for gathering basic HTTP auth information.

The Disadvantages of Ajax

Unfortunately, thanks to the wide variety of web browsers in use, you’ll need to deal with a whole new set of corner cases if you want your application to work in all browsers. Later on I’ll show you code libraries and code snippets that work around the corner cases.

If you try out this program, you’ll also run into the problem I talked about at the end of Chapter 8: why should the end user trust the web service client? You’d trust your browser with your del.icio.us username and password, but this isn’t your browser. It’s a web service client that uses your browser to make HTTP requests, and it could be doing anything in those requests. If this was an official web page that was itself served from api.del.icio.us, then your browser would trust it to make web service calls to the server it came from. But it’s a web page that comes from a file on your hard drive, and wants to call out to the Web at large. To a web browser, this is very suspicious behavior.

From a security standpoint, this is no different from the standalone del.icio.us clients I wrote in other programming languages. But there’s no real reason why you should trust a standalone web service client, either. We just tend to assume they’re safe. A web browser is constantly loading untrusted web pages, so it has a security model that restricts what those pages can do in JavaScript. If strangers were always dumping executables into your home directory, you’d probably think twice before running them.

Which is why I called netscape.security.PrivilegeManager.enablePrivilege, asking the browser if it won’t let me make an HTTP request to a foreign domain (“UniversalBrowserRead”), and won’t it also let me use the browser’s XML parser on some data from a foreign domain (“UniversalBrowserRead” again, but in a different JavaScript function). Even with these calls in place, you’re likely to get browser security messages asking you if you want to accept this risky behavior. (These are not like the browser messages you might get when you do something innocuous like submit an HTML form, messages that Justin Mason once characterized as “are you sure you want to send stuff on the intarweb?”. These are more serious.) And that’s with this file sitting on your (presumably trusted) filesystem. If I tried to serve this Ajax application from oreilly.com, there’s no way your browser would let it make an HTTP request to api.del.icio.us.

So why don’t we see these problems all the time in Ajax applications? Because right now, most Ajax applications are served from the same domain names as the web services they access. This is the fundamental difference between JavaScript web service clients and clients written in other languages: the client and the server are usually written by the same people and served from the same domain.

The browser’s security model doesn’t totally prevent you from writing an XMLHttpRequest application against someone else’s web service, but it does make it difficult. According to the web browser, the only Ajax application safe enough to run without a warning is one that only makes requests against the domain it was served from. At the end of this chapter I’ll show you ways of writing Ajax clients that can consume foreign web services. Note, though, that these techniques rely heavily on cheating.

REST Goes Better

Ajax applications are web service clients, but why should they be clients of RESTful web services in particular? Most Ajax applications consume a web service written by the same people who wrote the application, mainly because the browser security model makes it difficult to do anything else. Why should it matter whether a service used by one client is fully RESTful or just a resource-oriented/RPC hybrid? There are even programs that turn a WSDL file into a JavaScript library for making RPC SOAP calls through XMLHttpRequest. What’s wrong with that?

Well, in general, the interface between two parts of an application matters. If RESTful architectures yield better web services, then you’ll benefit from using them, even if you’re the only one consuming the service. What’s more, if your application does something useful, people will figure out your web service and write their own clients—just as if your web site exposes useful information, people will screen-scrape it. Unless you want to obfuscate your web service so only you can use it, I think the Resource-Oriented Architecture is the best design.

The web services that Ajax applications consume should be RESTful for the same reasons almost all web services should be RESTful: addressability, statelessness, and the rest. The only twist here is that Ajax clients are embedded inside a web browser. And in general, the web browser environment strengthens the argument for REST. You probably don’t need me to reiterate my opinion of Big Web Services in this chapter, but SOAP, WSDL, and the rest of the gang look even more unwieldy inside a web browser. Maybe you’re a skeptic and you think the REST paradigm isn’t suitable as a general platform for distributed programming—but it should at least be suitable for the communication between a web browser and a web server!

Outside of a web browser, you might decide to limit yourself to the human web’s interface of GET and POST. Many client libraries support only the basic features of HTTP. But every Ajax application runs inside a capable HTTP client. Almost every web browser gives XMLHttpRequest access to the five basic HTTP methods, and they all let you customize the request headers and body.

What’s more, Ajax calls take place in the same environment as the end user’s other web browsing. If the client needs to make HTTP requests through a proxy, you can assume they’ve already configured it. An Ajax request sends the same cookies and Basic auth headers as do other browser requests to your domain. You can usually use the same authentication mechanisms and user accounts for your web site and your Ajax services.

Look back at steps 4 and 5 of the Ajax architecture—basically “GET a URI” and “use data from the URI to modify the view.” That fits in quite well with the Resource-Oriented Architecture. An Ajax application can aggregate information about a large number of resources, and incrementally change the GUI as the resource state changes. The architectural advantages of REST apply to Ajax clients just as they do to other clients. One example: you don’t need to coordinate the browser’s application state with the server if the server never keeps any application state.

Making the Request

Now I’d like to look at the technical details underlying the most common client language for Ajax: JavaScript. The major web browsers all implement a JavaScript HTTP client library called XMLHttpRequest. Its interface is simple because the browser environment handles the hairy edge cases (proxies, HTTPS, redirects, and so on). Because XMLHttpRequest is so simple, and because I want to drive home the point that it’s fundamentally no different from (say) Ruby’s open-uri, I’m going to cover almost the whole interface in this section and the next. If you’re already familiar with XMLHttpRequest, feel free to skim this section, or skip to the end where there’s a nice chart.

To build an HTTP request you need to create an XMLHttpRequest object. This seemingly simple task is actually one of the major points of difference between the web browsers. This simple constructor works in Mozilla-family browsers like Firefox:

request = new XMLHttpRequest();

The second step is to call the XMLHttpRequest.open method with information about the request. All but the first two arguments in this sample call are optional:

request.open([HTTP method], [URI], true, [Basic auth username], [Basic auth password]);

Pretty self-explanatory, except for the third argument, which I’ve hard-coded to true. This argument controls whether the browser carries out the request asynchronously (letting the user do other things while it’s going on) or synchronously (locking up the whole browser until it gets and parses the server response). Locking up the browser never creates a good user experience, so I never recommend it, even in simple applications. This does mean you have to set up a handler function to be called when the request completes:

request.onReadyStateChange = [Name of handler function];

If you want to set any HTTP request headers, you use setrequestHeader:

request.setRequestHeader([Header name], [Header value]);

Then you send the request to the HTTP server by calling send. If the request is a POST or PUT request, you should pass the entity-body you want to send as an argument to send. For all other requests, it should be null.

request.send([Entity-body]);

If all goes well, your handler function (the one you set to request.onReadyStateChange) will be called four times over the lifetime of the HTTP request, and the value of request.readyState will be different every time. The value you’re looking for is the last one, 4, which means that the request has completed and it’s time to manipulate the response. If request.readyState doesn’t equal 4, you’ll just return from the handler function.

XMLHttpRequest uses the underlying web browser code to make its requests. Since the major web browsers are among the most complete HTTP client implementations around, this means that XMLHttpRequest does pretty well on the feature matrix I introduced for HTTP clients back in Chapter 2. Cookies, proxies, and authentication tend to work in Ajax applications as they do in normal web access.

Handling the Response

Eventually the request will complete and the browser will call your handler function for the last time. At this point your XMLHttpRequest instance gains some new and interesting abilities:

• The status property contains the numeric status code for the request.

• The responseXML property contains a preparsed DOM object representing the response document—assuming it was served as XML and the browser can parse it. HTML, even XHTML, will not be parsed into responseXML, unless the document was served as an XML media type like application/xml or application/xhtml+xml.

• The responseText property contains the response document as a raw string—useful when it’s JSON or some other non-XML format.

• Passing the name of an HTTP header into the getResponseHeader method looks up the value of that header.

Web browsers epitomize the tree-style parsing strategy that turns a document into a data structure. When you make a web service request from within JavaScript, the responseXML property gives you your response document as a tree. You can access the representation with a standardized set of DOM manipulation methods.

Tip

Unlike the XMLHttpRequest interface, the DOM interface is extremely complex and I won’t even think about covering it all here. See the official standard, the Mozilla DOM reference, or a book like Dynamic HTML: The Definitive Reference by Danny Goodman (O’Reilly).

You can navigate the tree with methods like getElementByID, and run XPath queries against it with evaluate.

But there’s another treelike data structure in town: the HTML document being displayed in the end user’s web browser. In an Ajax application, this document is your user interface. You manipulate it with the same DOM methods you use to extract data from an XML web service representation. An Ajax application acts as glue between the raw data the web service sends, and the HTML GUI the end user sees. Useful DOM methods here are createTextNode and createElement, both of which I used in Example 11-5.

JSON

I covered JSON briefly in Chapter 2. I brought it up again in Chapter 9 as one of my recommended representation formats. But since it comes from JavaScript, I want to show it in action in the Ajax chapter.

Example 11-6 shows of an Ajax client for Yahoo!’s image search web service.

Example 11-6. An Ajax client that calls out to a service that serves JSON representations

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"
"http://www.w3.org/TR/REC-html40/transitional.dtd">
<!--yahoo-ajax.html-->
<!--An Ajax application that uses the JSON output feature of Yahoo!
    Web Services-->

<html>
<head><title>Javascript Yahoo! JSN</title></head>
<body>
<h1>Javascript Yahoo! JSON example</h1>

<p id="message">Loading pictures of baby elephants!</p>

<div id="images">
</div>

<script type="text/javascript">
function formatImages(result)
{
  var images = document.getElementById("images");
  items = result["ResultSet"]["Result"];
  document.getElementById("message").firstChild.textContent =
    items.length + " baby elephant pictures:";
  for (var i = 0; i < items.length; i++)
  {
    image = items[i];

    // Create a link
    var link = document.createElement("a");
    link.setAttribute("href", image["ClickUrl"]);

    // Put a thumbnail image in the link.
    var img = document.createElement("img");
    var thumbnail = image["Thumbnail"];
    img.setAttribute("src", thumbnail["Url"]);
    img.setAttribute("width", thumbnail["Width"]);
    img.setAttribute("height", thumbnail["Height"]);
    img.setAttribute("title", image["Height"]);
    link.appendChild(img);    
    images.appendChild(link);
  }
}
</script>

<script type="text/javascript" 
 src="http://api.search.yahoo.com/ImageSearchService/V1/imageSearch
?appid=restbook&query=baby+elephant&output=json&callback=formatImages" />

</body>
</html>

If you load this HTML file in your web browser you’ll see some cute pictures of baby elephants, courtesy of Yahoo! Image Search. What you won’t see is a browser security warning. The del.icio.us example had to ask the browser if it was OK to make an XMLHttpRequest to another domain, and even then the browser imposes strict rules about when it is OK. But this Ajax client just makes the web service call. That’s because it doesn’t make the call through XMLHttpRequest. It uses a technique described as JavaScript on Demand (JoD). JoD bypasses the browser’s security policy by fetching custom-generated JavaScript from a web service. Because any JSON data structure is a valid JavaScript program, this works especially well with web services that serve JSON representations.

Don’t Bogart the Benefits of REST

It’s easy for an Ajax application to take all the advantages of REST for itself, and leave none of them for the end user. Gmail is a good example of this. The Gmail Ajax application benefits greatly from its use of an addressable, stateless web service. But in terms of user experience, all the end user sees is one constantly changing HTML page. No addressability for you! If you want to bookmark a search or a particular email message, you need to start off at Gmail’s plain HTML interface.

Ordinarily, your browser’s back and forward buttons move you back and forth through application state. This works because the Web is stateless. But if you start using a typical Ajax application, your back button breaks. Clicking it doesn’t take you backward in application state: it takes you to the page you were on before you started using the Ajax application. No statelessness for you!

The underlying cause is the same thing that gives Ajax applications their polished look. Ajax applications disconnect the end user from the HTTP request-response cycle. When you visit the URI of an Ajax application, you leave the Web. From that point on you’re using a GUI application that makes HTTP requests for you, behind the scenes, and folds the data back into the GUI. The GUI application just happens to be running in the same piece of software you use to browse the Web. But even an Ajax application can give its users the benefits of REST, by incorporating them into the user interface. I’m basically asking you to reinvent some of the features of the web browser within your application.

The best example of this is Google Maps, the application that started the Ajax craze. At first glimpse, Google Maps looks about as addressable as Gmail. You visit http://maps.google.com/ and are presented with a large-scale map. You can use Ajax to zoom in and navigate to any point on the globe, but the URI in your browser’s address bar never changes.

But Google Maps also uses Ajax to maintain a “permalink” for whatever point on the globe you’re currently at. This URI is kept not in your browser’s address bar but in an a tag in the HTML document (see Figure 11-1). It represents all the information Google Maps needs to identify a section of the globe: latitude, longitude, and map scale. It’s a new entry point into the Ajax application. This link is the Google Maps equivalent of your browser’s address bar.

Figure 11-1. An entry point into a web service (that is, a URI), maintained through Ajax

Thanks to the extra DOM work that keeps this a tag up to date as you navigate the map, every point on every map is on the Web. Any point can be bookmarked, blogged about, and emailed around. Anyone who visits one of these URIs enters the Google Maps Ajax application at the right point, instead of getting a view centered on the continental US (as would happen if you navigated to a place on a map and then reloaded http://maps.google.com/). Addressability, destroyed by Ajax but added back by good application design, has allowed communities like Google Sightseeing to grow up around the Google Maps application.

Your Ajax applications can give statelessness back by reproducing the functionality of the browser’s back and forward buttons. You don’t have to reproduce the browser’s behavior slavishly. The point is to let the end user move back and forth in his application state, instead of having to start from the beginning of a complex operation if he makes a mistake or gets lost.

Cross-Browser Issues and Ajax Libraries

As always when web browsers are involved, different clients have different levels of support for XMLHttpRequest. And as always seems to happen, Internet Explorer is the major outlier. This isn’t quite fair, because XMLHttpRequest was a Microsoft invention and Internet Explorer was the first browser to support Ajax at all. But until the release of Internet Explorer 7, Ajax was implemented as Windows-specific technology: an ActiveX control called XMLHttp.

The cross-platform Mozilla project adopted the API of the XMLHttp control, but implemented it as a class you could instantiate directly from JavaScript. Other browsers followed this lead, and all current browsers now use the XMLHttpRequest name (including the new Internet Explorer). But old versions of Internet Explorer still make up a big portion of the user base, so cross-browser issues are still a problem.

Example 11-7 is a JavaScript function that always creates an object that acts like an XMLHttpRequest, even though under the covers it may be an ActiveX control. It was written by Bret Taylor and comes from his site at http://ajaxcookbook.org/.

Example 11-7. A cross-browser wrapper for XMLHttpRequest

function createXMLHttpRequest() { 
  if (typeof XMLHttpRequest != "undefined") { 
    return new XMLHttpRequest();
  } else if (typeof ActiveXObject != "undefined") { 
    return new ActiveXObject("Microsoft.XMLHTTP"); 
  } else { 
    throw new Error("XMLHttpRequest not supported");
  }
}

This function is a drop-in replacement for the XMLHttpRequest constructor in Example 11-3, instead of this:

request = new XMLHttpRequest();

you might write this:

request = createXMLHttpRequest();

I know of two other major cross-browser issues. First, the Safari browser doesn’t support the PUT and DELETE methods. If you want your service to be accessible from Safari, you’ll need to allow your clients to simulate PUT and DELETE requests with overloaded POST. Second, Microsoft Internet Explorer caches successful responses indefinitely. This makes it look to the user like your resources haven’t changed, even when they have. The best way to get around this is to send proper ETag response headers with your representations, or to disable caching altogether with Cache-Control. You can use the XMLHttpRequest test suite to find out about more minor cross-browser quirks.

Because Ajax is a very important niche for JavaScript applications, some JavaScript libraries include wrappers for hiding the differences between browsers. I’m not going to cover these frameworks in detail, because they act more as standard libraries for JavaScript than tools for building web service clients. I will show how to make simple HTTP requests with two popular libraries, Prototype and Dojo. Another popular library, script.aculo.us, is based on Prototype.

Prototype

Prototype introduces three classes for making HTTP requests:

• Ajax.Request: a wrapper around XMLHttpRequest that takes care of cross-browser issues and can call different JavaScript functions on the request’s success or failure. The actual XMLHttpRequest object is available as the transport member of the Request object, so responseXML will be through request.transport.responseXML.

• Ajax.Updater: a subclass of Request that makes an HTTP request and inserts the response document into a specified element of the DOM.

• Ajax.PeriodicalUpdater, which makes the same HTTP request at intervals, refreshing a DOM element each time.

I’ve implemented the del.icio.us Ajax client in Prototype, and it was mostly the same as the client I showed you starting in Example 11-1. The code snippet below mostly replaces the code in Example 11-3 where the XMLHttpRequest constructor used to be. Note the new script tag, the use of request.transport instead of request, and the use of Prototype’s onFailure hook to signal a failure (such as an authorization failure) to the user.

Example 11-8. A portion of ajax-delicious-prototype.html

...
<script src="prototype.js"></script> 
<script type="text/javascript">
  ... 
  var request = new Ajax.Request("https://api.del.icio.us/v1/posts/recent", 
                                 {method: 'get', onSuccess: populateLinkList, 
                                 onFailure: reportFailure});
  function reportFailure() { 
    setMessage("An error occured: " + request.transport.status);
  }

  // Called when the HTTP request has completed. 
  function populateLinkList() { 
    setMessage("Request complete."); 
    if (netscape.security.PrivilegeManager.enablePrivilege) {
      netscape.security.PrivilegeManager.enablePrivilege("UniversalBrowserRead");
    }

    posts = request.transport.responseXML.getElementsByTagName("post");
    ...

In its quest to simplify XMLHttpRequest, Prototype hides some of the features. You can’t set request headers, or specify a username and password for basic HTTP auth. So even if you’re using Prototype, you might want to keep around a snippet of code like the one in Example 11-7. On the other hand, the Prototype implementation of the del.icio.us client doesn’t need the username and password text fields at all: it just needs a button. The end user’s browser will prompt her anyway for her del.icio.us username and password.

Dojo

The Dojo library provides a uniform API that not only hides the differences between browsers when it comes to XMLHttpRequest, it hides the difference between XMLHttpRequest and other ways of getting the browser to send an HTTP request. These “transports” include tricks that use HTML tags, such as JoD. All the variants on XMLHttpRequest are kept in the dojo.io.XMLHttp transport class. For all transports, the bind method is the one that makes the HTTP request.

As with Prototype, I’ve implemented the del.icio.us Ajax client with Dojo, and it’s mostly the same as the original, except for the section in Example 11-3 where the XMLHttpRequest constructor used to be. Example 11-9 shows the relevant portions of ajax-delicious-dojo.html.

Example 11-9. Some portions of ajax-delicious-dojo.html

...
  <script src="dojo/dojo.js"></script>
  <script type="text/javascript">
    ...
    dojo.require("dojo.io.*");
    dojo.io.bind({ url: "https://api.del.icio.us/v1/posts/recent", load:
                   populateLinkList, error: reportFailure });

    function reportFailure(type, error) { 
      setMessage("An error occured: " + error.message); 
    }

    // Called when the HTTP request has completed. 
    function populateLinkList(type, data, request) { 
      setMessage("Request complete.");
      if (netscape.security.PrivilegeManager.enablePrivilege) {
        netscape.security.PrivilegeManager.enablePrivilege("UniversalBrowserRead");
      }

      posts = request.responseXML.getElementsByTagName("post");
    ...

The error-handling function is passed a dojo.io.Error object with members called number and message. You can ignore the first argument: it’s always “error.” You can also ignore the first argument to the success-handling function (it’s always “load”). The second argument, called data above, is an interface to use Dojo’s DOM manipulation interface. If you want to use the XMLHttpRequest interface instead, you can ignore that argument too.

Subverting the Browser Security Model

That’s a provocative title but I stand by it. A web browser enforces a general rule that’s supposed to prevent it from using code found on domain A to make an HTTP request to domain B. I think this rule is too strict, so I’m going to show you two ways around it: request proxying and JoD. I’m also going to show how these tricks put you at risk by making you, the Ajax programmer, accept responsibility for what some foreign server does. These tricks deserve to be regarded as cheats, because they subvert rather than fulfill the web browser’s intentions. They often make the end user less secure than if his browser had simply allowed domain A’s JavaScript to make an HTTP request to domain B.

There is a secure method of getting permission to make foreign web service calls in your JavaScript applications, which is to ask for the permission by calling:

netscape.security.PrivilegeManager.enablePrivilege("UniversalBrowserRead");

(There’s also an insecure method, which is to have your users use Internet Explorer with the security settings turned way down.)

If your script is digitally signed, the client’s browser shows your credentials to the end user. The end user makes a decision whether or not to trust you, and if he trusts you he gives you permission to make the web service calls you need to make. This is similar to the technique I mentioned in Chapter 8, where an untrusted web service client was trying to gain the end user’s trust. The difference here is that the untrusted web service client is running inside the end user’s trusted web browser.

There are two problems with the secure method. The first is that, as you might have guessed from the name netscape.security.PrivilegeManager, it only works in Mozilla, Firefox, and Netscape-like browsers. The second is that it’s quite painful to actually get a signed script set up. Once you do get one set up, you find you’ve stored your HTML files in a signed Java archive file, and that your application is off the Web! Search engines won’t pick up your HTML pages, and you’ll only be able to address them through a weird jar: URI like jar:http://www.example.com/ajax-app.jar!/index.html.

And that’s the right solution. As you can tell, this is an immature field. Until recently, web services weren’t popular enough for people to seriously think about these problems. Though the hacks described below are potentially dangerous, their inventors meant no harm. They were motivated only by zeal for the enormous possibilities of in-browser web service clients. The challenge is to come up with ways of getting the same functionality without sacrificing security, adding too much complexity, or moving Ajax applications out of view of the Web. The W3C is working on this problem (see “Enabling Read Access for Web Resources” at http://www.w3.org/TR/access-control/.)

Although I’m focusing again on JavaScript applications, Java applets and Flash also run under security models that prevent them from sending data to foreign servers. The request proxying trick, described below, works for any kind of Ajax application, because it involves work on the server side. As its name implies, the JoD trick is JavaScript-specific.

Request Proxying

You’re running a site, example.com, serving up Ajax applications that try to make XMLHttpRequest requests against yahoo.com. Naturally your clients’ web browsers will complain. But what if they never made a request to yahoo.com? What if they made requests to example.com, which you handled by making your own, identical requests to yahoo.com without telling the client?

Welcome to the request proxy trick, well described in Yahoo’s document “Use a Web Proxy for Cross-Domain XMLHttpRequest Calls”. In this trick, you set aside part of the URI space on your server to simulate the URI space on some other server. When you get a request to a URI in that space, you send it along without alteration to the foreign server, and then pipe the response right back to the client. From the client’s point of view, it looks like you’re providing someone else’s web service. Really, you’re just filing the domain names off their HTTP responses and replacing them with your own.

If you’re using Apache and have mod_proxy installed, the simplest way to set up a proxy is in the Apache configuration. If you also have mod_ssl installed, you can enable SSLProxyEngine and proxy HTTPS requests. So long as you have mod_ssl installed, you can even proxy HTTPS requests from an HTTP server: perhaps http://example.com/service/ is proxied to https://service.com/. Of course, this destroys the security of the connection. Data is secure between the proxy and your site, but not between your site and the end user. If you do this you’d better tell the end user what you’re doing.

Let’s say you want to make the del.icio.us Ajax application, given above, work from your site at example.com. You can set up a proxy so that all URIs beneath https://example.com/apis/delicious/v1/ are transparently forwarded to https://api.del.icio.us/v1/. The simplest way to set up a proxy is with the ProxyPass directive, which maps part of your URI space onto a foreign site’s URI space (see Example 11-10).

Example 11-10. Apache configuration with ProxyPass

SSLProxyEngine On
ProxyRequests Off # Don’t act as an open proxy.
ProxyPass /apis/delicious/v1 https://api.del.icio.us/v1/

A more flexible solution is to use a rewrite rule with the [P] flag. This gives you the full power of regular expressions to map your URI-space onto the foreign site’s. Example 11-11 shows a rewrite rule version of the del.icio.us API proxy:

Example 11-11. Apache configuration with rewrite rules

SSLProxyEngine On 
ProxyRequests Off # Don’t act as an open proxy. 
RewriteEngine On RewriteRule ^apis/delicious/v1/(.*)$ https://api.del.icio.us/v1/$1 [P]

With a setup like one of those two, you can serve the Ajax application defined in Example 11-1 through Example 11-5 from your own domain, without triggering browser security warnings. All you have to do is change this (from Example 11-4):

request.open("GET", "https://api.del.icio.us/v1/posts/recent",
             true, username, password);

to this:

request.open("GET", "https://example.com/apis/delicious/v1/posts/recent", 
             true, username, password);

Most Apache installations don’t have mod_proxy installed, because an open HTTP proxy is a favorite tool for spammers and other lowlife who want to hide their tracks online. If your web server doesn’t have built-in proxy support, you can write a tiny web service that acts as a transparent proxy, and run it on your server. To proxy del.icio.us API requests, this web service might be rooted at apis/delicious/v1. It would pass any and all HTTP requests it received—HTTP headers and all—to the corresponding URI beneath https://api.del.icio.us/v1/. Yahoo! provides a sample proxy service, written in PHP, hardcoded to access the yahoo.com web services. You can model your own proxy service after that one.

Even when your proxy is properly configured, when it only proxies requests for a very small subset of the Web, there is danger for you and your end users. When you set up a proxy for Ajax clients, you’re taking responsibility in your users’ eyes for what the other web site does. The proxy trick sets you up as the fall guy for anything bad that happens on the other site. You’re pretending what the other site is serving comes from you. If the web service crashes, cheats the end user, or misuses his personal data, guess what: it looks like you did those things. Remember, in an Ajax application the end user only sees your GUI interface. He doesn’t necessarily know his browser is making HTTP requests in the background, and he certainly doesn’t know that his requests to your domain are being proxied to another domain. If his web browser knew that was going on, it would step in and put a stop to it.

The proxy trick also sets you up as the fall guy for the requests your clients make. Your clients can make any web service request they like and it’ll look like you’re the cause. Depending on the nature of the web service this may cause you embarrassment or legal exposure. This is less of a problem for web services that require separate authorization.

JavaScript on Demand

It’s rare for a human being to demand JavaScript, except in certain design meetings, but it’s not uncommon among web browsers. The basis of this trick is that the HTML script tag doesn’t have to contain hardcoded JavaScript code. It might just have a src attribute that references code at another URI. A web browser knows, when it encounters a script tag, to load the URI in the src attribute and run its contents as code. We saw this in Example 11-6, the JSON example that does a Yahoo! image search for pictures of elephants.

The src attribute is traditionally used like C’s #include or Ruby’s require: to load in a JavaScript library from another URI. Example 11-12, reprinted from Chapter 2, shows this.

Example 11-12. Including a JavaScript file by reference

<!-- In a real application, you would save json.js locally
     instead of fetching it from json.org every time. --> 
<script type="text/javascript" src="http://www.json.org/json.js"></script>

As you can see, the URI in the src attribute doesn’t have to be on the same server as the original HTML file. The browser security model doesn’t consider this insecure because... well, near as I can figure, because the src attribute was already in wide use before anyone started seriously thinking about the security implications.

Now cast your mind back to the elephant example in Example 11-6. It includes this line:

<script type="text/javascript"
src="http://api.search.yahoo.com/ImageSearchService/V1/imageSearch
?appid=restbook&query=baby+elephant&output=json&callback=formatImages" />

That big long URI doesn’t resolve to a standalone JavaScript library, the way http://www.json.org/json.js does. If you visit it in your web browser you’ll see that URI’s representation is a custom-generated bit of JavaScript. In its developer documentation, Yahoo! promises that the representation of a resource like this one is a snippet of JavaScript code. Specifically, a snippet of JavaScript code that passes a data structure as the only argument into a callback function named in the URI (here, it’s formatImages). The resulting JavaScript representation looks something like this:

formatImage({"ResultSet":{"totalResultsAvailable":"27170",...}})

When the client loads the HTML page, it fetches that URI and run its body as JavaScript, incidentally calling the formatImage method. Great for our application; not so great for the web browser. From a security perspective this is just like JavaScript code that uses XMLHttpRequest to get data from the Yahoo! web service, and then calls formatImage on the result. It bypasses the browser security model by making the HTTP request happen as a side effect of the browser’s handling of an HTML tag.

JoD switches the traditional roles of a script embedded in an HTML page and a script included via <script src="...">. Your web browser requests a web service URI, thinking it’s just a JavaScript library that application code in the HTML page will eventually call. But the library function is the one defined locally (it’s formatImage), and the application code that calls that function is coming from a foreign site.

If you specify no callback in the URI when calling the Yahoo! web service, you get a “JavaScript” file containing nothing but a JSON data structure. Including this file in a script tag won’t do anything, but you can fetch it with a programmable HTTP client (like XMLHttpRequest, or the Ruby client from way back in Example 2-15) and parse it as data:

{"ResultSet":{"totalResultsAvailable":"27170",...}}

Dynamically writing the script tag

The only example of JoD I’ve given so far has a hardcoded script tag. The URI to the web service resource is fixed in stone, and if the end user wants to see baby penguins instead of baby elephants he’s just out of luck.

But one of the things you can do with JavaScript is add brand new tags to the DOM object representing the current HTML page. And script is just another HTML tag. You can use JavaScript to write a customized script tag into the document, and get the browser to load the URI mentioned in its src attribute as a side effect of the script processing. The browser allows this even if the src URI points to a foreign domain. That means you can use JavaScript to make requests to any URI that serves more JavaScript, and run it.

This works, but it’s a hack on top of a hack, and a security problem on top of a security problem. In fact, from a security perspective this is worse than using XMLHttpRequest to get data from a foreign site. The worst XMLHttpRequest will do is make an HTTP request and parse some XML into a tree-like data structure. With JoD you make an HTTP request and run previously unseen JavaScript code as though it was part of your original program.

You and your end user are completely at the mercy of the service you’re calling. Instead of JavaScript that does what you want, a malicious web service might decide to serve JavaScript that steals whatever cookies your domain set for this user. It might serve code that runs code as promised but also creates pop-up windows full of obnoxious ads. It might do anything at all. And since Ajax hides the HTTP request-response cycle from the end user, it looks like your site is responsible!

Now, maybe you trust a brand-name site like Yahoo! (unless it gets cracked), but you probably don’t trust Mallory’s House of Web Services. And that in itself is a problem. One of the nice things about the Web is that you can safely link to Mallory even if you don’t trust her, don’t have her permission, and think she’s wrong about everything. A normal web service client can make calls to Mallory’s web service, and examine the representation before acting on it in case she tries any trickery. But when the client is serving executable code, and the web service requested it through a hack that runs the code automatically, you’re reduced to operating on blind trust.

JoD is not only sketchy from a security standpoint, it’s a lousy tactic from a REST standpoint, because it forces you to use a crippled client. XMLHttpRequest supports all the features of HTTP, but with JoD you can only make GET requests. You can’t send request headers, see the response code or headers, or handle representation formats other than JavaScript code. Any representation you receive is immediately executed as JavaScript.

The underlying technique, of referencing a new object in a src attribute, is safer when you use it to grab resources other than custom-generated JavaScript. script isn’t the only HTML tag that makes the browser load a representation. Other useful tags include img and frame. Google Maps uses img tags rather than XMLHttpRequest calls to fetch its map tile images. Google’s JavaScript code doesn’t make the HTTP requests. It just creates the img tags and lets the browser make requests for the images as a side effect.

Library support

Jason Levitt has written a JavaScript class called JSONscriptRequest that makes JoD easy (http://www.xml.com/pub/a/2005/12/21/json-dynamic-script-tag.html). This class works sort of like XMLHttpRequest, except it supports fewer of HTTP’s features, and instead of expecting the server to send an XML representation, it expects a snippet of JavaScript.

Example 11-13 shows a dynamic implementation of the image search Ajax application. The first part should be familiar if you’ve looked at the other Ajax applications in this chapter.

Example 11-13. Dynamic Yahoo! image search Ajax application

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"
"http://www.w3.org/TR/REC-html40/transitional.dtd">
<!--yahoo-ajax-dynamic.html-->
<!--An Ajax application that uses dynamic SCRIPT tags to interactively
    fetch data from Yahoo! Web Services and call predefined functions
    on that data.-->

<html>
<head><title>Javascript Yahoo! JSON - Dynamic</title></head>
<body>
<h1>Javascript Yahoo! JSON example with dynamic SCRIPT tags</h1>

<form onsubmit="callYahoo(); return false;">
 What would you like to see? <input id="query" type="text" /><br />
 <input type="submit" value="Fetch pictures from Yahoo! Image Search"/>
</form>

<div id="images">
</div>

<script type="text/javascript">
function formatImages(result)
{
  // First clear out any old images.
  var images = document.getElementById("images");
  while (images.firstChild) 
  {
    images.removeChild(images.firstChild);
  }

  items = result["ResultSet"]["Result"];
  for (var i = 0; i < items.length; i++)
  {
    image = items[i];

    // Create a link
    var link = document.createElement("a");
    link.setAttribute("href", image["ClickUrl"]);

    // Put a thumbnail image in the link.
    var img = document.createElement("img");
    var thumbnail = image["Thumbnail"];
    img.setAttribute("src", thumbnail["Url"]);
    img.setAttribute("width", thumbnail["Width"]);
    img.setAttribute("height", thumbnail["Height"]);
    img.setAttribute("title", image["Title"])
    link.appendChild(img);    
    images.appendChild(link);
  }
}
</script>

Here’s where this application diverges from others. I include Jason Levitt’s jsr_class.js file, and then define the callYahoo function to use it (see Example 11-14). This is the function triggered when the end user clicks the submit button in the HTML form above.

Example 11-14. Dynamic Yahoo! image search Ajax application continued

<script type="text/javascript" src="jsr_class.js"></script>

<script type="text/javascript">
function callYahoo()
{
  var query = document.getElementById("query").value;
  var uri = "http://api.search.yahoo.com/ImageSearchService/V1/imageSearch" +
            "?query=" + escape(query) + 
	    "&appid=restbook&output=json&callback=formatImages";
  alert(uri);
  var request = new JSONscriptRequest(uri);
  request.buildScriptTag();
  request.addScriptTag();
}
</script>

</body>
</html>

To make a web service request I pass the URI of a resource into a JSONscriptRequest object. The addScriptTag method sticks a new script tag into the DOM. When the browser processes its new tag, it makes a GET request to the foreign URI, and runs the JavaScript that’s served as a representation. I specified “callback=formatImages” in the URI’s query string, so Yahoo! serves some JavaScript that calls my formatImages function on a complex data structure. You can serve this Ajax application from anywhere, and use it to search for anything on Yahoo!’s image search, without triggering any browser warnings.

The Dojo library makes the script trick easy by providing a dojo.io.SrcScript transport class that uses it. It also provides a dojo.io.IframeIO class which uses a similar trick involving the iframe tag. This trick also requires cooperation from the server, but it does have the advantage that it doesn’t automatically execute the response document as code.