Learning jQuery Deferreds

Chapter 4. More Time in the Mental Gymnasium

The recipes in the previous chapter are useful in a practical sense, yet we also wrote them to help you exercise the mental muscles needed for thinking about deferreds. Although conceptually simple, deferreds and promises can be mind-bending and yet a pleasure to ponder.

It may take some time to digest everything, but deferreds are worth exploring further (both more deeply and broadly), and that’s the purpose of this chapter.

Do You Really Understand jQuery Deferreds?

To warm up, let’s see if you really understand jQuery deferreds. If you do, you should be able to answer the following questions with little hesitation:

• Explain the differences between done, then, and $.when.

• Given a promise variable, what’s the difference between these two code fragments?

  promise.then(doneFunc, errorFunc)

  promise.done(doneFunc).fail(errorFunc)

• What happens if you return a value from a done callback? What should you probably be doing instead?
• What happens if you call then on a promise, but don’t provide any arguments?

• Take another look at this code fragment from Familiar Promises:

  $.when(
      $.ajax('http://google.com').then(function(){
          return $('#label_1').animate({ opacity: 0.25 }, 100);
      }),
      $.ajax('http://yahoo.com').then(function(){
          return $('#label_2').animate({ opacity: 0.75 }, 200);
      })
  ).then(
      function(){
          // Both URLs have been fetched and both animations have completed.
      }
  );

  In order of difficulty:

  • .promise() is not called on the result of the calls to animate. Why is that not needed?
  • What would be the effect of removing the two return keywords (i.e., call animate as above, just don’t return its result)?
  • How can you arrange to get the results of the $.ajax calls passed as arguments to the final function?

If you’re unsure of the answers to these questions, rereading Chapter 2 will probably help, as will writing small pieces of code to have a closer look. Make sure you understand all the points listed in Deferred Dynamics.

Promises/A+

Feeling warmed up? Let’s stretch our mental muscles a bit further by learning about a standard for promises.

We have mentioned the Promises/A+ proposal at several points in the book. It’s a library-agnostic clarification of the expected behavior of promises. There is also the original Promises/A (upon which jQuery promises is based) and a newer Promises/B proposal. Promises/A+ is the most popular current proposal. A+ is a clarification, extension, and simplification of the original Promises/A.

It is important that you know about Promises/A+, because most of the popular promise libraries conform to the A+ proposal. Its main (although not exclusive) focus is the means of interacting with promises via the then method. It’s a short, well-written document released in the public domain. It takes only minutes to read. We’ve included a copy in Appendix B.

To summarize, Promises/A+ states that the then method should return a new promise. In so doing, it facilitates the chaining of callbacks via further calls via then on subsequently returned promises. It makes explicit the expected behavior of both the onFulfilled and onRejected handlers and how a promise may transition between pending, fulfilled, and rejected states.

Rather than repeat an already well-written and compact document, we feel it necessary to explain the most important and subtle way it differs from the apparently similar jQuery deferred API.

It boils down to error handling.

In the Promises/A+ world, an exception raised in a callback passed to then is handled automatically. The call to then will return a promise in a rejected state, with the exception set as the reason attribute of the new promise.

However, jQuery promises don’t do this. Their then doesn’t return a rejected promise. Instead, exceptions are not caught. An exception in a function passed to then will break any neatly composed chain of then handlers. An exception in an always, done, or fail callback will terminate that callback but also prevent the invocation of the rest of the callbacks added to the promise. In all these cases, the exception will propagate up the call stack. If you’re running a node.js server, an uncaught exception will cause your server to exit, which is clearly a serious problem.

Therefore, when using jQuery deferreds you’ll need to be more careful about catching regular exceptions in your callback functions and finding ways to fail when the original call stack context (in which you arranged for the callbacks to be invoked) has long gone. Promises/A+ lets failures flow back via rejected promises.

This difference and its impact is elegantly discussed in Dominic Denicola’s article “You’re Missing the Point of Promises”, in which he introduces a test suite for Promises/A+ compliance. The slides of Dominic’s “Promises, Promises” presentation are also worth reading, particularly the description of the correspondence between asynchronous Promises/A+ code and regular synchronous code.

There are a number of other differences between jQuery deferreds and Promises/A+. For example, the jQuery version of then allows you to pass a progress function, but Promises/A+ does not require this (though Promises/A did). The Promises/A+ specification makes no mention of methods such as done, fail, etc., relying only on then. You will discover other differences—including runtime ones such as speed and memory usage—if you compare the implementations yourself or spend some time online reading comparison tests and benchmarks.

There are many packages for JavaScript (and some in other languages too) that conform to the Promises/A+ standard. The Promises/A+ website contains an up-to-date list.

Promises Are First-Class Objects for Function Calls

In JavaScript, we describe functions as first-class objects. It’s why we’re able to do all sorts of interesting, empowering, and downright fun things with functions: return a function from a function, pass a function as an argument into yet another function, and create anonymous functions on the fly. It’s hard to imagine JavaScript without such a powerful core feature.

With that in mind, try this exercise in a change of perspective: deferreds and promises are first-class objects for function calls.

If you think about it, they give us a way to take a function call (or the handling of an event) and pass it around: we can return a promise from a function, pass a promise as an argument into a function, store it in a data structure, and, obviously, arrange how to process the result (or error) of the function call independently of the function call itself.

To be a bit more abstract: a promise is a time-independent reification of a function call.

A promise is time-independent because, as we saw in Memoization, you don’t have to worry whether the underlying function call has already completed, is currently in progress, or has yet to run. There may even be no way to find out. The point is that you don’t have to care and you shouldn’t care.

Like many others, we described deferreds and promises in Chapter 1 as representing a result that may not be available yet. Describing promises as a reification of a function call takes the time factor completely out of the picture. If you’ve managed to switch your perspective, this way of looking at deferreds is a lot simpler to explain and understand.

Asynchronous Data Structures

Although deferreds are useful for handling familiar tasks such as network calls, user interface events, and talking to databases and other storage, nothing about deferreds is specific to those things. Because deferreds are not tied to any particular usage, you can use them to build asynchronous implementations of other things that might seem surprising.

For example, consider asynchronous data structures. A Deferred Queue gives a good example of this line of thinking. Let’s revisit the deferred queue code, slightly simplified:

var waiting = [], queue = [];

function get(){
    var deferred = $.Deferred();
    if (queue.length){
        deferred.resolve(queue.shift());
    }
    else {
        waiting.push(deferred);
    }
    return deferred.promise();
}

function put(item){
    if (waiting.length){
        waiting.shift().resolve(item);
    }
    else {
        queue.push(item);
    }
}

As we suggested in that example, think about code for a regular synchronous queue. What happens if you call get on a regular queue that’s empty? Almost certainly one of two things: you’ll get some kind of QueueEmpty error, or else your code will block until some other code puts something into the queue. That is, you either get a synchronous error or you get a synchronous nonempty response.

In contrast, look at the get function above. A new deferred is always made. If the queue is nonempty, the deferred is resolved and its promise is returned. If the queue is empty, the deferred is added to waiting and its promise is returned. Code calling get on an empty queue doesn’t get an error and it doesn’t block; it always receives a result immediately. How can you get a result from an empty queue? Easy! The result is a promise.

In either case, the caller just attaches callbacks to the returned promise, and goes on its merry way. It doesn’t need to know if the promise has already fired or when it will fire. It just needs to know how to process a queue item.

In effect, we’ve built an asynchronous data structure. We find that concept very attractive, and thinking in those terms helped us to see part of the reason why deferreds are so great. Deferreds are not specifically asynchronous or synchronous: they’re so simple they don’t even care, and as a consumer of promises, your code doesn’t have to either. That’s all remarkably elegant.

Bonus challenge: imagine you have a producer and a consumer wanting to exchange a dictionary data structure. The producer will produce values in a random key order, while the consumer requests keys in some other random order. Implement an asynchronous dictionary using deferreds in the style of the deferred queue above.

Advantages of Deferreds

It is typical to first gain an appreciation of deferreds (in general, not just the jQuery flavor) by using them in the context of a specific problem. But deferreds are useful and powerful in multiple ways. Your understanding will be stronger if you consider them all at once. Here’s a summary.

Deferreds provide a formal first-class place to hold a future result. You can pass a deferred to other pieces of code. This allows multiple independent pieces of code to act on its result. The first-class object gives your program a data structure that captures the fact that a function has been called. That data structure can be used from the moment the function has been called until it has returned. As we saw in Memoization, this can be very useful. Because the deferred is time-independent, you also don’t need to worry about events firing before you’ve added handlers for them—you can add event handlers (callbacks) after the fact.

Without deferreds, you get to attach just one callback handler to events. If other code needs to also handle the result of the event, calls to those other functions need to be placed in the original callback. This leads to a mess: callback handlers that do not serve a single purpose and that are harder to understand and test.

Deferreds make great building blocks. We’ve repeatedly seen how simple it is to solve problems by creating an additional deferred and hooking up things just as we need them. It is a testament to the elegance and power of deferreds that this can be done so cleanly and easily. Each time you do it, you just end up with another deferred, which allows for additional processing layers to be added in an identical way, if needed.

Deferreds make it simple(r) to work with multiple asynchronous calls. In Food for Thought and when, we looked at how awkward it can be to work with more than one asynchronous function call. The examples we considered were trivial. When building non-toy systems, things get much more complicated. Deferreds can’t make a complicated situation less complex, but they will allow you to write code to handle it in a cleaner way. Code that doesn’t use deferreds does not have an object to pass around that represents the handling of an event (or the future result of a function call). This forces us to set up complex event handling using the lexical structure of our code—callbacks in callbacks in callbacks, etc. The logical structure of one’s code can become extremely awkward, as can error handling. Complex programs using deferreds can also be very difficult to understand and arguably harder to follow logically, but there is more scope for cleaner separation of code and nicer error handling (at least with Promises/A+).

Deferreds are very useful for setting up processing of request responses in a free-form messaging environment (where both sides can send messages at any time). We saw examples of this in Communicating with a Web Worker and Using Web Sockets. An important advantage of this approach is that code that makes an asynchronous call can arrange to process the response without losing context. When you make the call, your code has access to its current scope, local variables, relevant UI components, etc. In free-form messaging, when the reply arrives, you often need to reconstruct the necessary parts of the original context in order to process the result. With deferreds, you can set this up on the spot by adding handlers to the deferred, which is extremely convenient in practice.

Difficulties with Deferreds

There are a few ways in which deferreds can be problematic.

Occasionally you will build a complex structure of dependent deferreds and one of them will fail to fire. For example, a network request is somehow lost or never returns. This jams up the whole system: a deferred never fires and all downstream deferreds fail to resolve. This kind of problem has to be dealt with by programmers in any case; it’s not specific to deferreds, but there can be a feeling of a certain brittle dependency. We’ve explored several ways in which these problems can be addressed: timing out promises (in when2: An Improved jQuery.when and Timing Out Promises), firing promises ourselves (in Controlling Your Own Destiny), and deactivating promises entirely (in Deactivating a Promise).

A second issue, which is more serious and which may preclude the use of deferreds at all, is that deferreds keep the result of their computation around. If it happens that you only need a result once, introducing deferreds will add to memory load and may make garbage collection more difficult for the system.

Deferreds are often viewed as resulting in code that is difficult to read and understand. This is subjective, of course. It is also the case that some programming situations are inherently complicated, and that complexity cannot really be mitigated. Nevertheless, making sense of deferred code can sometimes be more difficult than it is with more traditional JavaScript callback nesting. This is especially the case with very simple event handling. Once asynchronous code logic gets more than a couple of levels deep, we’ve found deferred code to be cleaner due to its separation and having first-class objects to pass around and store in data structures. There’s more to deferreds, though, than potential gains (or losses) in code organization and readability, as just discussed above. In our experience, any loss in code readability is more than made up for by gains in other ways.

There is sometimes the feeling of a lock-in effect when using deferreds. Once you start calling and writing functions that use deferreds, you’ll invariably end up doing more of it. That’s also the case with nested callback handling; it’s just a different style of getting things done. Fortunately, JavaScript is inherently event-based, so it is usually trivial to write deferred-based wrappers for functions you want to call that do not follow this style. It is also usually very easy to mix deferred-using code with regular code.

Finally, if you’re building a team, use of deferreds may have a human resources impact: you’ll likely end up looking for programmers who are already familiar with deferreds. Programmers who are unfamiliar with deferreds are definitely still in the majority. We hope to help change that!

Further Reading

James Coglan has written three excellent articles on promises: “Promises are the monad of asynchronous programming”, “Callbacks are imperative, promises are functional: Node’s biggest missed opportunity”, and “Callbacks, promises, and simplicity”. The articles may not be easy to understand on a first reading. You might find it easiest to start with the second. They are well worth rereading and thinking about until you do get it. Don’t give up—it’s important! For balance, you should also read Drew Crawford’s response, “Broken Promises”, which describes issues he ran into using promises in the highly memory-constrained iOS environment, and, more generally, why trying to use promises to solve every asynchronous flow of control problem can lead to difficulties.

Many people have written about jQuery deferreds online, and some of the articles are excellent. There was a flurry of writing when deferreds were introduced to jQuery in version 1.5 on January 31, 2011. As mentioned in Changes in the jQuery Deferred API, subsequent changes in the deferred API mean you need to be a little careful when reading. For example, most recent online articles still discuss the pipe function, which was deprecated in version 1.8 (August 9, 2012). But don’t worry too much: even if articles are slightly out of date with respect to the API, you can still learn a lot from them and may benefit from other people’s perspectives and explanations. Here’s a small selection, focused mainly on jQuery deferreds, that you may find useful:

• “Creating Responsive Applications Using jQuery Deferred and Promises”
• “Redemption from Callback Hell”
• “jQuery.Deferred is the most important client-side tool you have”
• “Using Deferreds in jQuery 1.5”
• “Asynchronous Programming in JavaScript with ‘Promises’”

You might also be interested to read how Twitter transitioned their architecture to use “services” that return futures (i.e., deferreds) via RPC using Finagle.

Deferreds have been around since 1976, under many names and guises, and in many programming languages. For some history, see the Wikipedia article on “Futures and promises”.