O'Reilly logo

Clojure Programming by Brian Carper, Christophe Grand, Chas Emerick

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Chapter 4. Concurrency and Parallelism

Writing multithreaded programs is one of the most difficult tasks many programmers will face. They are difficult to reason about, and often exhibit nondeterministic behavior: a typical program that utilizes concurrency facilities will sometimes yield different results given the same inputs, a result of ill-defined execution order that can additionally produce race conditions and deadlocks. Some of these conditions are hard to detect, and none of them are easy to debug.

Most languages give us paltry few resources to cope with the cognitive burden of concurrency. Threads and locks, in all their forms, are often the only real tools at our disposal, and we are often victims of how difficult they are to use properly and efficiently. In which order should locks be acquired and released? Does a reader have to acquire a lock to read a value another thread might be writing to? How can multithreaded programs that rely upon locks be comprehensively tested? Complexity spirals out of control in a hurry; meanwhile, you are left debugging a race condition that only occurs in production or a deadlock that happens on this machine, but not that one.

Considering how low-level they are, the continued reliance upon threads, locks, and pale derivatives as the sole “user-facing” solution to the varied complexities of concurrency is a remarkable contrast to the never-ending stampede of activity seen over the years in developing more effective and less error-prone abstractions. Clojure’s response to this has many facets:

  1. As we discussed in Chapter 2, minimize the amount of mutable state in your programs, with the help of immutable values and collections with reliable semantics and efficient operations.

  2. When you do need to manage changing state over time and in conjunction with concurrent threads of execution, isolate that state and constrain the ways in which that state can be changed. This is the basis of Clojure’s reference types, which we’ll discuss shortly.

  3. When you absolutely have no other choice—and are willing to shrug off the benefits of the semantic guarantees of Clojure’s reference types—make it straightforward to drop back to bare locks, threads, and the high-quality concurrency APIs provided by Java.

Clojure provides no silver bullet that makes concurrent programming instantly trivial, but it does provide some novel and now battle-tested tools to makes it far more tractable and reliable.

Shifting Computation Through Time and Space

Clojure provides a number of entities—delays, futures, and promises—that encapsulate discrete use cases for controlling when and how computations are performed. While only futures are solely concerned with concurrency, they are all often used to help implement specific concurrent semantics and mechanics.


A delay is a construct that suspends some body of code, evaluating it only upon demand, when it is dereferenced:

(def d (delay (println "Running...")
;= #'user/d
(deref d)
; Running...
;= :done!


The deref abstraction is defined by Clojure’s clojure.lang.IDeref interface; any type that implements it acts as a container for a value. It may be dereferenced, either via deref, or the corresponding reader syntax, @.[123] Many Clojure entities are dereferenceable, including delays, futures, promises, and all reference types, atoms, refs, agents, and vars. We talk about them all in this chapter.

You can certainly accomplish the same sort of thing just by using functions:

(def a-fn (fn []
            (println "Running...")
;= #'user/a-fn
; Running...
;= :done!

However, delays provide a couple of compelling advantages.

Delays only evaluate their body of code once, caching the return value. Thus, subsequent accesses using deref will return instantly, and not reevaluate that code:[124]

;= :done!

A corollary to this is that multiple threads can safely attempt to dereference a delay for the first time; all of them will block until the delay’s code is evaluated (only once!), and a value is available.

When you may want to provide a value that contains some expensive-to-produce or optional data, you can use delays as useful (if crude) optimization mechanisms, where the end “user” of the value can opt into the costs associated with that data.

Example 4-1. Offering opt-in computation with a delay

(defn get-document
  ; ... do some work to retrieve the identified document's metadata ...
  {:url "http://www.mozilla.org/about/manifesto.en.html"
   :title "The Mozilla Manifesto"
   :mime "text/html"
   :content (delay (slurp "http://www.mozilla.org/about/manifesto.en.html"))}) 1
;= #'user/get-document
(def d (get-document "some-id"))
;= #'user/d
;= {:url "http://www.mozilla.org/about/manifesto.en.html",
;=  :title "The Mozilla Manifesto",
;=  :mime "text/html",
;=  :content #<Delay@2efb541d: :pending>}                                      2

We can use delay to cheaply suspend some potentially costly code or optional data.


That delay’s code will remain unevaluated until we (or our code’s caller) opt to dereference its value.

Some parts of our program may be perfectly satisfied with the metadata associated with a document and not require its content at all, and so can avoid the costs associated with retrieving that content. On the other hand, other parts of our application may absolutely require the content, and still others might make use of it if it is already available. This latter use case is made possible with realized?, which polls a delay to see if its value has been materialized yet:

(realized? (:content d))
;= false
@(:content d)
;= "<!DOCTYPE html><html>..."
(realized? (:content d))
;= true


Note that realized? may also be used with futures, promises, and lazy sequences.

realized? allows you to immediately use data provided by a delay that has already been dereferenced, but perhaps opt out of forcing the evaluation of a delay if you know that doing so will be too expensive an operation than you’re willing to allow at that point in time and can do without its eventual value.


Before getting to more sophisticated topics like reference types, Clojure programmers often start off asking, “How do I start a new thread and run some code in it?” Now, you can use the JVM’s native threads if you have to (see Using Java’s Concurrency Primitives), but Clojure provides a kinder, gentler option in futures.

A Clojure future evaluates a body of code in another thread:[125]

(def long-calculation (future (apply + (range 1e8))))
;= #'user/long-calculation

future returns immediately, allowing the current thread of execution (such as your REPL) to carry on. The result of evaluation will be retained by the future, which you can obtain by dereferencing it:

;= 4999999950000000

Just like a delay, dereferencing a future will block if the code it is evaluating has not completed yet; thus, this expression will block the REPL for five seconds before returning:

@(future (Thread/sleep 5000) :done!)
;= :done!

Also like delays, futures retain the value their body of code evaluated to, so subsequent accesses via deref will return that value immediately.

Unlike delays, you can provide a timeout and a “timeout value” when dereferencing a future, the latter being what deref will return if the specified timeout is reached:[126]

(deref (future (Thread/sleep 5000) :done!)
;= :impatient!

Futures are often used as a device to simplify the usage of APIs that perform some concurrent aspect to their operation. For example, say we knew that all users of the get-document function from Example 4-1 would need the :content value. Our first impulse might be to synchronously retrieve the document’s :content within the scope of the get-document call, but this would make every caller wait until that content is retrieved fully, even if the caller doesn’t need the content immediately. Instead, we can use a future for the value of :content; this starts the retrieval of the content in another thread right away, allowing the caller to get back to work without blocking on that I/O. When the :content value is later dereferenced for use, it is likely to block for less time (if any), since the content retrieval had already been in motion.

(defn get-document
  ; ... do some work to retrieve the identified document's metadata ...
  {:url "http://www.mozilla.org/about/manifesto.en.html"
   :title "The Mozilla Manifesto"
   :mime "text/html"
   :content (future (slurp "http://www.mozilla.org/about/manifesto.en.html"))}) 1

The only change from Example 4-1 is replacing delay with future.

This requires no change on the part of clients (since they continue to be interested only in dereferencing the value of :content), but if callers are likely to always require that data, this small change can prove to be a significant improvement in throughput.

Futures carry a couple of advantages compared to starting up a native thread to run some code:

  1. Clojure futures are evaluated within a thread pool that is shared with potentially blocking agent actions (which we discuss in Agents). This pooling of resources can make futures more efficient than creating native threads as needed.

  2. Using future is much more concise than setting up and starting a native thread.

  3. Clojure futures (the value returned by future) are instances of java.util.concurrent.Future, which can make it easier to interoperate with Java APIs that expect them.


Promises share many of the mechanics of delays and futures: a promise may be dereferenced with an optional timeout, dereferencing a promise will block until it has a value to provide, and a promise will only ever have one value. However, promises are distinct from delays and futures insofar as they are not created with any code or function that will eventually define its value:

(def p (promise))
;= #'user/p

promise is initially a barren container; at some later point in time, the promise may be fulfilled by having a value delivered to it:

(realized? p)
;= false
(deliver p 42)
;= #<core$promise$reify__1707@3f0ba812: 42>
(realized? p)
;= true
;= 42

Thus, a promise is similar to a one-time, single-value pipe: data is inserted at one end via deliver and retrieved at the other end by deref. Such things are sometimes called dataflow variables and are the building blocks of declarative concurrency. This is a strategy where relationships between concurrent processes are explicitly defined such that derivative results are calculated on demand as soon as their inputs are available, leading to deterministic behavior. A simple example would involve three promises:

(def a (promise))
(def b (promise))
(def c (promise))

We can specify how these promises are related by creating a future that uses (yet to be delivered) values from some of the promises in order to calculate the value to be delivered to another:

  (deliver c (+ @a @b))
  (println "Delivery complete!"))

In this case, the value of c will not be delivered until both a and b are available (i.e., realized?); until that time, the future that will deliver the value to c will block on dereferencing a and b. Note that attempting to dereference c (without a timeout) with the promises in this state will block your REPL thread indefinitely.

In most cases of dataflow programming, other threads will be at work doing whatever computation that will eventually result in the delivery of values to a and b. We can short-circuit the process by delivering values from the REPL;[127] as soon as both a and b have values, the future will unblock on dereferencing them and will be able to deliver the final value to c:

(deliver a 15)
;= #<core$promise$reify__5727@56278e83: 15>
(deliver b 16)
; Delivery complete!
;= #<core$promise$reify__5727@47ef7de4: 16>
;= 31

Promises don’t detect cyclic dependencies

This means that (deliver p @p), where p is a promise, will block indefinitely.

However, such blocked promises are not locked down, and the situation can be resolved:

(def a (promise))
(def b (promise))
(future (deliver a @b))  1
(future (deliver b @a))
(realized? a)            2
;= false
(realized? b)
;= false
(deliver a 42)           3
;= #<core$promise$reify__5727@6156f1b0: 42>
;= 42
;= 42

Futures are used there to not block the REPL.


a and b are not delivered yet.


Delivering a allows the blocked deliveries to resume—obviously (deliver a @b) is going to fail (to return nil) but (deliver b @a) proceeds happily.

An immediately practical application of promises is in easily making callback-based APIs synchronous. Say you have a function that takes another function as a callback:

(defn call-service
  [arg1 arg2 callback-fn]
  ; ...perform service call, eventually invoking callback-fn with results...
  (future (callback-fn (+ arg1 arg2) (- arg1 arg2))))

Using this function’s results in a synchronous body of code requires providing a callback, and then using any number of different (relatively unpleasant) techniques to wait for the callback to be invoked with the results. Alternatively, you can write a simple wrapper on top of the asynchronous, callback-based API that uses a promise’s blocking behavior on deref to enforce the synchronous semantics for you. Assuming for the moment that all of the asynchronous functions you’re interested in take the callback as their last argument, this can be implemented as a general-purpose higher-order function:

(defn sync-fn
  (fn [& args]
    (let [result (promise)]
      (apply async-fn (conj (vec args) #(deliver result %&)))

((sync-fn call-service) 8 7)
;= (15 1)

[123] @foo is nearly always preferred to (deref foo), except when using deref with higher-order functions (to, for example, dereference all of the delays in a sequence) or using deref’s timeout feature, available only with promises and futures.

[124] And, therefore, not cause any potential side effects associated with the code provided to create the delay.

[125] future-call is also available if you happen to have a zero-argument function you’d like to have called in another thread.

[126] This option is not available when using the @ reader sugar.

[127] Which, technically, is in another thread!

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required