D3 for the Impatient

Chapter 1. Introduction

D3.js (or just D3, for Data-Driven Documents) is a JavaScript library for manipulating the Document Object Model (DOM) tree in order to represent information visually. It has become a de facto standard for information graphics on the web.

Its popularity notwithstanding, D3 has a reputation for having a steep learning curve. I don’t think this is because D3 is complicated (it is not), and not even because it has a large API (it does, but the API is well structured and extremely well designed). Instead, I believe that many of the difficulties experienced by new users are due to incorrect assumptions. Because D3 is used to create some impressive graphics, it is easy—even natural—to regard it as a “graphics library” that facilitates handling of graphical primitives and provides high-level support for common plot types. Approaching D3 with this expectation, the new user is unpleasantly surprised at the verbosity required to set something as elementary as the color of an element—and what’s that “selection” business anyway? Why can’t I simply get a canvas-like element and be done with it?

The mistake here is that D3 is not a graphics library: instead, it is a JavaScript library to manipulate the DOM tree! Its basic building blocks are not circles and rectangles, but nodes and DOM elements; and the typical activity does not involve the painting of a graphical shape on a canvas, but the styling of an element through attributes. The “current location” is not so much given through an xy-coordinate on a canvas, but through a selection of nodes in a DOM tree.

This leads to what I believe is the second major challenge for many new users: D3 is a web technology, and relies on a collection of other web technologies; the DOM API and event model, Cascading Style Sheet (CSS) selectors and properties, the JavaScript object model, and of course Scalable Vector Graphics (SVG). In many cases, D3 provides only a relatively thin layer around these technologies, and its own design frequently reflects the underlying APIs. This makes for a large and somewhat heterogeneous environment. If you are already familiar with the complete stack of modern web technologies that is known as HTML5, you will feel right at home, but if you are unfamiliar with them, then the lack of a distinct, unified abstraction layer can be very confusing.

Thankfully, you don’t need to study all these underlying technologies in depth: D3 does make them easier to use, and it provides a considerable amount of unification and abstraction on top of them. The only area where it is definitely not enough to “wing it” is SVG. You must have an adequate understanding of SVG—not only of its representational elements, but also of the structural elements that control how information is organized within a graph. I tried to put together the necessary information in Appendix B. If you are not familiar with SVG, then I suggest that you work through this appendix before perusing the rest of this book—you will be glad you did!

Whom This Book Is For

This book is intended for programmers and scientists who want to add D3 to their toolbox. I assume that you are reasonably proficient as a programmer and comfortable working with data and graphics. At the same time, I don’t expect that you have more than a cursory knowledge of contemporary professional web development.

Here is what you should already have:

• Knowledge of at least one or two programming languages (but not necessarily JavaScript), and the confidence to pick up the syntax of a new language from its language reference.

• Familiarity with contemporary programming concepts; that is, not just loops, conditionals, and common data structures, but also closures and higher-order functions.

• Basic understanding of XML and the hierarchical structure it imposes on documents. I expect that you know of the DOM and how it treats the elements of a web page as nodes in a tree, but I do not assume that you are familiar with either the original DOM API or any one of its modern replacements (like jQuery).

• Exposure to simple HTML and CSS (you should be able to recognize and use <body> and <p> tags, and so on), as well as some familiarity with CSS syntax and mechanisms.

But in particular, the reader I have in mind is impatient: proficient and capable, but frustrated by previous attempts to wrap his or her head around D3. If this describes your situation, this book is for you!

Why D3?

Why should D3 be of interest to programmers and scientists—or in fact anybody who is not primarily a web developer? The following reasons stand out:

• D3 provides a convenient way to deliver graphics over the web. If you work with data and visualizations, you know the drill: you create your figures in the plotting program of your choice, then save the results as PNG or PDF, and then create a web page with <img> tags, so that other people can see your work. Wouldn’t it be nice if you could create and publish your figures in one step?

• More importantly, D3 makes it easy and convenient to create animated and interactive graphics. This point cannot be overemphasized: scientific visualization has as much to benefit from animation and interactivity as any other field—but this goal has in the past been notoriously difficult to achieve. It frequently required intricate and unbefitting technologies (ever tried your hand at Xlib programming?) or specialized and often expensive commercial packages. D3 lets you leave these challenges behind and puts you into the present day for your visualization needs.

• Graphics aside, D3 is an accessible, easy-to-learn, and easy-to-use framework for general-purpose DOM handling. If you have the occasional need to manipulate the DOM, then D3 may be all you need, without the obligation to master all the other frameworks and APIs for web programming. The design of the library itself is also interesting as a model for the facilities it provides “out of the box” to handle common data manipulation and visualization tasks.

But more than anything, I believe D3 to be an enabling technology that quite generally broadens the range of solutions available to its users. The most interesting applications of D3 are possibly those that have not yet been invented.

What Is in This Book

This book tries to be a comprehensive, yet concise, introduction to D3, covering most major parts of functionality in sufficient depth.

• It tries to be a convenient, one-stop resource by including both API reference documentation as well as background information on ancillary topics that might not be familiar to the typical reader (such as SVG, JavaScript, and the DOM, but also color spaces or the HTML Canvas element).

• The book emphasizes mechanisms and design concepts, rather than ready-made recipes. The assumption is that readers will want to learn D3 deeply enough that they can apply it to their own, possibly novel and unforeseen, purposes.

Basically, the hope is that this book prepares you to do things with D3 that I would never have thought of!

… and What Is Not

This book intentionally restricts itself to D3 alone, its capabilities and mechanisms. This implies a number of omissions:

• No extensive case studies or cookbook-style recipes

• No introductions to data analysis, statistics, or visual design

• No mention of JavaScript frameworks other than D3

• No discussion of contemporary web development in general

I would like to emphasize the last two items. This book is strictly about D3 only, without any reference to or dependence on other JavaScript frameworks or libraries. This is quite intentional: I wanted to make D3 accessible to those readers as well who are unfamiliar or uncomfortable with JavaScript’s rich but heterogenous ecosystem. For the same reason, this book does not discuss other topics in contemporary web development. In particular, you will find no discussion of browser compatibility and related topics. The assumption is that you are generally using a modern, up-to-date, JavaScript-enabled browser that is capable of rendering SVG.¹

One other omission concerns D3 support for geographic and geospatial information. Although important, this topic seems sufficiently well-contained that it should not be too difficult to learn from the D3 Reference Documentation once the D3 basics are clear.

How to Read This Book

This book presents a continuous, progressive narrative, with new material systematically introduced from chapter to chapter. That being said, the later chapters in particular can be read in any order after the necessary groundwork has been laid in the first half of the book. Here is my suggested roadmap:

1. Unless you already have a solid foundation in SVG, I strongly recommend that you read Appendix B. Without this knowledge, nothing else will make much sense.

2. Everyone should read Chapter 2 as a tutorial warm-up and to set expectations for the topics that will be discussed.

3. Chapter 3 is required reading. Selections are the principal organizing concept in D3. Not only do selections represent a handle on the DOM tree, but they also manage the association between DOM elements and the data set. Pretty much every D3 program begins by taking a selection, and understanding them and their capabilities is mandatory when working with D3.

4. Strictly speaking, Chapter 4 on event handling, interactivity, and animations is optional. But because these are among the most exciting capabilities offered by D3, it would be a shame to skip it.

5. Chapter 5 is important because it explains some fundamental D3 design concepts (such as components and layouts) and introduces generally useful techniques (like SVG transformations and custom components).

6. The remaining chapters can largely be read in any order, as and when the need for that particular topic arises. In particular, I would like to draw attention to Chapter 7 and its detailed description of the inconspicuous but extremely versatile scale objects, and to the variety of functions for array handling in Chapter 10.

Conventions

This section explains some specific conventions used in the remainder of this book.

Conventions of the D3 API

The D3 API uses some uniform conventions that greatly enhance its usability. Some of them are common JavaScript idioms and not specific to D3, but may not be familiar to non-JavaScript programmers. By setting them out explicitly in one place, subsequent discussions don’t need to be cluttered with redundant information.

• D3 is primarily an access layer to the DOM tree. As a rule, D3 makes no attempt to encapsulate underlying technologies, and instead provides convenient, but otherwise generic, handles on them. For example, D3 does not introduce “circle” or “rectangle” abstractions of its own, but instead gives the programmer direct access to the SVG facilities for creating graphical shapes. The advantage of this approach is that D3 is tremendously adaptable and not tied to one particular technology or version. The disadvantage is that programmers need to have knowledge of the underlying technologies, in addition to D3, since D3 itself does not provide a complete abstraction layer.

• Because JavaScript does not enforce formal function signatures, all function arguments are technically optional. Many D3 functions use the following idiom: when called with appropriate arguments, these functions act as setters (setting the corresponding property to the supplied value); when called without arguments, these functions act as getters (returning the current value of the property). To entirely remove a property, call the appropriate setter while supplying null as argument.

• When called as setters, functions typically return a reference to the current object, thus enabling method chaining. (This idiom is so intuitive and consistent that it will rarely be mentioned explicitly again.)

• Instead of a value, many D3 setters can take an accessor function as argument, which is expected to return a value that will be used to set the property in question. The parameters expected by accessor functions are not the same across all of D3, but a set of related D3 functions will always call accessor functions in a uniform way. The details of accessor arguments are documented with the respective D3 functions.

• Some important D3 facilities are implemented as function objects. They perform their primary task when called as a function, but they are also objects, with member functions and internal state (examples are scale objects, see Chapter 7; and generators and components, see Chapter 5). It is a common pattern to instantiate such an object, configure it using its member functions, and finally invoke it to complete its purpose. Frequently, the final invocation does not use explicit function-call syntax, but instead employs one of JavaScript’s methods for “synthetic” function calls: the function object is passed to another function (such as call()), which supplies the required arguments and finally evaluates the function object itself.

Conventions for the API Reference Tables

Throughout the book, you will find tables showing parts of the D3 API in a reference format. Entries in these tables are sorted by relevance, keeping related functions together.

• D3 functions are either called on the global d3 object, or as member functions of some D3 object; some functions are available both ways. If a function is called through an object, this object is referred to as the receiver of the method call. Inside the member function, the receiver is the object pointed to by the this variable.

• All API reference tables indicate the type of the receiver in the caption. The tables do not refer explicitly to the object’s prototype.

• Function signatures attempt to indicate the type of each argument, but many functions accept such a wide variety of different argument types that no unambiguous notation is practical. Read the textual description for full details. Where they are used, brackets indicate an array. Optional function arguments are not indicated explicitly.

Conventions for the Code Examples

The code examples are intended to demonstrate D3 features and mechanisms. In order to bring out their respective point most clearly, the examples are stripped to the bare essentials. I dispensed with most “niceties,” such as pleasing colors or semantically interesting data sets. Colors are usually primary, and most data sets are small and simple.

On the other hand, each example is complete in itself, can be executed as is, and will create the associated graph. With few exceptions, I don’t show code fragments. I found that it is better to display simple examples in full, rather than show only the “interesting bits” from a longer example; this way, there is no danger that the overall context is getting lost. Everything is executable and ready to be extended and embellished at will.

Naming conventions

The examples use the following naming conventions for variables:

• First-letter acronyms for individual objects: c for “circle,” p for point, and so on. Append an “s” for collections: cs will be an array of circles, ps an array of points.

• Frequently occurring quantities have their own notation: pixels are denoted with px, scale objects with sc. Generators and components are function objects that “make” something and thus are called mkr.

• The letter d is used generically to indicate “the current thing” in anonymous functions. When working with D3 selections, d is usually an individual data point bound to a DOM element; when working with arrays, d is an array element (as in ds.map( d => +d )).

• Data sets are called data or ds.

• Selections representing either an <svg> or a <g> element are common and, when assigned to a variable, are denoted as svg or g.

Source file organization

Beginning with Chapter 3, I adopt a convention whereby for each code listing, the page is expected to already contain an <svg> element with a unique id and with properly set width and height attributes. The example code then selects this SVG element by its id attribute and often assigns this selection to a variable for future reference:

var svg = d3.select( "#fig" );

This avoids the ambiguity of using a more general selector (such as d3.select( "svg" )) and makes it easy to include several examples in a single HTML page.

To each graph, there corresponds a JavaScript function creating the SVG elements that make up the figure dynamically. By convention, the function names begin with make... and continue with the value of the id attribute of the destination SVG element.

With the exception of the examples in Chapter 2, there is one HTML page and one JavaScript file per chapter. (With rare exceptions, I don’t include JavaScript code in an HTML page directly.)

Platform, JavaScript, and browser

To run the examples, you need to run a local or hosted web server (see Appendix A). The examples should work in any contemporary, JavaScript-enabled browser. Several versions of JavaScript are currently extant.² With three exceptions, the code examples use only “classic” JavaScript (ES5, released in 2009/2011) without any further frameworks or libraries. The three features that require a newer version of JavaScript (ES6, released in 2015) are:

• The concise fat arrow notation for anonymous functions (see Appendix C), which is used throughout the examples.

• Destructuring assignment ([a, b] = [b, a]), which is used in a few places.

• Several examples access remote resources using D3 wrappers for the Fetch API (see Chapter 6); these depend on the JavaScript Promise object.

¹ This is in the spirit of D3 itself. As the D3 website states: “D3 is not a compatibility layer, so if your browser doesn’t support standards, you’re out of luck” (https://github.com/d3/d3/wiki).

² See https://en.wikipedia.org/wiki/ECMAScript.