“Most software systems are created with the implicit assumption that the entire system is under the control of one entity, or at least that all entities participating within a system are acting towards a common goal and not at cross-purposes. Such an assumption cannot be safely made when the system runs openly on the Internet.”
— Roy Fielding Architectural Styles and the Design of Network-based Software Architectures
“A Discordian Shall Always use the Official Discordian Document Numbering System.”
— Malaclypse the Younger and Lord Omar Khayyam Ravenhurst Principia Discordia
I’m going to show you a better way to do distributed computing, using the ideas underlying the most successful distributed system in history: the World Wide Web. I hope you’ll read this book if you’ve decided (or your manager has decided) that your company needs to publish a web API. It doesn’t matter whether you’re planning a public API, a purely internal API, or an API accessible by trusted partners—they can all benefit from the philosophy of REST.
This is not necessarily the book for you if you want to learn how to write API clients. That’s because most existing API designs are based on assumptions that are several years old, assumptions that I’d like to destroy.
Most of today’s APIs have a big problem: once deployed, they can’t change. There are big-name APIs that stay static for years at a time, as the industry changes around them, because changing them would be too difficult.
But RESTful architectures are designed for managing change. The World Wide Web is made of millions of websites, running atop thousands of different server implementations, and undergoing periodic redesigns. Websites are accessed by billions of users who are using hundreds of different client implementations on dozens of hardware platforms. Your deployment won’t look like this howling mess, but the closer you come to web scale, the more familiar this picture will look.
A very simple system is always easy to change. At small scales, a RESTful system has a larger up-front design cost than a push-button solution. But as your API matures and starts to change, you’ll really need some way—like REST—of adapting to change.
- An API that’s commercially successful will stay available for years on end. Some APIs have hundreds or even thousands of users. Even if the problem domain only changes occasionally, the cumulative effect on clients can be huge.
- Some APIs change all the time, with new data elements and business rules constantly being added.
- In some APIs, each client can change the workflow to suit its needs. Even if the API itself never changes, each client will experience it differently.
- The people who write the API clients usually don’t work on the same team as the people who write the servers. All APIs that are open to the public fall under this category. If you don’t know what kind of clients are out there, you need to be very careful about making changes—or you need to have a design that can change without breaking all the clients.
If you copy existing designs for your API, you will probably only repeat the mistakes of the past. Unfortunately, most of the improvements are happening below the surface, in experiments and through slow-moving standards processes. I’ll cover dozens of specific technologies in this book, including many that are still under development. But my main goal is to teach you the underlying principles of REST. Learn those, and you’ll be able to exploit whichever experiments pan out and whichever standards are approved.
There are two specific problems I’m trying to solve with this book: duplication of effort and avoidance of hypermedia. Let’s take a look at them.
Duplication of Effort
An API released today will be named after the company that hosts it. We talk about the “Twitter API,” the “Facebook API,” and the “Google+ API.” These three APIs do similar things. They all have some notion of user accounts and (among other things) they all let users post a little bit of text to their accounts. But each API has a completely different design. Learning one API doesn’t help you learn the next one.
Of course, Twitter, Facebook, and Google are big companies that compete with each other. They don’t want to make it easy for you to learn their competitors’ APIs. But small companies and nonprofits do the same thing. They design their APIs as though nobody else had ever had a similar idea. This interferes with their goal of getting people to actually use their APIs.
Let me show you just one example. The website ProgrammableWeb has a directory of over 8,000 APIs. As I write this, it knows about 57 microblogging APIs—APIs whose main purpose is posting a little bit of text to a user account. It’s great that there are 57 companies publishing APIs in this field, but do we really need 57 different designs? We’re not talking about something complicated here, like insurance policies or regulatory compliance. We’re talking about posting a little bit of text to a user account. Do you want to be the one who designs the 58th microblogging API?
The obvious solution would be to create a standard for microblogging APIs. But there already is a standard that would work just fine: the Atom Publishing Protocol. It was published in 2005, and almost nobody uses it. There’s something about APIs that makes everyone want to design their own from scratch, even when that makes no sense from a business perspective.
I don’t think I can single-handledly stop this wasted effort, but I do think I can break down the problem into parts that make sense, and present some ways for a new API to reuse work that’s already been done.
Hypermedia Is Hard
Back in 2007, Leonard Richardson and Sam Ruby wrote the predecessor to this book, RESTful Web Services (O’Reilly). That book also tried to address two big problems. One of the problems has been solved; the other is nowhere close to being solved.
The first problem: in 2007, the REST school of API design was engaged in a standoff against a rival school that used heavyweight technologies based on SOAP and questioned the very legitimacy of the REST school. RESTful Web Services was a salvo in this standoff, a defense of RESTful design principles against the attacks of the SOAP school.
Well, the standoff is over, and REST won. SOAP APIs are still used, but only within the big companies that were backing the SOAP school in the first place. Pretty much all new public-facing APIs pay lip service to RESTful principles.
Which brings me to the second problem: REST isn’t just a technical term—it’s also a marketing buzzword. For a long time, REST was a slogan that signified nothing beyond opposition to the SOAP school. Any API that didn’t use SOAP was marketed as REST, even if its design made no sense or betrayed the technical principles of REST. This was inaccurate, confusing, and it gave REST—i.e., REST as a technical term—a bad name.
This situation has improved a lot since 2007. When I look at new APIs, I see the work of developers who understand the concepts I’ll be explaining in the first few chapters of this book. Most developers who fly the REST flag today understand resources and representations, how to name resources with URLs, and how to properly use HTTP methods. The first three chapters of this book don’t do much but get new developers up to speed.
But there’s one aspect of REST that most developers still don’t understand: hypermedia. We all understand hypermedia in the context of the Web. It’s just a fancy word for links. Web pages link to each other, and the result is the World Wide Web, driven by hypermedia. But it seems we’ve got a mental block when it comes to hypermedia in web APIs. This is a big problem, because hypermedia is the feature that makes a web API capable of handling changes gracefully.
Starting in Chapter 4, my overriding goal for RESTful Web APIs will be to teach you how hypermedia works. If you’ve never heard of this term, I’ll teach it to you along with the other important REST concepts. If you’ve heard of hypermedia but the concept intimidates you, I’ll do what I can to build up your courage. If you just haven’t been able to wrap your head around hypermedia, I’ll show it to you in every way I can think of, until you get it.
RESTful Web Services covered hypermedia, but it wasn’t central to the book. It was possible to skip the hypermedia parts of the book and still design a functioning API. By contrast, RESTful Web APIs is effectively a book about hypermedia.
I did it this way because hypermedia is the single most important aspect of REST, and the least understood. Until we all understand hypermedia, REST will continue to be viewed as a marketing buzzword rather than a serious attempt to handle the complexity of distributed computing.
What’s in This Book?
The first four chapters introduce the concepts behind REST, as it applies to web APIs.
- Chapter 1, Surfing the Web
- This chapter explains basic terminology using a RESTful system you’re already familiar with: a website.
- Chapter 2, A Simple API
- This chapter translates the lessons of the Web to a programmable API with identical functionality to the website discussed in Chapter 1.
- Chapter 3, Resources and Representations
- Resources are the fundamental concept underlying HTTP, and representations are the fundamental concept underlying REST. This chapter explains how they’re related.
- Chapter 4, Hypermedia
- Hypermedia is the missing ingredient that ties representations together into a coherent API. This chapter shows what hypermedia is capable of, mostly using a hypermedia data format you’re already familiar with: HTML.
The next four chapters describe different strategies for designing a hypermedia API:
- Chapter 5, Domain-Specific Designs
- The obvious strategy is to design a completely new standard that deals with your exact problem. I use the Maze+XML standard as an example.
- Chapter 6, The Collection Pattern
- One pattern in particular—the collection pattern—shows up over and over again in API design. In this chapter, I show off two different standards that capture this pattern: Collection+JSON and AtomPub.
- Chapter 7, Pure-Hypermedia Designs
- When the collection pattern doesn’t fit your requirements, you can convey any representation you want using a general-purpose hypermedia format. This chapter shows how it works using three general hypermedia formats (HTML, HAL, and Siren) as examples. This chapter also introduces HTML microformats and microdata, which lead in to the next chapter.
- Chapter 8, Profiles
A profile fills in the gaps between a data format (which can be used by many different APIs) and a specific API implementation. The profile format I recommend is ALPS, but I also cover XMDP and JSON-LD.
In this chapter, my advice begins to outstrip the state of the art at the time this book was written. I had to develop the ALPS format for this book, because nothing else would do the job. If you’re already familiar with hypermedia-based designs, you might be able to skip up to Chapter 8, but I don’t think you should skip past it.
- Chapter 9, The Design Procedure
- This chapter brings together everything discussed in the book so far, and gives a step-by-step guide to designing a RESTful API.
- Chapter 10, The Hypermedia Zoo
- In an attempt to show what hypermedia is capable of, this chapter discusses about 20 standardized hypermedia data formats, most of them not covered elsewhere in the book.
- Chapter 11, HTTP for APIs
- This chapter gives some best practices for the use of HTTP in API implementations. I also discuss some extensions to HTTP, including the forthcoming HTTP 2.0 protocol.
- Chapter 12, Resource Description and Linked Data
- Linked Data is the Semantic Web community’s approach to REST. JSON-LD is arguably the most important Linked Data standard. It’s covered briefly in Chapter 8, and I revisit it here. This chapter also covers the RDF data model, and some RDF-based hypermedia formats that I didn’t get to in Chapter 10.
- Chapter 13, CoAP: REST for Embedded Systems
- This chapter closes out the core body of the book by covering CoAP, a RESTful protocol that doesn’t use HTTP at all.
- Appendix A, The Status Codex
- An extension of Chapter 11, this appendix provides an in-depth look at the 41 standard status codes defined in the HTTP specification, as well as a few useful codes defined as extensions.
- Appendix B, The Header Codex
- Similar to Appendix A, this appendix is also an extension of Chapter 11. It provides a detailed outline of the 46 request and response headers defined in the HTTP specification, as well as a few extensions.
- Appendix C, An API Designer’s Guide to the Fielding Dissertation
- This appendix includes an in-depth discussion of the foundational document of REST, in terms of what it means for API design.
- The glossary contains definitions to terms you’ll frequently encounter when working with RESTful web APIs. It’s a good place to turn for familiarizing yourself with basic concepts or if you need a quick, at-a-glance reminder of a particular concept’s definition.
What’s Not in This Book
RESTful Web Services was the first book-length treatment of REST, and it had to cover a lot of ground. Fortunately, there are now over a dozen books on various aspects of REST, and that frees up RESTful Web APIs to focus on the core concepts.
To keep this book focused, I’ve removed a few topics that you might have been expecting me to cover. I want to tell you what is not in this book, so that you don’t buy it and then feel disappointed:
Client programming is not covered here. Writing a client to consume a hypermedia-based API is a new kind of challenge. Right now, the closest thing we have to a generic API client is a library that sends HTTP requests. This was true in 2007, and it’s still true. The problem is on the server side.
When you write a client for an existing API, you’re at the mercy of the API designer. I can’t give you any general advice, because right now there’s no consistency across APIs. That’s why, in this book, I’m trying to drum up enthusiasm for a little server-side consistency. When APIs become more similar to each other, we’ll be able to write more sophisticated client-side tools.
Chapter 5 contains some sample client implementations and tries to classify different types of clients, but if you want a whole book on API clients, this is not your book. I don’t think the book you want exists right now.
- I spend quite a bit of time on the mechanics of HTTP (Chapter 11, Appendix A, and Appendix B), but I don’t cover any given HTTP topic in a lot of depth, and there are some topics—notably HTTP intermediaries like caches and proxies—which I barely cover at all.
RESTful Web Services focused heavily on breaking down your business requirements into a set of interlinked resources. My experience since 2007 has convinced me that thinking of API design as resource design is a very effective way to avoid thinking about hypermedia. This book takes a different approach, focusing on representations and state transitions rather than resources.
That said, the resource design approach is certainly valid. For advice on moving in that direction, I recommend RESTful Web Services Cookbook by Subbu Allamaraju (O’Reilly).
This book has two authors (Leonard and Mike), but for the duration of this book we’ve merged our identities into a single authorial “I.”
Nothing in this book is tied to any particular programming language. All of the code takes the form of messages (usually JSON or XML documents) sent over a network protocol (usually HTTP). I will be assuming that you’re familiar with common programming concepts like antipatterns and breadth-first search, and that you have a basic understanding of how the World Wide Web works.
I chose Node because it lets me use the same programming language for client and server code. You won’t need to mentally switch back and forth between programming langauges to understand both sides of a client-server transaction. Node is open source and available on Windows, Mac, and Linux systems. It is easy to install on these operating systems, and you shouldn’t have much trouble getting the examples up and running.
I’m hosting the code on GitHub because that will make it easy to update the implementations over time. This also makes it possible for readers to contribute ports of the example clients and servers to other programming languages.
The great thing about these standards is the solid baseline they give you. You can use them to build a completely new kind of website or API, something that no one has ever tried before. Instead of having to explain your entire system to all your users, you’ll only have to explain the part that’s new.
The bad news is that these agreements are often borderline unreadable: long walls of ASCII text written in tooth-achingly precise English in which everyday words like “should” have technical meanings and are capitalized “SHOULD.” A lot of technical books are bought by people who are hoping to avoid having to read a standards document.
Well, I can’t make any guarantees. If one of these standards looks like something you can use in your work, you need to be willing to dive into its spec and really understand it (or buy a book that covers it in more detail). I don’t have space to give more than a basic overview of standards like Siren, CoAP, and Hydra. Not to mention that giving a lot of detail would bore all the readers who don’t need those particular standards to do their work.
When navigating the forest of standards, it’s useful to keep in mind that not all standards have equal force. Some are extremely well established, used by everyone, and if you go against them you’re causing a lot of trouble for yourself. Other standards are just one person’s opinion, and that opinion might be no better than yours.
I find it helpful to divide standards into four categories: fiat standards, personal standards, corporate standards, and open standards. I’ll be using these terms throughout the book, so let me explain each one in a bit more depth before we move on.
Fiat standards aren’t really standards; they’re behaviors. No one agreed to them. They’re just a description of the way somebody does things. The behavior may be documented, but the core assumption of a standard—that other people ought to do things the same way—is missing.
Pretty much every API today is a fiat standard, a one-off design associated with a specific company. That’s why we talk about the “Twitter API,” the “Facebook API,” and the “Google+ API.” You may need to understand these designs to do your job and you may write your own clients for these designs, but unless you work for the company in question, there’s no expectation that you should use this design for your API. If you reuse a fiat standard, we don’t say your API conforms to a standard; we say it’s a clone.
The main problem I’m trying to solve in this book is that hundreds of person-years of design work is locked up in fiat standards where it can’t be reused. This needs to stop. Designing a new API today means reinventing a long series of wheels. Once your API is finished, your client developers have to reinvent corresponding wheels on the client side.
Even under ideal circumstances, your API will be a fiat standard, since your business requirements will be slightly different from everyone else’s. But ideally a fiat standard would be just a light gloss over a number of other standards.
When I describe a fiat standard, I’ll link to its human-readable documentation.
Personal standards are standards—you’re invited to read the documents and implement the standards yourself—but they’re just one person’s opinion. The Maze+XML standard I describe in Chapter 5 is a good example. There’s no expectation that Maze+XML is the standard way to implement a maze game API, but if it works for you, you might as well use it. Someone else has done the design work for you.
Personal standards generally use less formal language than other kinds of standards. Many open standards start off as personal standards—as side projects that are formalized after a lot of experimentation. Siren, which I cover in Chapter 7, is a good example.
When I describe a personal standard, I’ll link to its specification.
Corporate standards are created by a consortium of companies trying to solve a problem that plagues them all, or by a single company trying to solve a recurring problem on behalf of its customers. Corporate standards tend to be better defined and to use more formal language than personal standards, but they have no more force than personal standards. They’re just one company’s (or a group of companies') opinion.
Corporate standards include Activity Streams and schema.org’s microdata schemas, both of which are covered in Chapter 10. Many industry standards start off as corporate standards. OData (also discussed in Chapter 10) started as a Microsoft project, but it was submitted to OASIS in 2012 and will eventually become an OASIS standard.
When I describe a corporate standard, I’ll link to its specification.
An open standard has gone through a process of design by committee, or at least had an open comment period during which a lot of people read the specification, complained about it, and made suggestions for improvement. At the end of this process, the specification was blessed by some kind of recognized standards body.
This process gives an open standard a certain amount of moral force. If there’s an open standard that does more or less what you want, you really should use it instead of making up your own fiat standard. The design process and the comment period probably turned up a lot of issues that you won’t encounter until it’s too late.
In general, open standards come with some kind of agreement that promises you can implement them without getting hit with a patent infringement lawsuit from a company that was involved in the standards process. By contrast, implementing someone else’s fiat standard may incite them to file a patent infringement lawsuit against you.
A few open standards mentioned in this book came out of the big-name standards bodies: ANSI, ECMA, ISO, OASIS, and especially the W3C. I can’t say what it’s like to sit on one of these standards bodies, because I’ve never done it. But the most important standards body is one anyone can contribute to: the IETF, the group that manages the all-important RFCs.
Requests for Comments (RFCs) and Internet-Drafts
Most RFCs are created through a process called the Standards Track. Throughout this book, I’ll be referencing documents that are in different places on the Standards Track. I’d like to briefly discuss how the track works, so that you’ll know how seriously to take my recommendations.
An RFC begins life as an Internet-Draft. This is a document that looks like a standards document, but you’re not supposed to build implementations based on it. You’re supposed to find problems with the specification and give feedback.
An Internet-Draft has a built-in lifetime of six months. Six months after it is published, a draft must be approved as an RFC or replaced with an updated draft. If neither of those things happens, then the draft expires and should not be used for anything. On the other hand, if the draft is approved, it expires immediately and is replaced by an RFC.
Because of the built-in expiration date, and because an Internet-Draft isn’t technically any kind of standard, it’s tricky business mentioning them in a book. At the same time, API design is a field that’s changing rapidly, and an Internet-Draft is better than nothing. I will be mentioning many Internet-Drafts in this book under the assumption that they’ll become RFCs without major changes. That assumption has held up pretty well; several Internet-Drafts that I mention here became RFCs while I was writing the book. If a particular draft doesn’t pan out, all I can do is apologize in advance.
RFCs and Internet-Drafts are given code names. When I describe one of these, I won’t link to its specification. I’ll just refer to it by its code and let you look it up. For example, I’ll refer to the HTTP/ 1.1 specification as RFC 2616. I’ll refer to an Internet-Draft by its name. For example, I’ll use “draft-snell-link-method” to refer to the proposal to add LINK and UNLINK methods to HTTP.
Whenever you see one of these code names, you can do a web search and find the latest version of the RFC or Internet-Draft. If an Internet-Draft becomes an RFC after this book is published, the final version of the Internet-Draft will link to the RFC.
When I describe a W3C or OASIS standard, I’ll link to the specification, because those standards aren’t given code names.
Conventions Used in This Book
The following typographical conventions are used in this book:
- Indicates new terms, URLs, email addresses, filenames, and file extensions.
- Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords.
Constant width bold
- Shows commands or other text that should be typed literally by the user.
Constant width italic
- Shows text that should be replaced with user-supplied values or by values determined by context.
This icon signifies a tip, suggestion, or general note.
This icon indicates a warning or caution.
Using Code Examples
This book is here to help you get your job done. In general, if this book includes code examples, you may use the code in this book in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing a CD-ROM of examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission.
We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “RESTful Web APIs by Leonard Richardson and Mike Amundsen (O’Reilly). Copyright 2013 Leonard Richardson and amundsen.com, Inc., and Sam Ruby. 978-1-449-35806-8.”
If you feel your use of code examples falls outside fair use or the permission given here, feel free to contact us at email@example.com.
Safari® Books Online
Technology professionals, software developers, web designers, and business and creative professionals use Safari Books Online as their primary resource for research, problem solving, learning, and certification training.
Members have access to thousands of books, training videos, and prepublication manuscripts in one fully searchable database from publishers like O’Reilly Media, Prentice Hall Professional, Addison-Wesley Professional, Microsoft Press, Sams, Que, Peachpit Press, Focal Press, Cisco Press, John Wiley & Sons, Syngress, Morgan Kaufmann, IBM Redbooks, Packt, Adobe Press, FT Press, Apress, Manning, New Riders, McGraw-Hill, Jones & Bartlett, Course Technology, and hundreds more. For more information about Safari Books Online, please visit us online.
How to Contact Us
Please address comments and questions concerning this book to the publisher:
|O’Reilly Media, Inc.|
|1005 Gravenstein Highway North|
|Sebastopol, CA 95472|
|800-998-9938 (in the United States or Canada)|
|707-829-0515 (international or local)|
We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at http://oreil.ly/RESTful-Web-APIs.
To comment or ask technical questions about this book, send email to firstname.lastname@example.org.
For more information about our books, courses, conferences, and news, see our website at http://www.oreilly.com.
Find us on Facebook: http://facebook.com/oreilly
Follow us on Twitter: http://twitter.com/oreillymedia
Watch us on YouTube: http://www.youtube.com/oreillymedia
We owe a debt of thanks to Glenn Block who spent untold hours listening to ideas and working through real code to test those ideas. To Benjamin Young and all the folks at RESTFest who agreed to be part of our experiments, and who gave great feedback and advice even when we didn’t want to hear it. To Mike’s colleagues at Layer 7 Technologies, including Dimitri Sirota and Matt McLarty, who supported and encouraged his work on this project. To Sam Ruby and Mike Loukides, who were essential to RESTful Web Services, this book’s predecessor. To Sumana Harihareswara, Leonard’s supportive wife. To the social communities that create an excellent place to collaborate and converse on REST and APIs; especially Yahoo’s REST-Discuss, Google Groups’ API-Craft, and the Hypermedia group at LibreList.
And finally, to all those who read the early drafts of this manuscript and provided much-needed criticism and support: Carsten Bormann, Todd Brackley, Tom Christie, Timothy Haas, Jamie Hodge, Alex James, David Jones, Markus Lanthaler, Even Maler, Mark Nottingham, Cheryl Phair, Sergey Shishkin, Brian Sletten, Mark Stafford, Stefan Tilkov, Denny Vrandečić, Ruben Verborgh, and Andrew Wahbe.
 If you’re wondering, this is why we changed the title. The term “web services” became so tightly coupled with SOAP that when SOAP went down, it took “web services” with it. These days, everyone talks about APIs instead.
 The meaning of “SHOULD” is given in RFC 2119.
 For the purposes of this book, anyway. If you need standard sizes for screws and bolts, you want ANSI or ISO.