Designing great Web APIs

Organizational structure leaks

When an organizational structure leaks into the API, it announces to developers that the API isn’t a product; it is the output of a specific team within the organization. The result is inconsistent design and duplication or gaps in functionality.

Organizational structure creeps into an API for two reasons: 1) APIs are designed in isolation from other teams, or 2) APIs are designed around internal systems rather than external needs.

To avoid this, consider establishing an API governance board that encourages consistency across teams. A governance board isn’t meant to be an oppressive committee that stalls innovation. Rather, they represent the best interests of developers, your APIs, and your business. Great governance boards often act as internal consultants for your APIs to make them successful.

Database structure leaks

Database structure creeps in through the use of tools and frameworks that promise rapid API development at the expense of a thoughtful API design. They externalize the data through generated APIs but make one huge assumption that becomes the enemy of great API design: external developers want to use your API like you access your database.

Developers using your API don’t care how you store your data as long as it is stored correctly and reliably. Taking an outside-in design approach will prevent the database design from creeping in by encouraging you to see your API as external developers will see it.

When taking an outside-in approach to API design, focus on how the API will be used, rather than how it is built. This means looking for ways to make your API useful for web clients, where connectivity and CPU capabilities are abundant as well as mobile devices, where connectivity and battery enforce a variety of limitations.

Format

How you decide to distribute your documentation will heavily impact the audience. Many companies employ technical writers who use tools that produce beautiful, typeset documentation in PDF format. The documentation looks professional and amazing.

However, there is a big downside to PDF-based documentation: the PDF goes stale once it is downloaded. As new versions of the PDF are released, older versions remain on their hard drive for reference. This means that they won’t have the benefit of the latest examples, clarifications, and new features.

Instead, APIs need to have documentation that is always current for the reader. HTML-based documentation provides an always updated, easily accessible solution. It should be hosted on your website and easily accessed by developers to ensure the documentation is available at any time.

Completeness

At least one time in your life you have encountered bad documentation. It may have been the assembly instructions for do-it-yourself furniture. Perhaps it was an owner’s manual for a new device or software product. The experience of poor documentation can waste your time, leave you frustrated, or perhaps even cause you to replace it with something better.

API documentation can provide the same experience if it is left incomplete or unclear. Developers may continue to use the API in the short term, but they will likely look to replace your API at first chance.

It is important to remember that a public web API is a contract with every developer that will use it. Whether you target internal developers, developers within a partner organization, or public developers in general, you are making a contract with each individual developer.

Therefore, the documentation you provide to developers will provide the bulk of their developer experience. It will be the first thing that they experience when trying to understand your API, and the first place they go when they are stuck.

The challenge with API documentation is that it must serve a variety of situations. This may include the developer looking at your API for the first time or the expert developer looking to use a newly released feature. This means that complete documentation takes into account the following scenarios:

• Developers, product managers, and business users that are considering your API but haven’t committed to it yet

• The developer integrating your API for the first time

• The expert developer who has been using your API for some time and wants to explore previously hidden features or options

• Support staff trying to troubleshoot how your API works and why an application is failing

• Newly hired developers in your organization seeing the API (and the business) details for the first time

Too often, documentation lacks focus around one or more of these areas. This is particularly the case when using documentation written inline with the code itself. Tools exist that allow API docs to be built from inline comments and are sometimes used to expedite the documentation process. However, the result is documentation that is only focused on what that specific code does, not how to use it successfully. Complete documentation must consider all of the five scenarios previously listed.

Interactive

Finally, great documentation should be interactive. As mentioned, API documentation delivered in HTML format can be hosted on a website and kept up-to-date. But another added benefit is that we can use the browser to actually interact with our API.

Figure 2-1 is an example of interactive documentation using Swagger.

Interactive API documentation allows anyone, including developers, quality assurance staff, product managers, and support staff to make API calls into a live running server from within the documentation itself. This is possible because our web APIs use the HTTP standard and therefore don’t require special libraries or software to make them work. Imagine allowing developers and QA staff explore your API before they ever write a line of code or an automated test! That is power of interactive documentation.

Consistent naming

As developers approach your API, the first thing they will notice is the naming conventions that you use. This includes the names of the domain concepts referenced, many of which become the resources used in the URLs of your API. Therefore, naming is critical to the understanding and usage of your API. The following are some tips for incorporating consistent naming:

• Avoid abbreviations, as they can be difficult to read and often create confusion as some names may be unclear or inconsistently abbreviated.

• Be consistent with resource names to avoid confusion.

• Refrain from referencing internal systems, as this results in requiring insider knowledge to use or comprehend your API.

Consistent resource URLs

As you design your API, you will need to map out the various URLs that will represent your resources. It is highly recommended that you develop a resource ontology to ensure that URLs are consistently designed into the API.

If you are unfamiliar, ontology is a technique used in information science for the naming and typing of entities and their relationships. API ontologies define the structure of your API, from the top-level resources to the nested resources under them.

Figure 2-2 is an example of a URL ontology for an ecommerce website.

When designing your resource ontology, apply the following techniques to create a more consistent API:

• Use plural resource names when offering a collection of resources and a singular resource name for a single resource (e.g., /users for a collection of user resources, /user for a single user resource).

• Use nested resources to indicate relationships between resources.

• Avoid one-off URLs (e.g., /users/current rather than the more appropriate /user).

We will cover this topic in more detail in API Design Details.

Consistent payload formats

Finally, the payload format should be consistent to allow API consumers to easily construct request payloads and parse response payloads. Most developers prefer to write helper code to handle this logic. If an API is inconsistent in its approach, developers will have to write one-off code to handle the differences. Follow these patterns for more consistent API payloads:

• Reuse field names across payloads when possible for better predictability (e.g., use firstName and lastName or use fullName, but don’t interchange them).

• Avoid abbreviations in field names.

• Keep consistent casing rules, usually camelCase or snake_case (e.g., firstName or first_name, but not both).

• Select a single payload format or standard when handling payloads, including resource representations and search result collections.

• Ensure all error message formats are consistent across the API. This is especially important when an API is built by multiple developers who may otherwise handle error messaging differently.

Beyond data: Supporting API-based workflows

APIs often require more than just data. While data is important, developers seek out APIs to solve a problem. If they wanted to store data, they would push it into a database directly, unless you have data that no one else can provide. They are looking to your API to do something that they don’t want to build themselves. This may be solving difficult problems, building support for collaboration, or providing data analysis.

Great APIs need to look beyond data access endpoints. While these kinds of endpoints are essential to any API, adding workflow into your API helps developers get things done quickly and with fewer lines of code.

As an example, Twilio only supported conference calls by putting each caller on mute and then managing each caller using the low-level access APIs. The company realized that customers required an affordance to support conference calls at a higher level. The result is that providers were able to remove a lot of hard work from API developers by building in a higher-level conference call API that did the heavy lifting for them.

As an API provider, it is important to look for usage patterns of how your API may be used across a variety of different types of developers. Then begin to build in higher-level APIs that support these workflows, saving them effort and time.

Authentication

Authentication is the process of verifying the identity of a particular API consumer. For websites that are built for humans, this is often a form that asks for a username and password. For APIs, there are a number of options available:

Password-based authentication

With password-based authentication, a username and password is sent on each request. While this may be secured through the use of transport-level security (TLS), software integrations to APIs may stop working if a user resets her password for any reason.

API-key based authentication

Instead of sharing a username and password, APIs can require the use of an API key that is provided by the service. This key identifies the API client and is not directly tied to a user’s password. The API key may be embedded within the URL as part of the request body or in the request header. API keys may be shared with other applications, but revoking access to the API key will cause all applications sharing the key to stop working.

Delegation-based authentication

For scenarios when third-party applications may want to connect to an API on someone else’s behalf (e.g., a third-party Twitter or Facebook client), delegation-based authentication is often the best approach. OAuth is the most popular standard, since it is somewhat like a “protocol of protocols” that allows disconnecting the authentication provider that provides API tokens from the API provider. API tokens are granted and revoked as desired by the user, allowing third-party applications to make API calls on their behalf (i.e., delegated access)

Authorization

Once you have identified yourself through the authentication mechanism, the API must still authorize access to the appropriate data and functional access rules for the system. This may be a direct result of your user account permissions within the API, or the permissions granted through the use of a delegation-based authentication mechanism such as OAuth. Each API endpoint’s authorization requirements should be considered during the API design process to ensure that data and access to critical systems are secured properly while the core functionality of the API is maintained.

Data leakage

Even with the proper authentication and authorization mechanism, your API design can still have security leaks. While this can happen for a variety of reasons, the most common is that APIs are designed for internal consumption only and are eventually promoted to partner or public developers. Sensitive data once thought as accessible only by internal systems is now made available to developers outside of the organization.

As an example, the Tinder API experienced a security breach through data leakage. While its mobile applications did not surface an individual’s exact location, the API did return specific locations within the response payload. This means that any developer had access to an individual’s location, since the data was not removed or scrubbed properly for external consumption. Instead, internal knowledge of another Tinder user’s location was both stored within their backend systems and then made available directly via its API, exposing these users to potential harm.

To prevent data leakage, design your APIs as if you were releasing the API to the public. This includes adding proper authorization for API consumers to grant or revoke access to specific data fields and functional areas of the API as appropriate.

Always use TLS

While some API endpoints may be providing data that isn’t sensitive, it is generally best to use TLS to secure all data in motion. This will help protect authentication credentials, as well as prevent eavesdropping when transmitting sensitive data between the API client and server.

Designing with security is important

As you likely realize by now, if security is an afterthought in your API design, everything from the API documentation to assumptions in implementation may need to be revisited. In addition, improper design of your API by ignoring security implications can result in the sharing of sensitive data and perhaps even exposing unnecessary risk to users.

Milestone 1: First success

Initially, the developer needs to overcome any doubts that the API will solve his needs. This often starts with code examples that allow her to explore the API beyond the interactive documentation.

It is important to remember that during this phase, the developer just wants to see something work. This is often known as TTFHW, “Time to first Hello World” and is a key metric for determining API complexity. The longer it takes to get a developer to her first “win”, the more likely the developer will leave your API and find a better solution. If there are no alternatives for your API, the developer will build his own solution.

Products such as Twilio have made it their goal to onboard developers in less than five minutes. While this takes considerable focus and investment, your goal should be to find the shortest possible route, then keep working to shorten the time as your API matures.

To achieve a low TTFHW, provide concise examples that remove all need for boilerplate code. Look at the following example:

require "stripe"
Stripe.api_key = "sk_test_BQokikJOvBiI2HlWgH4olfQ2"

Stripe::Token.create(
  :card => {
    :number => "4242424242424242",
    :exp_month => 6,
    :exp_year => 2016,
    :cvc => "314"
  },
)

Notice in this example that there is little code to write. Simply fill-in your API key and the credit card credentials to try it out.

Bad example code requires that you write code to use the example. This requires you to learn more about the API before you can try it out. Never require developers to write code to complete an example when first trying out your API.

Milestone 2: Workflow support

After the developers have had some time to acquaint themselves with your API using your easy-to-use examples, the next step is to begin to demonstrate the API based on how they will want to use it.

Workflow examples focus more on achieving specific goals than applying coding best practices. This means that you want to convey clarity over code performance, clear intent over code quality. Use copious inline comments to explain why each step is necessary. Be willing to include hard-coded values for easier reading. Choose variable and method names that make it easy to read. Focus more on behavior than handling errors at this point.

It is important to note that while these examples will be more complex than those found from the first milestone, they shouldn’t exceed the height of the screen. The examples need to be short enough to explain the concepts but not too long that they require considerable time to understand. It is often best to demonstrate scenarios that are easily understood and likely map to your customer needs.

Milestone 3: Production-ready integration

Once the developer has followed your simple examples and then tried some workflows, the final step is to help her understand how to integrate your API into her production environment. This includes how to catch errors to help developers properly troubleshoot their integration code. It also includes demonstrating how to catch and recover from bad data provided by end users. Finally, if you are enforcing rate limiting, then show not only how to obtain the current rate limits for their account, but also provide tips for reducing the number of API calls required.

Step 1: Identify the Participants

The first step is to identify the participants, sometimes called actors, who will interact with the API. Unlike a user interface, an API design must factor in both humans and non-humans who may interact with your API. Some examples include:

• System administrators (i.e., your company’s administrators)

• Account administrators (i.e., your customer’s administrators)

• Users of the system, including their various roles (e.g., users, managers, and moderators)

• Internal or external software

• Other Internet-connected devices

For each participant, you will capture their name and perhaps a short description of what they do (see Table 3-1).

By considering the different types of participants, it will help you consider your API design from a variety of intended uses.

Step 2: Identify the Activities

Activities are the outcomes that your participants will expect your API to provide. These activities focus on the job to be done, not how to do it. We will capture the steps required to accomplish the activities in our next modeling process.

Although your API may focus on only one activity, most APIs must support more than one activity to deliver value to developers. If you find yourself with only one activity, try asking what each participant is trying to achieve. Most likely, you will find more than one activity will be required of your API once you evaluate the activities for each of your participants identified in step 1.

For each activity, capture a short name, description, and the participant(s) that will be involved in the activity (see Table 3-2).

Step 3: Separate the Activities into Steps

Once you have a list of activities, it is time to break them into steps. Each step may involve one or more participants but must be executed by a single participant at a time.

You may find that you may be missing some details about how an activity should be performed. This is an indicator that you should involve one or more subject matter experts (SMEs) who can provide greater insight and details into how the system should work. These experts may be business analysts, customers, and/or quality-assurance teams familiar with the requirements.

For each activity step, capture the activity name, a short name for the step, a description, and a list of participants who may perform the step. Table 3-3 is an example of documenting the activity steps for the “manage project tasks” activity in the previous modeling step.

You may notice that you see recurring patterns or steps across activities. That is expected and may provide insight into possible API design and/or code reuse.

Step 4: Identify the Resources and Candidate APIs

Once you have the activities and steps identified, you will begin to see specific business entities emerge. These are your resource candidates that may become actual API resources. Others may appear to be resources but may instead be activities that will be performed outside of the API itself and therefore won’t become actual API resources. Since this is the modeling phase, that is perfectly fine. Our API design process will help filter out these unnecessary items.

Because this is the modeling and not the design phase, we do not want to focus on the HTTP specifics. Instead, focus on the high-level API only. The details of how you will implement the API, including the specific URLs, HTTP verbs, and response codes will emerge as we move into the full API design process. Table 3-4 is an example of how to document the project tasks API that we have identified as a result of our API design model.

Step 5: Validate the API Model

Once you have an API model defined, you can use the model to validate that it meets the requirements of internal developers, partner or public developers, and the end user. There are three techniques that can help validate your API model:

The API walkthrough

If you have mobile or web mockups, walk through each screen to determine which API will be used to satisfy the data being displayed or functionality being performed. Revisit any areas that you may have missed to ensure that you have a complete API model.

Use-case validation

Using existing use cases, step through each one to identify any gaps in your API. This is especially useful if user interface mockups haven’t been completed yet, or if some of the API will not have a user interface associated with it. This is often the case for APIs designed for machine-to-machine integration rather than end-user functionality.

Business-process diagrams

For APIs that will be supporting business-workflow processes, first diagram the workflow(s). For each step in the workflow, map your API model to each step to ensure that you can accomplish all necessary aspects of the workflow.

As you validate your APIs, look for methods that are missing. You may also want to make notes about APIs that have dependencies on other APIs or that may experience heavy usage. While not necessary, this may guide some of your decisions as you move into the design and development phases.

HTTP Is Request/Response

HTTP is a request/response protocol. The HTTP client contacts a server and sends a request. The server processes the request and returns a response indicating a success or failure. It is important to note that HTTP is stateless, which means that every request must provide all of the details necessary to process the request on the server.

Uniform Resource Locators (URLs)

Uniform resource locators, or URLs, provide a address of where to locate a resource, such as a web page, image, or data, from an API. A URL is divided into the following parts:

Scheme

How we want to connect, (e.g., HTTP [unsecure] or HTTPS [secure]).

Hostname

The server to contact (e.g., api.example.com).

Port number

A number ranging from 0 to 65535 that identifies the process on the server where the request is to go (e.g., 443 [optional, defaults to 80 for HTTP and 443 for HTTPS]).

Path

The path to the resource being requested (e.g., /projects [default is /, which indicates the homepage]).

Query string

Contains data to be passed to the server. Starts with a question mark and contains name/value (e.g., foo=bar) pairs, using an ampersand as a separator between (e.g., ?page=1&per_page=10).

HTTP Verbs

HTTP request verbs indicate the type of action being requested from the URL. For APIs, they are often one of the following:

GET

Retrieve a collection or individual resource.

POST

Create a new resource or request custom action.

PUT

Update an existing resource or collection.

DELETE

Delete an existing resource or collection.

HTTP Requests

A client request is composed of the following parts:

Request verb

Informs the server about the type of request being made (e.g., retrieve, create, update, delete, etc.)

URL

The universal address of the information being requested

Request header

Information about the client and what is being requested in name:value format

Request body

The request details (may be empty)

The following is an example HTTP request with no request body:

GET http://www.oreilly.com/ HTTP/1.0
Proxy-Connection: Keep-Alive
User-Agent: Mozilla/5.0 [en] (X11; I; Linux 2.2.3 i686)
Host: oreilly.com
Accept: image/gif, image/x-xbitmap, image/jpeg, */*
Accept-Encoding: gzip
Accept-Language: en
Accept-Charset: iso-8859-1, *, utf-8

The following are examples of API requests:

GET /accounts

Retrieves all accounts in the accounts resource collection

GET /accounts/{id}

Retrieves a specific account by the given ID

POST /accounts

Creates a new acccount

PUT /accounts/{id}

Updates an account by the given ID

DELETE /accounts/{id}

Deletes an account by the given ID

HTTP Responses

A server response is composed of the following parts:

Server response code

A number indicating if the request was successful or not

Response header

Information about what happened in name:value format

Response body

Contains the response payload, often HTML, XML, JSON, or an image (may be empty)

The following code is an example HTTP request with an HTML response body:

HTTP/1.1 200 OK
Date: Tue, 26 May 2015 06:57:43 GMT
Content-Location: http://oreilly.com/index.html
Etag: "07db14afa76be1:1074"
Last-Modified: Sun, 24 May 2015 01:27:41 GMT
Content-Type: text/html
Server: Apache

<html>...</html>

Response codes are grouped into families, with the 2xx response codes indicating success, 4xx response codes indicating that the client failed to format the request properly, and 5xx response codes indicating a server error. The following are the most common server response codes used for web APIs:

200 OK

The request has succeeded.

201 Created

The request has been fulfilled and resulted in a new resource being created.

202 Accepted

The request has been accepted for processing, but the processing has not been completed.

204 No Content

The server has fulfilled the request but does not need to return a body. This is common for delete operations.

400 Bad Request

The request could not be understood by the server due to malformed syntax.

401 Unauthorized

The request requires user authentication.

403 Forbidden

The server understood the request, but is refusing to fulfill it.

404 Not Found

The server has not found anything matching the requested URI.

500 Internal Server Error

The server encountered an unexpected condition which prevented it from fulfilling the request.