Learning API Styles

Chapter 4. Webhooks

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 7th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at [e-mail].

Although HTTP is a message protocol for synchronous communication, there are certain cases where it can be used to send messages between two systems asynchronously. The described use case is used by webhooks. A webhook is a technology that supports lightweight event-driven communication by allowing sending (source) systems to push messages to receiving (destination) systems asynchronously, but it uses the HTTP protocol to do so.

In this chapter, you will be introduced to the fundamental concepts of webhooks, their origins, classification, strengths and weaknesses, and when to use them. Besides conceptual knowledge, you will gain practical skills to implement, secure and document webhook APIs.

What Is A Webhook?

A webhook goes by many names such as, HTTP push API, callback API or reverse API. The purpose of a webhook is to send a message from a source to a destination system without destination system explicitly requesting the message. The most common implementation uses HTTP’s POST method, but you could also have a webhook that uses the HTTP GET method, which is less common. The webhook messages are serialized — usually text-based encoded as JSON or XML, etc. — and are sent in real-time after the occurrence of an event in the source system — following a message flow from the source system to the destination system. The destination system processes the message and takes action. Note that the destination system does not request the message; instead, the message is pushed to it by the source system. As such, a webhook can be interpreted under the common client-server model introduced in the network chapter, where the client is defined as the side that initiates the connection and the server as the side that is awaiting connections from a client.

Note

It’s important to note that webhooks’ source and destination systems are typically machines and direct human interaction is not involved. In the context of the intuitive understanding of web applications, servers are identified as machines, leading to webhooks sometimes being referred to as server-to-server communication. This chapter uses the more general terms source and destination to avoid confusion.

Webhook is a lightweight method for event-driven communication because it minimizes communication overhead. The data size is limited to the content of the triggered event.

Is Webhook an API?

A webhook is an API, but it’s its peculiar flavor of it. A webhook only works in collaboration between the source/emitting system and the destination/receiving system. The responsibility of the source system is to send a message by executing an HTTP’s POST or GET request that points towards a URL of the destination system API. The message is sent from the source system after the occurrence of an event. On the other hand, the responsibility of the destination system is to act upon the received message. In essence, the destination system exposes webhooks API that will accept HTTP requests and process them. Moreover, what makes a webhook different from an API is that a webhook is designed for a specific function — sending notifications about events. Whereas the API can fulfill many functions.

Origins Of Webhooks

The term web hook (with the space between two words) is attributed to Jeff Lindsay, who in 2007 shared it in his blog web hooks to revolutionize the web. Webhook is an answer to a vision of websites acting as data sources that could be remixed and reused. The inspiration for webhook came from Unix’s pipes. A pipe in Unix allows a series of programs to be chained together by their standard streams — where the output of program A becomes an input to program B. Furthermore, Jeff noticed that the data flow between piped programs in Unix is synchronous, but the nature of data flow on the web is asynchronous. Hence, website piping cannot rely on synchronous techniques such as data polling. Jeff’s thinking about HTTP POST as if it was Unix’s stdout, where the POST’s payload would be treated as event data and the destination URL as a web script to handle the data, enables the vision of piping the web and opening the gates of systems integration.

Webhook became an alternative to polling. In polling, a client repetitively makes a request to the server to check for availability of new data. Then, the server holds the request (doesn’t return the response immediately) until the new data is available. Finally, the client receives the latest data from the server, and the cycle repeats.

Incoming And Outgoing Webhooks

Webhooks are classified into two categories: incoming and outgoing webhooks — this classification depends on the point of view. Incoming webhooks are used to receive data from a system, whereas outgoing webhooks are used to send data to a system. Figure 4-1 shows webhooks message flow between three systems. We’ll focus on the message flow from the perspective of System B — the central system. Namely, System B is integrated with System A via an incoming webhook — System B receives the message from System A. Meanwhile, System B and System C integration happens via an outgoing webhook — System B pushes the message to System C.

Figure 4-1. Incoming vs Outgoing Webhook

Tip

A good way to think about incoming and outgoing webhooks is to think about message flow from a system’s perspective. A message arrives at the system in an incoming webhook, whereas in an outgoing webhook, a message leaves the system.

Integration And Dataflow

Let’s zoom into the Figure 4-1 diagram and apply the incoming webhook pattern for a fictitious business named books.example.com. The result of this operation will create a diagram illustrated in Figure 4-2 — we’ll use this diagram to explain system integration and webhook data flow.

Figure 4-2. Incoming Webhook

In this example, books.example.com is a fictitious online bookstore. Its core domain is selling books, and its strategic partner is warehouse.example.com — a fulfillment center. As illustrated in Figure 4-2 books.example.com exposes two API POST endpoints for handling incoming webhooks. When warehouse.example.com receives information about a new order from books.example.com, it starts processing it. While the order is being processed, it generates many events presented as events a, b, and c. Additionally, both systems share a WEBHOOK_SECRET that is used to sign and verify the message’s payload — more on this in the section about webhook security. This security measure ensures that the message is trusted.

Independently of which external system you will work with, the procedure for integrating webhooks will follow steps from Example 4-1.

Example 4-1. Webhook Integration And Data Flow

1. Destination/Receiving System:
  * implements an HTTP endpoint through which it will receive messages from the source
    system.
  * contains a secret that is also known by the source system.
2. Source/Emitting System:
  * registers receiving system's endpoints.
  * contains the secret that is also known by the destination system. This secret
    is used to sign the message's payload.
  * triggers the HTTP method against the registered endpoint.
3. Destination/Receiving System:
  * verifies the message's payload using the shared secret key.
  * processes the message and takes action.
  * returns an HTTP response to the source system.

Tip

If you apply Example 4-1 to the example described in Figure 4-2, then the destination system that receives income webhook message will be books.example.com, and the source system that emits messages will be warehouse.example.com

Implementation

In this section, we’ll turn webhooks theory into practice and expand the offerings of weather forecast service (WFS Chapter 2) about webhook implementation. We’ll create two different implementations of an echo webhook endpoint — v1 and v2, as shown in Example 4-2. The purpose of this endpoint is to receive a payload from a source system and print it out.

Note

The code snippets below are trimmed for brevity. You’ll know that the code is trimmed (folded) when you see the …​ (three dots) notation. We do it to leave only the most essential code you need to comprehend the solution.

Example 4-2. src/django/app/config/urls.py

...
from core.api.webhook.v1.views import WebhookStandardView
from core.api.webhook.v2.views import WebhookCustomView

....
urlpatterns = [
    ...
    path("webhook/v1/echo", WebhookStandardView.as_view()), 
    path("webhook/v2/echo", WebhookCustomView.as_view()), 
    ...
]
...

As the name of the endpoints suggests, their purpose is to print+link+ and summary out the data that is sent to them. The implementation has two variants and we’ll follow the first variant.

Class WebhookStandardView() — implements webhook in accordance with standard webhooks specification.

Class WebhookCustomView() — is a custom webhook implementation.

After defining the endpoints, we must implement the logic to process the incoming HTTP requests. The implementation of logic for the first class is shown in Example 4-3.

Example 4-3. src/django/app/core/api/webhook/v1/views.py

...
from standardwebhooks import Webhook 
...
from django.views import View
from django.views.decorators.csrf import csrf_exempt

from config.constants import WEBHOOK_SECRET_B64


@method_decorator(csrf_exempt, name="dispatch") 
class WebhookStandardView(View):
    wh = Webhook(WEBHOOK_SECRET_B64) 

    def post(self, request): 
        try:
            payload = self.wh.verify(request.body, request.headers) 
        except Exception as e:
            return JsonResponse({"errors": [str(e)]}, status=HTTP_400_BAD_REQUEST)

        return JsonResponse(payload) 

import of standardwebhooks library.

The method_decorator() is a decorator that is applied to the WebhookStandardView() class. By default, the Django framework treats this class endpoint as a form, so it protects all incoming data against Cross-Site Request Forgery (CSRF) attack due to csrf_exempt decorator applied to the class view.

The Webhook() class is instantiated with a base64 encoded secret — WEBHOOK_SECRET_B64 — that is used to verify the message’s payload. Its instance is assigned to wh variable, a class attribute.

The class’ post() method is called when the HTTP POST is executed against the endpoint and is exempted from CSRF protection.

The self.wh.verify() method is called to verify the message’s payload. If the verification fails, an exception is raised and JSON response is returned containing the error message.

If the verification is successful, the payload is returned as a JSON response.

Standard Webhooks

Standard webhooks is a project that aims to standardize the way that webhooks are implemented across the IT industry by defining a standard webhook specification. The project proposes a set of guidelines — based on the best practices such as security, compatibility etc. — that should be followed when implementing webhooks. Moreover, it comes with a set of libraries for various programming languages. The one that we are using is Python’s standardwebhooks library.

Security

Implemented in Example 4-3 webhook uses Python’s standardwebhook library. A custom Example 4-4 is also provided as an example to illustrate an approach to webhook implementation.

Common security measures used to protect webhooks include Transport Layer Security (TLS) — for data in transit — and Hash-based Message Authentication Code (HMAC) — we’ll implement the last one. HMAC is a cryptographic hash function that uses a secret key — a cryptographic key — to sign and verify the message’s payload. The key is shared between the sending and receiving systems. The sending system uses the key to sign the message’s payload, and the receiving system uses the same key to verify the message’s payload. The shared secret is shown back in Figure 4-2.

Let’s make theory real. Example 4-4 implements the message’s HMAC verification.

Example 4-4. src/django/app/core/api/webhook/v2/views.py

import base64
import hmac
...
from django.views.decorators.csrf import csrf_exempt
from standardwebhooks import Webhook
...
from config.constants import WEBHOOK_SECRET


@method_decorator(csrf_exempt, name="dispatch") 
class WebhookCustomView(View):
    def post(self, request):
        """
        Method echoes the request body
        """
        if len(request.body) > 20 * 1024: 
            err = "Payload is to big."
            return JsonResponse({"errors": [err]}, status=HTTP_400_BAD_REQUEST)

        wh_id = request.headers.get("Webhook-Id") 
        if not wh_id:
            err = "Missing Webhook-Id header."
            return JsonResponse({"errors": [err]}, status=HTTP_400_BAD_REQUEST)

        wh_received_signature = request.headers.get("Webhook-Signature") 
        if not wh_received_signature:
            err = "Missing Webhook-Signature header."
            return JsonResponse({"errors": [err]}, status=HTTP_400_BAD_REQUEST)

        wh_timestamp = request.headers.get("Webhook-Timestamp") 
        if not wh_timestamp:
            err = "Missing Webhook-Timestamp header."
            return JsonResponse({"errors": [err]}, status=HTTP_400_BAD_REQUEST)

        timestamp = datetime.fromtimestamp(wh_timestamp, tz=timezone.utc)
        offset = (datetime.now(timezone.utc) - timestamp).total_seconds()
        wh_expired = (int(offset) >= 5) 
        if wh_expired:
            err = "Request is too old."
            return JsonResponse({"errors": [err]}, status=HTTP_400_BAD_REQUEST)

        # Generate standard webhook's signature
        payload = request.body.decode("utf-8")
        secret = WEBHOOK_SECRET.encode("utf-8")
        sig_scheme = f"{wh_id}.{int(wh_timestamp)}.{payload}".encode("utf-8") 
        signature = hmac.new(secret, msg=sig_scheme, digestmod="SHA256").digest() 
        signature_b64 = base64.b64encode(signature).decode("utf-8")
        wh_expected_signature = f"v1,{signature_b64}"

        # Compare signatures
        if wh_received_signature != wh_expected_signature: 
            err = "HMAC mismatch"
            return JsonResponse({"errors": [err]}, status=HTTP_401_UNAUTHORIZED)

        try:
            data = json.loads(request.body) 
        except json.JSONDecodeError:
            return request.body

        return JsonResponse(data)

The method_decorator() is a decorator that is applied to the WebhookCustomView() class. By default, the Django framework treats this class endpoint as a form, so it protects all incoming data against Cross-Site Request Forgery (CSRF) attack. Applying this decorator exempts incoming data from CSRF protection.

The code snippet checks if the payload’s size is within the limit of 20 KiB — you will adjust this value to your needs. If not, an error message with HTTP 400 status code is returned. This check prevents from handling big payloads that might lead to a Denial of service attack (DoS attack).

These statements check for the presence of Webhook-Id, Webhook-Signature and Webhook-Timestamp headers — mandatory headers in standard webhooks specs. If any of them is missing, an error message with HTTP 400 status code is returned.

The webhook is checked for expiration age. If the timestamp is older than 5 seconds, then the request expires, and the response is returned with an HTTP 400 status code and an error message. It’s important to notice that checking for expiration date prevents replay/repeat attack.

This line generates the message’s signature scheme (sig_scheme). The scheme is a concatenation of the webhook’s ID, timestamp, and payload. This is required by standard webhook signature scheme. The timestamp in a signature is used to avoid bypassing the expiration age check, and the webhook ID is used to verify the identity of the webhook.

Next, the codeblock generates the request’s symmetric signature. This procedure involves the calculation of the HMAC signature (signature) from the signature scheme (sig_scheme), then converts it to base64 value (signature_b64), and finally concatenates it with the version number (wh_expected_signature).

This line compares the received and expected signatures. If they don’t match, the response with HTTP 401 status code and an error message is returned. This check ensures that the message’s payload is not tampered with.

Try-except block checks if the arrived HTTP POST data is JSON-encoded. If not, then the request’s body is returned. Otherwise, it returns data as a JSON response.

Webhooks Security

We made a few assumptions while writing this code that impacted implementation. First, we treat the webhook endpoint as an internal — intended to be consumed within an organization. Because of that, we are exposing detailed error messages regarding missing headers. However, if the endpoint is designed to be consumed by external systems, we should not expose detailed error messages. Second, we are using a shared secret to sign and verify the message’s payload to prevent the message from being tampered with. Sharing secrets might be a security risk if it’s not handled properly. Finally, we are using the timestamp to check the message’s age. The timestamp prevents the message from being replayed — replay/repeat attack. Our implementation calculates the timestamp after checking for the presence of the timestamp header. However, it should start at the moment of request handling. For readability, we decided to initiate time in the validation if statement and accept the loss of the time needed to execute validation logic that precedes the if statement.

Let’s put implementations of Example 4-3 and Example 4-4 to the test. This test will simulate server-side message push between source and destination systems. Example 4-5 lab tests standard and custom webhooks implementations. Each implementation is chosen based on the --endpoint CLI argument. The CLI receives a payload that is sent to the webhook’s endpoint with HTTP POST method. If the test is successful, we expect to see the request payload printed to the console.

Example 4-5. Testing Webhooks

# Lab setup 
cd src/django
docker compose build --build-arg UID=$(id -u) --build-arg GID=$(id -g)
docker compose up --detach --wait

# Set lab's environment variables 
EPOCHSECONDS=$(date +%s)
PAYLOAD='{"type": "dummy.event", "timestamp": '$EPOCHSECONDS', "data": {"echo": "test"}}'

# Create an event addressed to v1 echo webhook 
docker compose exec app python manage.py app_event --endpoint webhook/v1/echo \
  --payload "$PAYLOAD"
POST webhook/v1/echo
{"type": "dummy.event", "timestamp": 1709111631, "data": {"echo": "test"}}

# Create an event addressed to v2 echo webhook 
docker compose exec app python manage.py app_event --endpoint webhook/v2/echo \
  --payload "$PAYLOAD"
POST webhook/v2/echo
{"type": "dummy.event", "timestamp": 1709111631, "data": {"echo": "test"}}

Execute the lab setup instructions: navigate to the lab’s directory, build the lab’s service images, and start service containers.

Prepare EPOCHSECONDS and PAYLOAD environment variables.

Run the first test, which tests the incoming webhook implemented according to standard webhook specification.

Run the second test, which tests incoming webhooks based on the custom implementation of standard webhook specification.

The Example 4-6 contains an implementation of tests from Example 4-5. A detailed explanation of the code is listed below.

Example 4-6. Testing Webhooks Implementations

...
from standardwebhooks import Webhook

from config.constants import WEBHOOK_SECRET_B64
...


class Command(BaseCommand):
    ...
    wh = Webhook(WEBHOOK_SECRET_B64)

    def handle(self, *args, **options): 
        endpoint = options["endpoint"]
        msg_id = options["msg_id"]
        payload = options["payload"]
        timestamp = datetime.fromtimestamp(time.time(), tz=timezone.utc)
        ...
        if endpoint == "webhook/v1/echo":
            return self.__webhook_v1(endpoint, timestamp, payload, msg_id) 

        return self.__webhook_v2(endpoint, payload, msg_id) 

    def __webhook_v1(self, endpoint, timestamp, payload, msg_id): 
        signature = self.wh.sign(msg_id, timestamp, payload) 

        headers = { 
            "Webhook-Id": msg_id,
            "Webhook-Signature": signature,
            "Webhook-Timestamp": timestamp,
        }
        ...
        request = factory.post(endpoint, payload, headers=headers, ...) 
        ...
        response = view(request)

        return f'POST {endpoint} {response.content.decode("utf-8")}' 

    def __webhook_v2(self, endpoint, timestamp, payload, msg_id): 
        signature = self.wh.sign(msg_id, timestamp, payload) 

        headers = { 
            "Webhook-Id": msg_id,
            "Webhook-Signature": signature,
            "Webhook-Timestamp": timestamp,
        }
        ...
        request = factory.post(endpoint, payload, headers=headers, ...) 
        ...
        response = view(request)

        return f'POST {endpoint} {response.content.decode("utf-8")}' 

Executing Example 4-5 triggers the handle() method. The method returns the webhook response to stdout.

Depending on the endpoint parameter, the handle() method calls either the __webhook_v1() or __webhook_v2() method that will test the message flow from the source to destination systems.

The signature variable holds the message’s signature generated from standard webhooks’ signature schema.

The headers variable holds appropriate to the implementation message headers.

The system that produces an event — outgoing webhook — pushes the message to the system that consumes it — incoming webhook — by executing the HTTP POST request.

The response containing HTTP POST method, endpoint and payload is returned to stdout.

Documentation

The OpenAPI specification is a language-agnostic interface that allows developers to describe HTTP-based APIs. On the one hand, the document is used by developers to capture API’s functionality. On the other hand, the document is used by API consumers to understand how to work with the described API. The document is a contract between the API provider and the API consumer.

Because a webhook is an HTTP method, it can be documented with the OpenAPI specification. Example 4-7 shows an OpenAPI specification used to document webhook/v1/echo endpoint.

Example 4-7. OpenAPI Specification For Webhook v1

openapi: 3.1.0 
info:
  title: Webhook API
  description: Webhook API for receiving notifications
  version: v1
servers: 
  - url: http://localhost:8000/
    description: Development server
tags:
  - name: Webhook
    description: Operations related to webhooks
paths:
  /webhook/v1/echo: 
    post: 
      tags:
        - Webhook
      summary: Receive a webhook notification
      description: Receive a webhook notification
      parameters: 
        - name: Webhook-Id
          in: header
          required: true
          schema:
            type: string
        - name: Webhook-Signature
          in: header
          required: true
          schema:
            type: string
        - name: Webhook-Timestamp
          in: header
          required: true
          schema:
            type: string
      requestBody: 
        description: JSON object with the notification data
        required: true
        content:
          application/json:
            schema:
              type: object
              properties:
                type:
                  type: string
                  description: The type of the event
                timestamp:
                  type: string
                  description: The timestamp of the event
                data:
                  type: object
                  description: The data associated with the event
      responses: 
        '200':
          description: OK
          content:
            application/json:
              schema:
                type: object
        '400':
          description: Bad Request
          content:
            application/json:
              schema:
                type: object
                properties:
                  errors: 
                    type: array
                    description: The error message
                    items:
                      type: string

The openapi — Holds the version of OpenAPI specification used to document the API.

The servers — Lists the servers where the API is hosted. In this case, it’s the development server. This field is optional, but it’s good practice to include it.

The paths — Lists the API’s endpoints. In this case, it’s only one endpoint — the /webhook/v1/echo endpoint.

The post — Describes the HTTP POST method used to send data to the endpoint that it describes.

The parameters — Lists the required parameters in the request. In this case, the necessary parameters are headers: Webhook-Id, Webhook-Signature, and Webhook-Timestamp.

The requestBody — Describes the request body that the endpoint expects. In this case, it’s a JSON object that contains type, timestamp, and data keys — where data is an object type and other keys are string type.

The responses — Lists the possible responses the endpoint can return. In this case, it can return a 200 — ok response — or a 400 — bad request — response.

The errors — Hold unsuccessful response will contain the errors key with a list of error messages that the endpoint can return.

Presented in Example 4-7, OpenAPI specification is of great value for a developer. But, from the consumer perspective, the ability to work and use is more important than browsing textual specifications.

Below is the list of tools that can be used to work with the OpenAPI specification. Depending on your needs, you will choose one or another.

Swagger Online Editor — is an online web-based editor that allows you to write and validate OpenAPI specifications. What’s so good about it is the live preview of the OpenAPI specification. To see for yourself, copy the specification from Example 4-7 and paste it into the editor. The outcome of this lab should produce a result that looks like the one illustrated in Figure 4-3.

Warning

A general warning about online editors: although they are convenient to work with, you might not be certain about their sources. Online tools should not be suitable for working with sensitive data if not trusted. If you need to work with sensitive data, you should use alternatives that can work locally.

Figure 4-3. Swagger Editor Next

Swagger UI — If you work locally and you need to test documented with OpenAPI specification, then the Swagger UI tool allows you to do that. Execute instructions from Example 4-8 to see how it works. The outcome of this lab should produce a result that looks like the one illustrated in Figure 4-4. Remember to visit http://localhost:8888 in your web browser.

Example 4-8. Displaying OPEN API Schema With Swagger UI

docker run --rm --publish 8888:8080 \
  --env SWAGGER_JSON=/usr/src/openapi/v1-schema.yaml \
  --volume ${PWD}:/usr/src/openapi swaggerapi/swagger-ui

Figure 4-4. Swagger UI

Redocly — If you need to generate documentation from the OpenAPI specification, you can use Redocly. Execute instructions from Example 4-9 to create an HTML page that you can share. Completing this lab should produce an HTML document that looks in Figure 4-5 — open index.html in your web browser.

Example 4-9. Redocly CLI

docker run --rm --user $(id -u):$(id -g) --volume ${PWD}:/data redocly/cli \
  build-docs /data/v1-schema.yaml --output /data/index.html
Bundled successfully in: /data/index.html (46 KiB) [⏱ 3ms].

Figure 4-5. Redocly

Advantages and Disadvantages of Webhooks

Advantages

Webhooks have several advantages over traditional APIs. The most significant advantages are:

No polling

In polling, the request to get the latest data originates from the client. In server-side push, the message being sent to the client comes from the server. The server-side push eliminates the need for message polling — frequent requests sent to the server to check for the new data — thereby simplifying the client application and reducing the server’s resource utilization and network usage. Webhooks work in a similar manner to server-side push as they eliminate the need for polling.

Real-time operations

The triggering point for the message — execution of HTTP’s POST or GET method — is the occurrence of an event. Although the event happens in real-time, it still takes time for the request to propagate over the network. Even though the event is happening in real-time, then the message transportation is in near-real time at best.

Use of the HTTP protocol

Developers generally understand the HTTP protocol well. Moreover, many programming languages and frameworks support the protocol, making it fast to implement and set up. Last, the message encapsulated in HTTP protocol propagates over the internet, firewalls and various networks (organizations) and devices.

Asynchronous communication

The nature of the communication in webhooks is asynchronous. The message flows from the source to the destination system. However, processing of the message on the destination system can be executed either synchronously or asynchronously. In synchronous processing, the destination system receives the message and starts processing. The processing takes time, and during this time destination system holds the request until it’s completed. Once the message is processed, the destination system returns the response to the source system. In asynchronous processing, the destination system receives the message and sends a response to the source system immediately, and processing of the message happens outside of the request-response flow.

Message fan-out

With the implementation of a publish-subscribe pattern, webhook messages can be broadcast to multiple destinations. In this case, webhooks would send the same message from the webhook source system (producer) to all webhook destinations (subscribers), ensuring that all active destination systems are notified about the same event. Contrasting pub-sub implementation with tools like RabbitMQ to webhooks, the difference is that in RabbitMQ, the producer and consumers are logically decoupled by the broker. In the case of webhook, there is no broker, and the webhook source system knows the specific locations of the webhook destination systems through the webhook URLs, therefore coupling them logically. Moreover, in webhook the message is delivered in one-to-many fashion — one producer and many consumers, whereas in RabbitMQ it can be many-to-many — multiple producers and consumers. The Final point is the communication angle between webhook and classic pub-sub systems. In outgoing webhook — due to the use of HTTP protocol — the communication is one-to-one.

Disadvantages

As good as webhooks are, they have some disadvantages. The most significant disadvantages that you must think of are:

Reliability

While working with webhooks, you need to know that they inherit the brittleness of the HTTP protocol. You need to consider unreliable network (request timeouts), application errors (300, 400 and 500 status codes), authentication errors and the effect of duplicated requests on the application.

Security

Another disadvantage of webhooks is that they can be used with the unencrypted protocol (HTTP). If you control the destination system — message receiving end — application, then you could implement TLS by yourself. On the other hand, if the source system — message-sending end — application is out of your control, then you must rely on the sending system to implement TLS. Moreover, you must consider the security of the message’s payload. The message’s payload can be tampered with, and the message can be sent from an untrusted source. To mitigate these risks, you can use: HMAC with shared secret — symmetric and asymmetric key — to sign and verify the message’s payload; use timestamp to check the message’s age; and use some forms of authentication such as basic authentication, JWT, etc. Finally, you can use the IP allow list to limit the source of the message.

Maturity

Although the technology was introduced in 2007 — 17 years at the moment of writing this book — it’s still not standardized. As described, the standard webhooks project tries to do that by unifying how to work with webhooks.

When To Use Webhooks?

Described below are use cases where implementing webhooks is a good idea:

Integrating with external systems

Webhooks are a good choice when you need to act in real-time on events from an external system. For example, when a new order is created in an e-commerce system, a webhook can be used to notify the warehouse system to start processing the order.

Automating processes

Webhooks can be used to automate processes by triggering actions. For example, you trigger the deployment pipeline when a new commit is pushed to the repository.

Notifying or logging

When you need to notify users about events that are relevant to them that happen in your system. For example, when you update the monitoring page of your services, a webhook can be used to notify the users about the change.

Exercises

1. In the chapter’s section titled the integration and dataflow, we described standard webhook integration process. Now, it’s your turn to implement it. You will implement incoming webhooks between WFS and github.com. Your task is to fork the repository of this book and create a webhook that will notify the WFS’s endpoint /webhook/v3/echo (callback endpoint) about any event that happens to the repository — e.g. push, or repository description update. Hint: to solve this exercise, use ngrok CLI to expose the WFS local server to the internet. The solution to this exercise can be found in this book repository.

2. In the custom implementation of standard webhook we manually created the standard webhook signature in python. Your task is to implement standard webhook signature but as a shell script. Once you’ll have the signature create curl command that sends POST data to /webhook/v1/echo and /webhook/v2/echo endpoints. The solution to this exercise can be found in this book repository.

Summary

This chapter introduced the concept of webhooks and its origins. It described that a webhook is an HTTP callback that uses HTTP’s POST or GET methods. The chapter explains the webhook integration process and classification between incoming and outgoing webhooks. The theoretical knowledge of webhooks was put into practice in the chapter’s exercise designed to practice the webhook integration process and in the chapter’s labs, which implement webhooks according to standard webhook specification with a Python library standardwebhooks — implementation v1. Implementation v2 shows how to manually secure the webhook message’s payload with HMAC. Furthermore, the chapter shows how to document a webhook with OpenAPI specification and work with it using swagger online editor, swagger UI and redocly. Finally, the chapter analyses the advantages and disadvantages of webhooks and in what use cases to use them.