Learning Apache OpenWhisk

Chapter 11. Using Kafka with OpenWhisk

OpenWhisk can be considered an event-based system in which some event sources produce events that are processed by actions. So far we’ve looked at event sources like HTTP requests, handled by web actions and direct action invocations originated by other actions.

Those are only two of many possible event sources. The strength of OpenWhisk is that you can plug in as event sources other internet-based services. For example, an event source can be Cloudant notifying of database changes, GitHub of notifying commits; or Slack announcing messages. Packages for managing those sources are included with OpenWhisk.

Event-based systems are frequently built using Apache Kafka. Kafka is a popular solution for event processing because it has many desirable properties for building event processing systems. First, it is scalable and distributed, so it can grow to handle millions of events per second. You deploy Kafka in clusters, and you can add more servers to spread the load among them. Second, it is durable, in the sense that notifications of events (also called messages) are not thrown away but can be saved for later reuse. Third, it is replicated and partitioned, so messages are not stored only on one server; there are copies in different servers.

OpenWhisk provides a /whisk.system/messaging package to manage messages. It is a frontend package for using Kafka, similar to the /whisk.system/cloudant package, a frontend to CouchDB and Cloudant.

In this ...