Transforming Industry Through Data Analytics

Chapter 3. Edge-to-Enterprise Analytics

The technical challenges brought on by the advent of the IoT require new system architectures in order to capture and process the data. The first challenge is capturing all of the data from the edge; in other words, gathering information from the IoT and devices and transferring it to the enterprise datacenter. When you consider there are potentially millions of devices involved, it quickly becomes apparent that making direct connections from the edge devices to the enterprise datacenter is not a workable solution.

Additionally, depending on the use case, the number of edge devices can change, up or down, over time as new devices come online and others are retired. Even devices that will persist in sending data can drop off the network and come back online later. To address this, a transfer layer (commonly called an IoT gateway) is inserted between the edge devices and the enterprise datacenter, as depicted in Figure 3-1.

Figure 3-1 is, admittedly, a very simple diagram, but it illustrates the point. The message brokers between the edge and the enterprise act as a temporary holding area. Edge devices transfer data to the brokers as they generate it, and the enterprise consumes it from the brokers. The enterprise communicates only with the brokers. Each edge device needs to know only the address of the broker. The message broker needs to be highly available, fault-tolerant, resilient, and capable of handling large volumes of data (Apache ...