Scalable Architecture for the Internet of Things

Chapter 4. Mainflux

This chapter focuses on Mainflux. We will provide an overview of the architecture and discuss steps for setting up the ecosystem, event management and system dashboards, the application integration layer, and security. We will also build an end-to-end example.

Introduction to Mainflux

Mainflux is a modern, highly secured open source and patent-free IoT cloud platform written in Go, based on a set of microservices. It allows device, user, and application connections over various network protocols, like HTTP, MQTT, WebSocket, and CoAP, making a seamless bridge between them.

It is used as the IoT middleware for building complex IoT solutions. As is often the case with these platform systems, Mainflux exposes southbound API to connect devices and northbound API to connect applications, then it allows message routing between them. The whole bottom-to-top system (device, Mainflux platform, and application) together forms what is called a “vertical solution,” very specific to the given use case.

Figure 4-1 depicts a typical IoT system, and shows where Mainflux sits in the overall architecture.

Figure 4-1. Location of Mainflux inside the IoT system

We can see that Mainflux is the middle block that does a lot of the heavy lifting. The IoT product designer can reuse this block and focus only on the leftmost part (connected devices—that is, the physical design, firmware, etc.) and rightmost part (the end-user application—that is, the GUI application that controls the device).

Mainflux will connect the device and application and relay messages between them in a highly secure manner, while offering a lot of other services needed for device and application management.

To put it in terminology often applied when describing IoT systems, Mainflux offers multiprotocol device-agnostic message relay with distributed time-series data storage and multiuser, multitenant device and application management middleware.

Overview of the Architecture

Mainflux is used as middleware, meaning it employs a SW server (a set of servers, in fact) that provides several functionalities and services that are usually needed in designing IoT applications.

Figure 4-2 lists the services offered by Mainflux. They are as follows:

• Messaging bridge (HTTP Sender, MQTT and WS, CoAP)—relays messages between devices and applications

• System manager (Manager)

• Device manager (Device)—accepts device connections on southbound interface

• Application manager (App)—accepts application connections on northbound interface

• User manager (Mainflux UI, Mainflux CLI)—provides user management for the applications

• Time-series storage engine (Message Reader, Message Writer, Cassandra)—stores and queries measurements data points in the time-series format

• Complex Event Processing (uses the system event bus NATS)—consumes the incoming time-series streams and can automatize triggers and actions based on a configurable set of rules

Figure 4-2. The key constituent elements of Mainflux

Message Relay and Multiprotocol Bridge

Mainflux is thus a messaging bridge to interconnect applications and devices.

Both applications and devices can be context providers and/or context consumers. For example, in one scenario one device can be a temperature sensor (provides temperature measurements) and send measurements to Mainflux, which relays them to some application that presents these measurements in the GUI (thus it consumes them). In some other scenarios, the device can be an actuator (consumes commands)—for example, in the case of a smart lock with an electromotor to lock/unlock the door. An application that commands the lock would be a context provider in that case.

There are of course scenarios in which an application can be both context provider and consumer, or scenarios in which two or more applications (or devices) exchange messages between them. It is not necessarily a direct path from app to device; it can be device-to-device—or as we say M2M (machine-to-machine). It can also be app-to-app—i.e., two applications talking between them, without involving physical devices. And so on.

Mainflux is a “multiprotocol” messaging bridge because it supports several networking protocols that proliferate in IoT world:

• HTTP

• WebSocket

• MQTT

• CoAP

System Manager

In order to explain the role of the System Manager in more detail, it is necessary to present entities that this service creates and manages.

Users

Users consist of the accounts of the people (administrator, application creators, and developers) who will use Mainflux platform. A user is an owner of other entities, so it must be created first. As an owner, a user gets sufficient permissions to manage devices, applications, and channels under its control.

Clients

Clients are a structure that is used to represent an actual client of the Mainflux server. There are two types of clients that connect to Mainflux: devices and applications.

Devices

Devices are usually constrained microcontrollers with a limited power budget. Because of constrained memory (several KB of RAM and ROM) they usually cannot hold heavyweight protocol stacks in their firmware. That’s where MQTT and CoAP come into play—as they are lightweight protocols designed for M2M communications.

Also, on the security side, TLS can be heavyweight. CoAP and LwM2M propose usage of the DTLS instead.

Mainflux exposes MQTT and CoAP APIs to connect devices via these protocols.

If the devices are more powerful machines (like, for example, Raspberry Pi), they can be connected via standard HTTP or WebSocket protocols—as this is usually the case for web applications.

Applications

Applications are run on the servers, and Mainflux developers can create an application similar to the Twitter or Facebook application—this app will use Mainflux in the background.

Applications connect often via HTTP REST or WebSockets, as this is a fast and bidirectional socket, so updates can be pushed from Mainflux (server) to the application (client) in the RT.

Nevertheless, applications can choose any of the other protocols for connections (MQTT or CoAP).

Channels

A channel structure is used to model a communication channel. It is a generic bidirectional message stream representation. Channels are like MQTT topics—several devices or applications can subscribe or publish on a channel. All the values that flow through channels are persisted in the database.

Device Manager

Mainflux holds the internal representation (data structure model) of every device provisioned in the system. This model represents the current state of the device.

Each device that connects to Mainflux thus has its own model (record, structure) in the internal Mainflux database (device registry) that is regularly updated by the device. So, when a device changes internal (physical) state it sends an UPDATE request to Mainflux, and then Mainflux changes its model (record, structure) in its internal database. As a consequence, when the application (or some other device) now queries the database, it finds a new (updated) device model—and this way information (message) is relayed from device to application (and vice versa).

Bearing in mind that Mainflux has all these internal device models and it keeps devices connected to it, we can use Mainflux to manage devices—i.e., to obtain various types of device management information, like:

• How many devices are connected?

• Where are they located?

• What is the firmware version?

• What is the battery status?

• What is the serial number of each device?

We can also use Mainflux to send various commands to devices as a part of the device management process:

• Enable/disable device

• Push firmware updates

• Group devices or change access privileges

• And more

User and application manager

Mainflux is a multiuser and multitenant application management platform. As such, Mainflux developers can create new multiuser applications on top of Mainflux without the need to handle their end users themselves—i.e., Mainflux handles user management for these applications.

In this sense, Mainflux is similar to Twitter or Facebook—creating a Mainflux application should be a process similar to creating a Twitter application. For example:

• User creates developer account on Mainflux.

• User creates new application using Mainflux public API.

• User develops his application that calls Mainflux public API for user creation, messaging, management, etc.

The main point here is that this “Mainflux application” can itself be multiuser. So, Mainflux is a multiuser platform (it can have many user accounts, some of which are of type developers), but it can host multiuser applications themselves.

Time-Series Storage Engine

Mainflux utilizes a Cassandra distributed NoSQL database for storing events in chronological order. For more information about Cassandra, refer to Cassandra: The Definitive Guide, 2nd Edition by Eben Hewitt and Jeff Carpenter (O’Reilly).

Most of the industrial IoT use cases are closely tied to telemetry. Sensors take measurements and then report them to the cloud (Mainflux). Time-series databases are particularly adapted for telemetry data and allow fast complex queries on this type of data.

Complex Event Processing Engine

Complex Event Processing (CEP) is a method of tracking and analyzing (processing) streams of information (data) about things that happen (events), and deriving a conclusion from them. Mainflux uses Apache Spark to perform powerful data transformations, using Cassandra as a sink. For more details about Apache Spark, refer to Learning Spark by Holden Karau et al. (O’Reilly).

Event Management and System Dashboards

The Mainflux messaging subsystem is built around the notion of asynchronous events, like measurements coming from sensors, or commands coming from applications. Messages are accepted by protocol adapters, and distributed toward the message broker. Mainflux uses NATS message broker, a performant pub/sub message broker written in Go.

It is important to note that Mainflux, using the aforementioned NATS message broker, internally constructs a so-called “protocol bridge.” That means that if the client was, for example, subscribed to an MQTT topic mainflux/channels/:channel_id/messages it would receive the message published via HTTP POST on mainflux/channels/:channel_id/messages (assuring that the channel_id is identical in the both cases).

Besides protocol bridging, NATS message broker forwards the messages toward a microservice called “Writer,” which is capable of receiving these events and putting them into the appropriate format to be stored in the Cassandra database.

It is interesting to look at more details of the Mainflux message format. Messages coming from the sensors are expected to be formatted using SenML semantics, a standard for message modeling that demands a closer look. Equally, we will also describe the Mainflux internal message model, in which messages are passed internally in the system.

SenML

In order to conform to existing standards as much as possible, the Mainflux system uses SenML semantics as a message model coming from the clients. SenML is an IETF standard that provides a simple model for retrieving data from sensors and controlling actuators. Its motivation is to define minimal semantics for the data inline and allow for more metadata with inline extensions and links.

Example of a SenML-formatted message:

[
  {
    "bn" : "urn:dev:ow:10e2073a0108006;",
    "bt" : 1.276020076001e+09,
    "bu":"A",
    "bver" : 5,
    "n" : "voltage",
    "u" : "V",
    "v" : 120.1
  },
  {"n" : "current", "t" : −5, "v" : 1.2},
  {"n" : "current", "t" : −4, "v" : 1.3},
  {"n" : "current", "t" : −3, "v" : 1.4},
  {"n" : "current", "t" : −2, "v" : 1.5},
  {"n" : "current", "t" : −1, "v" : 1.6},
  {"n" : "current", "v" : 1.7}
]

Mainflux Message

Mainflux uses the following format to encapsulate the SenML messages coming from the sensors:

type RawMessage struct {
    Channel     string `json:"channel"`
    Publisher   string `json:"publisher"`
    Protocol    string `json:"protocol"`
    ContentType string `json:"content_type"`
    Payload     []byte `json:"payload"`
}

It is each of the adapters that construct this type of message, putting in the Payload field the SenML content that came from the sensors. Each adapter also adds the message metadata, like the ID of the publisher or the protocol over which the message was published.

Normalization

The mainflux Writer microservice subscribes to the NATS broker to receive all events from protocol adapters. Messages come to Writer in the Mainflux internal message format. The Writer microservice parses these “raw” messages and extracts the useful SenML payload.

However, SenML itself uses the semantic for measurement aggregation, so it can be “normalized” into the series of messages—each of which has its own timestamp.

Writer executes this process of SenML normalization, then writes the produced messages in a Cassandra database.

Finally, the message that is stored in the database has the following content:

type Message struct {
    Channel     string
    Publisher   string
    Protocol    string
    Version     int     `json:"bver,omitempty"`
    Name        string  `json:"n,omitempty"`
    Unit        string  `json:"u,omitempty"`
    Value       float64 `json:"v,omitempty"`
    StringValue string  `json:"vs,omitempty"`
    BoolValue   bool    `json:"vb,omitempty"`
    DataValue   string  `json:"vd,omitempty"`
    ValueSum    float64 `json:"s,omitempty"`
    Time        float64 `json:"t,omitempty"`
    UpdateTime  float64 `json:"ut,omitempty"`
    Link        string  `json:"l,omitempty"`
}

Security

The Mainflux system offers fine-grained security based on access control lists and client roles. It also offers encrypted communication, using the latest encryption standards like TLSv1.3. Equally, for encrypting CoAP communication, which is UDP-based, Mainflux uses DTLS.

The main players in maintaining system security are the Manager service and reverse proxy, which is at this moment NGINX.

The Manager service is in charge of providing authentication by checking the validity of JSON Web Tokens (JWTs), as well as providing authorization via the access control lists stored in an internal management database.

The NGINX proxy has several important roles in the Mainflux system:

• Reverse proxy

• TLS termination

• Load balancer

• High availability (scalability)

Regarding the security subject, it is important to underline the role of reverse proxy, which practically forms a firewall, closing all non-public routes for external access. It is also to note that for the simplicity of the system, TLS/DTLS traffic is terminated on entry in the system, as Mainflux deployments are done in the secure private network that lies behind the NGINX firewall.

There are two aspects of Mainflux security that should be examined in greater detail: authentication and authorization.

Authentication (AuthX)

The Mainflux Manager service creates the system entities, but is also in charge of their authentication to the system. During the process of entity creation (so-called provisioning), each user or client (device or application) is issued an authentication token. For simplicity and in order to conform to the standard practices, these tokens are produced in the JWT format.

There is however a slight difference in the way JWTs are generated for the applications, and the ones that are prepared for devices. This important difference is the expiry date, or TTL (time-to-live): JWTs for applications are stateless with short TTL, while JWTs dedicated to devices never expire.

In some way, this approach defeats the stateless nature of the JWT, which should have been periodically refreshed in order to protect the system from leaked keys. These kinds of tokens are never revoked—they just expire after a short period of time and if not refreshed, they are not valid anymore.

However, in the case of constrained devices, refreshing a JWT can be a complex operation, as the device would have to be capable of doing a login procedure. To avoid this requirement, JWTs prepared for the devices should be observed as simple bearer tokens—a secret key that should be flashed in the device’s firmware and protected by physical means (for example, hardware anti-tampering procedures).

Because these tokens are not stateless, they can be revoked—in which case a new JWT for the device will be generated (and has to be reflashed in the device).

As a side note, besides the JWT key, one other file has to be equally flashed into each device—the server’s public certificate, which is needed for establishing TLS encryption.

In the future, the Mainflux team will evaluate options for obtaining even higher security based on “client-side” certificates, which can be used to uniquely authorize devices, and allow encrypted communication. For this purpose, research with crypto-algorithms that are small in size should be leveraged—like, for example, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256. The main requirement is to have reliable encryption while being capable of fitting the certificate in the constrained device’s nonvolatile flash memory.

Other axes of research mentioned by the Mainflux team are exploiting dedicated hardware units for encryption (to relieve constrained microcontroller CPUs from difficult calculations), as well as leveraging blockchains—i.e., immutable distributed ledger properties.

Authorization (AuthZ)

The Manager service is in charge of authorization as well as authentication. To handle authorization, the service first verifies whether the user has sufficient rights to manipulate devices and applications—i.e., it determines whether the user has ownership over them. Second, during the provisioning phase, certain roles can be attached to Mainflux clients—they can be grouped into a communication groups called channels. In Mainflux jargon, a device or an application can be “plugged” into the channel—i.e., connected to it. When they are plugged into the same channel, devices and applications can communicate over it. Any try of an “unplugged” client to post messages or subscribe to a given channel will fail. And only an administrator user, who owns devices, applications, and channels, can interconnect them.

In other words, it can be said that Mainflux uses access control policies based on system roles. Clients are grouped, and if they belong to a certain group, they can access the requested resource.

Let’s examine this using a concrete example. A device X tries to post a message over MQTT protocol to an address mainflux/channels/7209d9b8-90af-11e7-9cf0-080027b77be6/messages. On each MQTT PUB or SUB to a given channel, an internal MQTT broker will forward an authorization request to a Manager service. Since channels (client groups) correspond to MQTT topics, the Manager service will check if the device X is actually “plugged” into a channel 7209d9b8-90af-11e7-9cf0-080027b77be6. If so, it will respond to the MQTT broker to safely accept the request. Otherwise the MQTT broker will refuse this unauthorized access.

Building an End-to-End Example

Freeflex

Freeflex is a system for complete office management. It is especially useful in so-called flex-office environments, where employees do not have a fixed desk, but rather look for an unoccupied desk upon their arrival at the office in the morning.

The Freeflex system is comprised of:

• Physical sensors installed on office desks or in the conference rooms

• Mainflux IoT cloud

• Application with visual UI and dashboards

Hardware Sensor

A Freeflex sensor is based on the ESP32 microprocessor, which besides sophisticated processing capabilities offers both Bluetooth Low Energy and WiFi connectivity, as shown in Figure 4-3. A whole set of sensors is connected to the processor via various hardware interfaces:

• PIR (presence) sensor

• Temperature and humidity sensor

• Noise sensor (microphone)

• Light sensor

• Air quality sensor

• NFC and RFID transceiver

Figure 4-3. Freeflex ESP32 sensor

Firmware is programmed in the C programming language, using open source libraries provided by ESP32 manufacturer Espressif. The device is powered via 5V USB and has very low power consumption.

Connection to Mainflux

Freeflex connects to the Mainflux system via the MQTT protocol. Sensor measurements are aggregated in one SenML message that is periodically sent to Mainflux over the dedicated message channel.

Besides the message channel allocated for this device, another dedicated channel is provisioned: the command channel. This channel is used for Mainflux to push the commands coming from the application on the northbound interface toward the device.

That means that for each device two Mainflux channels are provisioned: the “message” channel and the “command” channel.

Besides these two channels, a special “bootstrap” Mainflux channel is provisioned for system management and configuration. This channel is shared by all Freeflex devices, and it is used for obtaining initial configuration.

For simplicity purposes, all ESP32 devices are manufactured and flashed with the same default firmware in the factory. This lowers the price of manufacturing and avoids potential fabrication mistakes. Then once deployed, Freeflex devices connect to the pre-configured “bootstrap” channel and demand initial configuration, which is specific for each device.

When it obtains a configuration request via the “bootstrap” channel, the Mainflux system provisions one device model and two channels that would be dedicated for this device (“message” and “command). Then it sends the information (device UUID, authentication credentials, and channel UUIDs) to the device, which stores it in its flash.

From this point on, the device uses this newly obtained data to authenticate to the Mainflux system and post messages to its channels.

Application and dashboards

The Freeflex application consists of a backend written in NodeJS and a frontend written in AngularJS.

The application backend uses the MQTT protocol to communicate with Mainflux middleware and a MongoDB database to store its internal data.

Its user interface (application frontend) has several screens that are used both for system management and for data visualization:

• The complete device list shows all connected sensors and the channels that were provisioned for them.

• Various dashboards visualize sensor data, as shown in Figure 4-4.

• A complete office floorplan (Figure 4-5) offers visual information as to where the sensors are deployed in the building (in which offices), and also shows occupancy of conference rooms (color-coded information).

Figure 4-4. Freeflex’s dashboard showing room occupancies

Figure 4-5. Freeflex dashboard showing the floorplan

LoRa Water Meter

According to the Lora Alliance,¹ Low-Power, Wide-Area Networks (LPWAN) are projected to support a major portion of the billions of devices forecasted for the Internet of Things (IoT). LoRaWAN™ is designed from the bottom up to optimize LPWANs for battery lifetime, capacity, range, and cost.

For the purpose of enabling connection of long-range low-power sensors, Mainflux has been integrated with a LoRaWAN server. The LoRaWAN system is composed of hardware gateways, or base-stations, connected to the LoRaWAN software in the cloud. Together, they form LoRaWAN network. Devices and sensors connect to gateways via the LoRa physical protocol, and gateways later forward the data coming from the devices to LoRaWAN server using the UDP protocol.

Hardware

On the hardware side, a MatchX LoRa gateway is used to connect LoRa sensors, as illustrated in Figure 4-6.

Figure 4-6. MatchX LoRa gateway (taken from matchx.io)

Software

The Mainflux system has been integrated with one of the most popular open source (see LoRa Server implementations). A dedicated adapter has been implemented to enable this integration, as shown in Figure 4-7.

Figure 4-7. LoRa server integration diagram

The Mainflux LoRa Adapter uses the MQTT protocol to connect the Mainflux system with LoRa Server. It is basically an MQTT client to both LoRa Server and the Mainflux server, so it opens an MQTT connection to both systems upon start. Both LoRa Server and Mainflux have MQTT brokers internally, over which they expose an MQTT API, so they are capable of accepting LoRa Adapter as their client.

The adapter then receives event notifications coming from LoRa Server and forwards them to the Mainflux MQTT broker.

LoRa Server specifies the format in which messages from devices are forwarded. The main purpose of the adapter is to accept and reformat these messages in the SenML format that can be accepted by Mainflux.

An application sitting on the top of the Mainflux server receives these messages coming from LoRa devices and visualizes the incoming data. The application uses a WebSocket connection in order to update the visual representation without having to poll constantly for new values that should be presented.

Summary

Mainflux is an open source, Apache-2.0 licensed IoT platform, with modern architecture based on microservices. It is written in the Go programming language to leverage the powerful features that this new programming language provides, such as concurrency patterns, fast and robust execution, type safety, and easy deployment of a single, statically linked portable binary per microservice. The overall impression is that a “lean” approach is taken when writing the software, as unnecessary bloat has been trimmed, and following Go’s maxim “simplicity is complicated,” authors worked very hard to find simple and readable solutions.

Mainflux shows maturity in design and implementation, and has recently reached its production-ready v1.0 version.

Deployment of the system has been facilitated through the use of Docker containers and Kubernetes. Special care has been put to make these containers as tight as possible. Combining “scratch” base images with a new multistage build feature resulted in images with cumulative size slightly below 30 MB.

The following features of the system have to be underlined:

• Protocol bridging (i.e., HTTP, MQTT, WebSocket, CoAP)

• Device management and provisioning

• Linearly scalable data storage

• Fine-grained access control

• Platform logging and instrumentation support

• Container-based deployment using Docker

These features are indispensable in building “vertical solutions,” end-to-end applications for smart city, smart agriculture, health, transportation, and many other fields where industrial context is important and demand for quality, testing, and system robustness is of primary importance.

Finally, it is important to note that although the IoT space is crippled by a lack of standardization, Mainflux implementation leans toward standards, or even promising standard drafts where standards are missing. For example, standard protocol implementations are used (MQTT or CoAP), and moreover even for the message semantics adheres to the IETF mature standard draft called SenML. This ensures system interoperability, a feature very important in the industrial space. It also guarantees “futureproofness,” relying on standards that are popular and widely used, and that will be further developed by industry players (as opposed to proprietary custom implementations that quickly become obsolete).

Table 4-1 summarizes the characteristics of Mainflux.

Table 4-1. Summary of Mainflux capabilities

Trait name	Trait value
Device bindings	HTTP, MQTT, WebSockets CoAP
Analytics	Apache Spark
Visualization	HTML5
Rules engine and alarming	N/A
Security	JSON Web Tokens, ACLs, TLS
License	Apache 2.0
Deployment technology	Kubernetes, Docker
Auto-scaling	Microservices architectural style
Device data persistence	Apache Cassandra
Management database	Apache Cassandra
Implementation language	Go
Data model	SenML

With the possibility to be deployed both in the cloud and on premise, scaling from gateway to full-blown clusters, Mainflux has a lot to offer when it comes to building modern industrial IoT systems.

¹ For more information, check out the whitepaper LoRaWAN: What Is It?.