Key Building Blocks Of An Iot System
The overall hype around IoT has unexpectedly hindered the understanding of how it works. If you ask twenty people about how it works, you will get twenty answers. Most of those answers would cover how outcomes of IoT or manifested IoT solutions work; not the way it works. There is a lot of technology under the hood that makes IoT possible. Here we will discuss key building blocks of an IoT solution. We will also explore how these blocks work together in a meaningful way. This will help us identify how it all works and then we can plan the building of our IoT platform in an effective way.
When it comes to various terminologies used, often interchangeably, there is a difference between IoT solution and IoT application. An IoT solution is usually would mean an end-to-end product, service, or mix of both. Whereas, the IoT application usually would refer to an IT software or mobile application part of it, or a combination of both. Clearly, IoT solution encompasses many more things than just an IoT application. A lot of business context, customer context, and geopolitical context plays would influence IoT solution as such. However, from an IoT platform perspective, it sits on the edge of IoT application and is usually borderline system to deal with physical objects aka things and software systems.
Functional blocks of an IoT solution
At a high level, we can identify IoT solution comprising of four major functional blocks. If any of these blocks are missing, then it will not be prudent to call it an IoT solution. Devices aka things: These are physical sensors and actuators. They measure various parameters and translate them into electrical or digital data. These sensors either would be connected to the host devices (typical for legacy upgrades) or could be integrated into the host devices (modern). These devices are critical nodes of an IoT application and are required to deliver full solution functionality, either by way of acting as inputs, outputs, or both. Typical examples of such devices are thermostats, intelligent mousetraps, connected refrigerators, etc. Gateways: Edge devices can communicate with the upstream system in one of the two ways, with or without a gateway. Some of the devices have the capability to communicate directly over the internet protocols (IP) using various communication protocols such as ReST, MQTT, AMQP, CoAP, etc. These capabilities are usually a result of integrated communication modules such as WiFi or GSM chips. These enable the device to connect to the network gateways such as WiFi router and Mobile towers and communicate with the upstream layer directly. In these cases, routers and mobile towers perform the job of the gateway.
However, not all devices are capable of direct internet connectivity and do not have the necessary hardware built-in. In these cases, they need to piggyback on some other device to help their data pushed to the upstream layer. Gateways help devices to do this. Usually, hardware gateways are built with dual communication technologies, which enables them to communicate with downstream devices with one type of channel and with upstream layers with another. Typical examples of such gateway capabilities include GSM + RF, GSM + Bluetooth, WiFi + Bluetooth, WiFi + Xbee, LoRaWAN + Ethernet, and likewise. In some cases, smartphones are also used as gateways and this is more prominent for Bluetooth Low Energy (BLE) type of devices. In addition to providing the transport mechanism, the gateway can also provide optional functions such as data segregation, cleanup, aggregation, deduplication, and edge computing.
IoT Platform
This is an orchestrator of the whole IoT solution and is almost often hosted in the cloud. This block is responsible for communicating with downstream devices and ingesting large amounts of data at a very high speed. The platform is also responsible for storage of the data in time-series and structured format for further processing and analysis. Depending upon the sophistication built in the platform, some platforms may also support deep data analysis and other operations. However, the core of the IoT platform remains as an orchestrator of the whole system.
Application
In most scenarios, applications are the front face to the whole solution and must present it to the end user in a meaningful way. These applications are either desktop based, mobile based, or both. Applications also enrich the data from the platform in various ways and present it to the users in a usable format. Additionally, these applications also integrate with other systems & applications at the interface level and enable inter-application data exchange. A typical example of such an operation is inventory-tracking devices equipped with tracking mobile application to the users, and the data fed to the ERP system for stock keeping. The detailed block diagram of an IoT platformWe are more interested in the mechanics of the third block - IoT platform. Let us see all the fundamental inclusions that an IoT platform should have to perform effectively. The platform itself is installed on a virtual cloud machine or VPS (virtual private server). It is highly recommended to use Linux based operating system such as Ubuntu, Centos, Debian, etc. for better performance, security features, and overall control of the platform. However, the concept and block level architecture would not change for any of these operating systems.
Edge interface, message broker and message bus
This is the module, which deals and talks with the physical world out there, especially heterogeneous devices and sensors. Since devices could be communicating over a multitude of communication technologies, such as WiFi, Bluetooth, LoRaWAN, GPRS, etc. , this module needs to be able to cater to all those. We can achieve this in a modular format where each type of communication protocol is dealt with separately. As an example, for WiFi capable device, it can be the ReST API, which caters to the constrained devices. It could be an MQTT based message-broker, which enables communication in a pub-sub manner. For LoRaWAN based devices there would be another plugin to the main message broker, which deals and talks with LoRaWAN network servers and performs decoding of packets. This module decouples the whole platform from devices in an effective way. Just one type of edge interface is not acceptable for a properly functional platform as not all the devices may have the same channel and method of communication. Regardless of the medium of communication, network type used, and protocols in play, message broker's job is to consolidate the data in a unified manner and push it to the common message bus. All the other functional blocks share this message bus for further operation. The broker acts as a coordinator and consolidator of messages.
Message router and communication management
Once the messages are available on the main message bus, they often may not be enough for processing by another module. Some messages need feature enrichment and additional information to be appended or added separately, and this depends upon the context of the device deployment and application requirements. The functionality of enriching existing data messages, re-broadcasting them to the message bus, publishing additional contextual information and more messages after the main message arrives, and tagging them as appropriate is the job of communication management module. Communication management functions in coordination with message broker and rule engine block and interacts with the device manager as required. In addition to this, the communication management module also performs the duties of format conversion, for example, translate data from CSV to JSON, or binary to Text format, and likewise. We can also task it to perform certain operations like deduplication of messages. Deduplication is the process of eliminating or discarding multiple duplicate messages or redundant data packets from the devices, as they may not be of any use. Deduplication schemes are dependent on device or sensor types and we need to implement them on a case-by-case basis, although the methodology remains the same. As a communications router, this module then can control further messaging and communication on the platform.
Time-series storage and data management
As the name suggests, this block stores all the received and parsed data that is available on the message bus in sequential i. e. time-series style. Whilst data storage is not the core function of the IoT platform, usually, modules outside the platform handle it. Although, it is an essential activity for coordination and orchestration perspective. Many a time, communication and routing module or the message broker itself need recent past data for specific functional purposes, this storage comes in handy for all such instances.
For many IoT applications, users prefer to extract the data away from the IoT platform and store it in an application data warehouse for further processing. Therefore, this storage is often utilized for interim storage of the device data and is not meant for large-sized dataset storage.
Rule engine
In my view, this is a very powerful block and provides enhanced capability to the platform. The rule engine, as the name suggests is execution block that monitors the message bus and events across the platform and takes an action based on set rules. For example, a typical rule engine function may look like as - "trigger and broadcast Alert message when the downstream device sends a data packet containing a keyword ka-boom". The rule engine is constantly listening to the message bus broadcasts. When the communication block puts up a decoded data packet from the downstream device on to the message bus - rule triggers. Rule engine broadcasts another message (Alert) on to the message bus. Since this happens all within the IoT platform and amongst closely coordinated modules, execution speed is quite fast. Rule engine also helps in building modular rules for decoding and enriching existing or received data from the devices and would, therefore, augment communication module's functionality. In addition to that, it is easy to implement callbacks to other modules, applications, programs, and systems.
ReST API interface
Restful APIs are useful for support functions and utilities which do not need constant or real-time connectivity and access. These APIs, although typically used by upstream programs and applications, downstream devices can also access them as and when needed. A classic example of such a use case would be, a temperature sensor with WiFi connectivity that sends readings every 15 minutes. Due to such a long time between two subsequent readings, real-time connection or always-on connectivity is undesired. A simple HTTP operation can do the data sending job relatively more efficiently. In this case, this sensor can send the data over ReST API to the platform. This ReST API works with message broker and communications manager to present this received data post on to the main message bus and may also use time-series database records to send back the response to the sensor. This response may contain additional information for the sensor to do its job in a certain way for the next round. Additionally, this API block can also support data aggregation and bulk operational functionalities, such as querying of multiple records by the upstream application. This way, upstream applications, and systems remain decoupled from the core platform blocks thereby maintaining the partition of functions and ensuring the security. Various role-based authentications can be built-in for access to this API.
ReST API block can also feed into Rule Engine and allow applications to configure or trigger specific rules at any given point in time. This also makes it possible for downstream devices to utilize the same functionality which could become handy when devices need to initiate certain workflow automatically in place of application triggers. A good example could be the smart lock. Say for instance, when there is an activity at the door that needs owner's attention who is away from home; the upstream application may notify the user when smart lock reports this activity and then would expect the user to respond or react for further steps. If the user is not available or reachable then the application can itself trigger the rule for predefined actions. However, if the severity of the alert is relatively higher, then the device may be configured not to wait for user action or response, but directly trigger the default workflow (say notifying security, etc. ). These functionalities can come in handy when designing and operating an autonomous and intelligent fleet of devices.
Microservices
Besides data management, manipulation, and exchange functionalities, the IoT platform also needs certain support functions to function effectively. Services such as text messaging or email notifications, verifications, captcha, social media authentications or payment services' integration are a few examples of these auxiliary services. These services are bundled in the micro-services block. Not only this but also in case of frequent use of certain functionality within the platform, it can be bundled and packaged under this block to separate it from the mainstream platform. Once separated and packaged, it then can be exposed to the blocks within and outside of the platform for reuse.
Device manager
When the platform starts to host 50 or more devices, things would become difficult to manage. It almost becomes necessary to have some type of central control in place for managing of those things aka devices. This is where the device manager block helps. It essentially provides the generic functionality of managing devices as assets. Listing of all the devices, their active-inactive status, battery levels, network conditions, access keys, readings, stored data access, device details, session information, and many such things. Device manager also helps in managing over the air updates for the fleet of devices or at least central monitoring function for system admins. In certain use cases, devices also need access rights and users may also be assigned certain access rights to the set of devices. Management of such an accessibility matrix becomes easy with the device manager.
Application and user management
This block provides almost similar functionality as device manager, the only difference is, it provides this functionality for upstream applications and users. Typical user management, such as password and credentials, access keys, logins, and rights are managed through this block. For upstream applications and various other integrated systems, API keys, credentials, and access can be managed through the same block. While it may appear to be more of application-level functionality, it remains in IoT platforms interest to bind it together as a platform function, so that it is integrated tightly with overall architecture and set of things. IoT is the system of systems and heterogeneous systems are the fact of this phenomena. Letting these systems function out of sync is the last thing you would want to happen with IoT solutions.
Is everything from this block architecture mandatory?
Ofcourse not. While above mentioend eight of the blocks define a very well architected IoT platform, not all of them are mandatory or necessary. A specific use case or industry vertical may define this situation differently. You may not need all those blocks at the outset and they may be added later in the life cycle of the platform development. Core functional blocks such as, device interface & message broker, message router and communications module, data storage, device management and rule engine are critical for effective functioning of an IoT platform. Other blocks viz. ReST APIs, micro services, and application & user management are good to have and often make life easy. But are not mandatory and would not obstruct functionality of the IoT platform as such. When developing our IoT platform from ground up, we will keep these functionalities on the back burner and would only implement them if time permits and resources are available.
What is the proposed approach for developing our IoT platform?
To develop an IoT platform in quickest possible time, we will not only develop it in modular form but will also do it in an agile way. Each module will be planned, functions and features set out, then developed and then deployed on the cloud for testing. Once we test individual module and found it to be working as expected, we can go to the next. As a first step, we will set up the cloud environment for the platform. Followed by setting up essential components we will also develop our first module - edge interface and message broker. It will be logical next step to setup time-series data storage post this step. Then we will develop basic ReST APIs for the platform followed by message router functionality. Some of the microservices would be developed after we have set up a fundamental wireframe of the platform in place. We will then iterate through all these blocks a few more times to make a stable core of the platform.
Once we are happy with the core functionalities, the rule engine can be setup followed by device management functions. Application and user management will be reviewed in the end as it will be one of the non-essential modules.
