Support of device config and state in Hono

[matthiasfeurer]

Summary

In order to qualify Hono as a replacement for Google’s IoT Core we also need to find a way how config and state of a device are supported. Google IoT Core defines state and config as follows:

Device state
An arbitrary, user-defined blob of data that describes the current status of the device. Device state data can be structured or unstructured, and flows only in the device-to-cloud direction.

Device configuration
An arbitrary, user-defined blob of data used to modify a device's settings. Configuration data can be structured or unstructured, and flows only in the cloud-to-device direction.

Configurations are sent from cloud to device each time a device subscribes to the configuration topic and are persisted on cloud side (see also eclipse-hono/hono-extras#19) . States are sent from device to cloud and forwarded to the messaging infrastructure (here: Google Pub/Sub) and also persisted so that they can be retrieved via REST API.
See https://cloud.google.com/iot/docs/how-tos/config/configuring-devices and https://cloud.google.com/iot/docs/how-tos/mqtt-bridge#setting_device_state for details.
As of now we do not see that Hono provides something similar out of the box (ignoring ditto as a separate full blown digital twin solution).

Integration into Hono

For supporting configs and states for deployments without ditto we propose the following integration.
A configuration switch is added to enable / disable the feature (only if requested by Hono project owners).
A new root level MQTT topic“config” is introduced (shorthand to be defined). For sending config messages from cloud to device we would like to extend the existing command router messaging infrastructure, so that config messages are handled a bit differently. The config messages would need to be published as retained messages, so that once a device reconnects, it will receive the latest config message.
Support of "states" is done via a MQTT subtopic “states” in the events topic which is then published to a subtopic on the messaging infrastructure. For storing the state messages we see two possible solution:

1. We could either store them in the newly created device_config table, which is used by the extra package “Device Communication API”.
2. Or we could store them as metadata inside the data field of the device_registration table.

[gdimitropoulos-sotec]

Config Implementation

After comparing multiple solutions, we propose to implement the config functionality as shown in the following two cases:

1. Send Config to the device

When the device communication API receives a new configuration, first it will update the database and then the config will be passed to the command router via messaging infrastructure ( here: Pub/Sub). The command router sends the config to the MQTT adapter, which forwards the configs to the device. If the device successfully gets and acknowledges the configs, the MQTT adapter sends the acknowledgment via pub-sub direct to the device communication API. After the API receives the ack message updates the “deviceAckTime” field in the database.

2. Device is connected to the MQTT adapter

Each time a device connects to the MQTT adapter and subscribes to the config topic, the MQTT adapter sends the event via pub-sub to the device communication API. API checks if the latest config for this device is already acknowledged from the device and if not then the config will send to the device.

[sophokles73]

This looks good 👍 It should work well based on the existing concepts and functionality.

I assume that you are aware of the empty notification that the MQTT adapter will send downstream when a device subscribes to the command topic. You can use this notification event as the trigger to perform the steps you outlined under 2.

[matthiasfeurer]

In order to not extend Hono's root topics we have discussed two solutions:

1. Use a subtopic under the command topic structure and add single level wildcard support using "+" (see https://www.hivemq.com/blog/mqtt-essentials-part-5-mqtt-topics-best-practices/). This allows subscribers to use command/tenant/device/req/+/config for subscribing only to config messages and command/tenant/device/req/+ to subscribe to all other command messages. Existing Hono clients would not see any difference.
2. Make use of the already existing property bag approach that is used for telemetry and event topics. This would mean that we add a special property for config messages like: command/tenant/device/req/#/?contentType=config.
   The question is how clients could then subscribe to all other commands except those of type config. The only solution we see at the moment is to add a /?contentType=command to all other commands which could maybe influence existing Hono clients, so this is mabye not the best appraoch.
   @sophokles73 do you have any other idea?

[sophokles73]

The question is how clients could then subscribe to all other commands except those of type config.

I see your point. We could also support specifying other comparison operators in addition to =. For example, a device could use command/tenant/device/req/#/?contentType!=config in order to not receive config messages. But this seems a little odd. Using the standard wildcards together with the subtopic names seems more practical here. As you pointed out, a device could easily exclude messages for a subtopic by using the req/+ wildcard instead of req/#.

Adding the property bag to an outgoing command message's topic could still be added later to provide more meta data to the device.

So, in order to be able to use subtopics for command messages, you will need to figure out a way to (transparently) include the subtopic in command messages sent from applications. One way of doing this would be to add an optional message property (e.g. subtopic) to command messages.

If we go for this approach I would prefer for this new feature to be available in all protocol adapters. This does not mean, however, that it needs to be implemented for all of them in one go. But we should at least think this through at the conceptual level now, i.e. can we use this approach for the other adapters as well? Does the transport protocol support conveying they subtopic in a reasonable way?

[sophokles73]

I just realized that the MQTT adapter already appends the commandName to the topic that a command message is being published to. So, for config messages which are sent with a command name config, the adapter will publish a message with topic name
command/[${tenant-id}]/[${device-id}]/req//config
So, devices could easily filter on the command name to distinguish between config messages and other commands, while still using the standard topic filter when subscribing.

We could also think about extending the topic filter handling to allow for a subscription on a particular command name, e.g. using a filter like this to only subscribe to config messages:
command/[${tenant-id}]/[${device-id}]/req/+/config

[StFS]

This issue has been dormant for quite a while but I am curious to know if there is a plan to take this further. Also, how would this mesh with other adapters (such as coap)? Would this feature only work for MQTT devices?