python: migrate applications to use IoT3 Core SDK #149

ymorin-orange · 2024-08-14T13:51:38Z

Features:

Fix IoT3 Core SDK
- Fixes python/iot3: connecting to a UNIX socket raises an exception #155
- Fixes python/iot3: subscribing to all topics at once raises an exception #156
Use IoT3 Core SDK
- use the MQTT wrapper
- implement telemetry

Prepare an OpenTelemetry collector on localhost:

$ docker container run \
    --rm \
    -p 16686:16686 \
    -p 4318:4318 \
    jaegertracing/all-in-one:1.58

then open a browser on the Jaegger UI:

http://localhost:16686/

Prepare an MQTT broker on localhost (adapt based on your distribution):
```
$ sudo systemctl start mosquitto.service
```
and then start an MQTT client to dump the messages on the broker:
```
$ mosquitto_sub -V 5 -t '#' -F '%j' |jq .
```
Start a gpsd daemon
1. create a fake NEMA log, e.g. in file NMEA.log, e.g. the sample available
  on wikipedia
2. start a gpsd daemon sending fake data:
```
$ TMPDIR=/tmp gpsfake -q -P 2947 -u -n -c 0.1 NMEA.log
```

Test IoT3 Core SDK:

if your python is older than 3.11, install an environment with python 3.11
(you'll need to have a development environment with gcc and at least libffi-dev
and libssl-dev installed; adapt to your distribution):

$ wget https://www.python.org/ftp/python/3.11.9/Python-3.11.9.tar.xz
$ tar xJf Python-3.11.9.tar.xz
$ cd Python-3.11.9
$ ./configure --prefix "${HOME}/tmp/python/3.11" --with-openssl=/usr
$ make install
$ export PATH="${HOME}/tmp/python/3.11/bin:${PATH}"

create a /venv/ in a writable location, and activate it:

$ python3.11 -m venv ~/tmp/its
$ . ~/tmp/its/bin/activate

install the ITS components:

(venv) cd python
(venv) pip3 --disable-pip-version-check install \
        --no-binary ':all:' \
        ./iot3 \
        ./its-interqueuemanager \
        ./its-quadkeys \
        ./its-status \
        ./its-vehicle

Note: its-info is not modified, so tested is not needed, so
it's not istalled.

test the its-status component:

create a configuration file, e.g. /tmp/its-status.cfg, with this content:

[generic]
id = test-status
frequency = 1.0

[stdout]
enabled = false

[mqtt]
enabled = true
host = localhost
port = 1883
username = USERNAME
password = PASSWORD
client_id = its-status
topic = status/system

[system]
data-dir = /

[gnss]
host = localhost
port = 2947
persistence = 5.0

Note: adapt USERNAME and PASSWORD with the appropriate credentials,
and adapt host and port is you are usign a remote broker and/or it
is listening on a non-standard port.

start its-status with that configuration file:
```
(venv) $ its-status -c /tmp/its-status.cfg
```

test the its-vehicle component:

create a configuration file, e.g. /tmp/its-vehicle.cfg, with this content:

[general]
instance-id = test-me
type = CAM
report-freq = 0.5
topic-pub-prefix = PROJECT/inQueue/SUFFIX/
topic-sub-prefix = PROJECT/outQueue/SUFFIX/
depth = 22
messages = cam cpm denm
depth-sub-cam = 18
depth-sub-cpm = 18
depth-sub-denm = 15
speed-thresholds = 8.33 19.44 27.78

[telemetry]
endpoint = http://localhost:4318

[broker.main]
host = localhost
port = 1883
username = USERNAME
password = PASSWORD

[gpsd]
port = 2947
persistence = 10.0

start its-vehicle with that configuration file:
```
(venv) $ its-vehicle -c /tmp/its-vehicle.cfg
```

test the its-iqm component:

create a configuration file, e.g. /tmp/its-iqm.cfg, with this content:

[general]
instance-id = ora_geo_1234
prefix = PROJECT
suffix = SUFFIX

[telemetry]
endpoint = http://localhost:4318

[local]
host = localhost
port = 1883

start its-iqm with that configuration file:
```
(venv) $ its-iqm -c /tmp/its-iqm.cfg
```

Look up traces in the Jaeger UI
Stop the ITS components with Ctrl-C on each

Expected results

Pre-requisites are all met.

The Jaegger UI is displayed in your browser
The Mosqiutto server and clients are running
The gpsd daemon is running

The ITS components behave as expected;

the contaienr starts
the venv is activated
the ITS components are installed

the its-status component emits status messages, and the verbose dump
from mosquitto_sub would look like:

{
  "tst": "2024-09-20T08:01:04.047514Z+0200",
  "topic": "status/system",
  "qos": 0,
  "retain": 0,
  "payloadlen": 836,
  "payload": {
    "time_sources": null,
    "gnss": {
      "software": "gpsd 3.22",
      "model": "NMEA0183",
      "mode": 3
    },
    "cellular": [],
    "version": "1.2.0",
    "type": "status",
    "id": "test-status",
    "system": {
      "type": "obu",
      "hardware": "Unknown",
      "os_release": {
        "PRETTY_NAME": "Ubuntu 22.04.5 LTS",
        "NAME": "Ubuntu",
        "VERSION_ID": "22.04",
        "VERSION": "22.04.5 LTS (Jammy Jellyfish)",
        "VERSION_CODENAME": "jammy",
        "ID": "ubuntu",
        "ID_LIKE": "debian",
        "HOME_URL": "https://www.ubuntu.com/",
        "SUPPORT_URL": "https://help.ubuntu.com/",
        "BUG_REPORT_URL": "https://bugs.launchpad.net/ubuntu/",
        "PRIVACY_POLICY_URL": "https://www.ubuntu.com/legal/terms-and-policies/privacy-policy",
        "UBUNTU_CODENAME": "jammy"
      },
      "cpu_load": [
        0.650390625,
        0.8369140625,
        0.9921875
      ],
      "memory": [
        67095187456,
        53034876928
      ],
      "storage": [
        981132795904,
        506785951744
      ]
    },
    "timestamp": 1726812064.0466053
  }
}

the its-vehicle component emits status messages, and the verbose dump
from mosquitto_sub would look like (notice the traceparent user
property):

{
  "tst": "2024-09-20T08:05:52.970202Z+0200",
  "topic": "PROJECT/inQueue/SUFFIX/cam/test-me/1/2/0/2/2/2/0/3/0/2/2/3/0/2/2/1/0/3/1/0/3/3",
  "qos": 0,
  "retain": 0,
  "payloadlen": 641,
  "properties": {
    "user-properties": {
      "traceparent": "00-057264776dc53a957fbb14530684cc45-47096d87822883fa-00"
    }
  },
  "payload": {
    "type": "cam",
    "origin": "self",
    "version": "1.1.3",
    "source_uuid": "test-me",
    "timestamp": 1726812352969,
    "message": {
      "protocol_version": 1,
      "station_id": 7603010,
      "generation_delta_time": 47411,
      "basic_container": {
        "station_type": 5,
        "reference_position": {
          "latitude": 436189667,
          "longitude": 15019833,
          "altitude": 0
        },
        "confidence": {
          "position_confidence_ellipse": {
            "semi_major_confidence": 10,
            "semi_minor_confidence": 50,
            "semi_major_orientation": 1
          },
          "altitude": 1
        }
      },
      "high_frequency_container": {
        "heading": 3007,
        "speed": 998,
        "longitudinal_acceleration": 161,
        "drive_direction": 0,
        "vehicle_length": 40,
        "vehicle_width": 20,
        "confidence": {
          "heading": 2,
          "speed": 3,
          "vehicle_length": 0
        }
      }
    }
  }
}

the its-iqm component emits status messages, and the verbose dump from
mosquitto_sub would look like (notice the traceparent user property,
and the different topics: inQueue, outQueue, and interQuweue):

{
  "tst": "2024-09-20T08:08:49.388984Z+0200",
  "topic": "PROJECT/inQueue/SUFFIX/cam/test-me/1/2/0/2/2/2/0/3/0/2/2/3/0/2/0/2/2/3/0/0/2/3",
  "qos": 0,
  "retain": 0,
  "payloadlen": 641,
  "properties": {
    "user-properties": {
      "traceparent": "00-e6386928b78d3f87f403ef63ce71c33e-4c1e46c2e528c427-00"
    }
  },
  "payload": {
    "type": "cam",
    "origin": "self",
    "version": "1.1.3",
    "source_uuid": "test-me",
    "timestamp": 1726812529387,
    "message": {
      "protocol_version": 1,
      "station_id": 7603010,
      "generation_delta_time": 27284,
      "basic_container": {
        "station_type": 5,
        "reference_position": {
          "latitude": 436209667,
          "longitude": 14956333,
          "altitude": 0
        },
        "confidence": {
          "position_confidence_ellipse": {
            "semi_major_confidence": 10,
            "semi_minor_confidence": 50,
            "semi_major_orientation": 1
          },
          "altitude": 1
        }
      },
      "high_frequency_container": {
        "heading": 2729,
        "speed": 998,
        "longitudinal_acceleration": 161,
        "drive_direction": 0,
        "vehicle_length": 40,
        "vehicle_width": 20,
        "confidence": {
          "heading": 2,
          "speed": 3,
          "vehicle_length": 0
        }
      }
    }
  }
}
{
  "tst": "2024-09-20T08:08:49.390629Z+0200",
  "topic": "PROJECT/outQueue/SUFFIX/cam/test-me/1/2/0/2/2/2/0/3/0/2/2/3/0/2/0/2/2/3/0/0/2/3",
  "qos": 0,
  "retain": 0,
  "payloadlen": 641,
  "properties": {
    "user-properties": {
      "traceparent": "00-f431be1d1d66f07b110507d58185a3fd-4851c79cb4026051-00"
    }
  },
  "payload": {
    "type": "cam",
    "origin": "self",
    "version": "1.1.3",
    "source_uuid": "test-me",
    "timestamp": 1726812529387,
    "message": {
      "protocol_version": 1,
      "station_id": 7603010,
      "generation_delta_time": 27284,
      "basic_container": {
        "station_type": 5,
        "reference_position": {
          "latitude": 436209667,
          "longitude": 14956333,
          "altitude": 0
        },
        "confidence": {
          "position_confidence_ellipse": {
            "semi_major_confidence": 10,
            "semi_minor_confidence": 50,
            "semi_major_orientation": 1
          },
          "altitude": 1
        }
      },
      "high_frequency_container": {
        "heading": 2729,
        "speed": 998,
        "longitudinal_acceleration": 161,
        "drive_direction": 0,
        "vehicle_length": 40,
        "vehicle_width": 20,
        "confidence": {
          "heading": 2,
          "speed": 3,
          "vehicle_length": 0
        }
      }
    }
  }
}
{
  "tst": "2024-09-20T08:08:49.390795Z+0200",
  "topic": "PROJECT/interQueue/SUFFIX/cam/test-me/1/2/0/2/2/2/0/3/0/2/2/3/0/2/0/2/2/3/0/0/2/3",
  "qos": 0,
  "retain": 0,
  "payloadlen": 641,
  "properties": {
    "user-properties": {
      "traceparent": "00-f431be1d1d66f07b110507d58185a3fd-2a7a8db15dec456d-00"
    }
  },
  "payload": {
    "type": "cam",
    "origin": "self",
    "version": "1.1.3",
    "source_uuid": "test-me",
    "timestamp": 1726812529387,
    "message": {
      "protocol_version": 1,
      "station_id": 7603010,
      "generation_delta_time": 27284,
      "basic_container": {
        "station_type": 5,
        "reference_position": {
          "latitude": 436209667,
          "longitude": 14956333,
          "altitude": 0
        },
        "confidence": {
          "position_confidence_ellipse": {
            "semi_major_confidence": 10,
            "semi_minor_confidence": 50,
            "semi_major_orientation": 1
          },
          "altitude": 1
        }
      },
      "high_frequency_container": {
        "heading": 2729,
        "speed": 998,
        "longitudinal_acceleration": 161,
        "drive_direction": 0,
        "vehicle_length": 40,
        "vehicle_width": 20,
        "confidence": {
          "heading": 2,
          "speed": 3,
          "vehicle_length": 0
        }
      }
    }
  }
}

The Jaegger UI displays traces for the its-vehicle and its-interqueuemanager
services, with appropriate links between traces.
The ITS components all stop

Although the unix socket transport does not require a TCP port, paho-mqtt expexts that a valid TCP port be specified when connecting. Technically, specifying a port is not necessary in that case, so we could in theory change our call to connect_async() to not pass a port at al. However, that would make the code a bit more complex. So, to simplify things, just use a valid TCP port, i.e. one that is strictly greater than 0. We could have decided to use the default port, 1883, but that would be a bit misleading. Instead, we just use 1, which is odd enough that it will at least raise an eyebrow to an unsuspecting reviewer. Signed-off-by: Yann E. MORIN <[email protected]>

The unsubscribe() method expects keyword arguments, not positional ones. Signed-off-by: Yann E. MORIN <[email protected]>

The vareious ITS clients (its-vehicle, its-iqm...) currently use the APIs of paho.mqtt 1.6.1, while the newly-introduce IoT3 Core SDK uses those of 2.1.0. In paho.mqtt 2.0.x, there was a breaking API change, so a client written against paho.mqtt 1.6.1 (or earlier) could not be used as-is with paho.mqtt 2.0.x. but in paho.mqtt 2.1.0, there was another API change that made it possible to use clients written against 1.6.1, without any modification, while also alowing clients written againt the 2.0.0 API as well. So we can accept anything from 1.6.1 onward, except for versions 2.0.x. Signed-off-by: Yann E. MORIN <[email protected]>

Signed-off-by: Yann E. MORIN <[email protected]>

nbuffon · 2024-09-24T09:06:50Z

Typo in PR description at point 4.v (venv) $ its-status -c /tmp/its-vehicle.cfg
Command should be its-vehicle not status

ymorin-orange requested review from tigroo and nbuffon August 14, 2024 13:52

ymorin-orange mentioned this pull request Aug 14, 2024

python: implement IoT3 Core SDK #144

Merged

ymorin-orange force-pushed the yem/its-iot3 branch 2 times, most recently from 3135f4d to 865cce9 Compare September 9, 2024 13:51

tigroo removed their request for review September 11, 2024 07:49

tigroo added enhancement New feature or request Python Python code labels Sep 11, 2024

tigroo added this to the Sprint 2 milestone Sep 11, 2024

ymorin-orange force-pushed the yem/its-iot3 branch from 865cce9 to 102170e Compare September 11, 2024 13:49

ymorin-orange self-assigned this Sep 20, 2024

ymorin-orange added 18 commits September 20, 2024 15:43

python/iot3/core: fix unsubscribing to all MQTT topics

583b3bd

The unsubscribe() method expects keyword arguments, not positional ones. Signed-off-by: Yann E. MORIN <[email protected]>

python/status: use iot3.core.mqtt

9191a84

Signed-off-by: Yann E. MORIN <[email protected]>

python/iqm: no one ever join()s the authority threads

bb7fd80

Signed-off-by: Yann E. MORIN <[email protected]>

python/iqm: use iot3.core.mqtt for queue-manager

7960edd

Signed-off-by: Yann E. MORIN <[email protected]>

python/iqm: use iot3.core.mqtt for MQTT authority client

169568b

Signed-off-by: Yann E. MORIN <[email protected]>

python/iqm: drop custom MQTT client

1eddf54

Signed-off-by: Yann E. MORIN <[email protected]>

python/iqm: add telemetry

3c692a4

Signed-off-by: Yann E. MORIN <[email protected]>

python/vehicle: drop tracking from ITS message

6fc0bde

Signed-off-by: Yann E. MORIN <[email protected]>

python/vehicle: don't instanciate dummy custom tracker

ae3170a

Signed-off-by: Yann E. MORIN <[email protected]>

python/vehicle: drop custom, dummy tracker

2ee8aaa

Signed-off-by: Yann E. MORIN <[email protected]>

python/vehicle: don't use abbrev() which is going to disapear

c4f9aee

Signed-off-by: Yann E. MORIN <[email protected]>

python/vehicle: only start() clients once all have been created

cecdc18

Signed-off-by: Yann E. MORIN <[email protected]>

python/vehicle: insert trampoline on message reception

1f0061d

Signed-off-by: Yann E. MORIN <[email protected]>

python/vehicle: use iot3.core.mqtt

c482cb1

Signed-off-by: Yann E. MORIN <[email protected]>

python/vehicle: drop our custom MQTT Client

4b88b93

Signed-off-by: Yann E. MORIN <[email protected]>

python/vehicle: add telemetry

9d6852c

Signed-off-by: Yann E. MORIN <[email protected]>

ymorin-orange force-pushed the yem/its-iot3 branch from 102170e to 9d6852c Compare September 20, 2024 13:43

ymorin-orange marked this pull request as ready for review September 20, 2024 13:43

nbuffon approved these changes Sep 24, 2024

View reviewed changes

ymorin-orange merged commit 661c018 into Orange-OpenSource:master Sep 24, 2024
32 checks passed

ymorin-orange deleted the yem/its-iot3 branch September 24, 2024 09:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

python: migrate applications to use IoT3 Core SDK #149

python: migrate applications to use IoT3 Core SDK #149

ymorin-orange commented Aug 14, 2024 •

edited

Loading

nbuffon commented Sep 24, 2024

python: migrate applications to use IoT3 Core SDK #149

python: migrate applications to use IoT3 Core SDK #149

Conversation

ymorin-orange commented Aug 14, 2024 • edited Loading

nbuffon commented Sep 24, 2024

ymorin-orange commented Aug 14, 2024 •

edited

Loading