Excessive readings lost

[neilh10]

For a stable site TUCA-PO03 I'm seeing a lot of readings lost in the last few days
(https://monitormywatershed.org/sites/TUCA_PO03/)
The readings loss is typically two lost, two received.
This pattern started at 2021-12-03 20:15 PST, up to the last reading
2021-12-07 5:30 PST
and no further readings/

It also has one reading that was partially received, at 2021-12-06 7:15:00 PM PST only the first sensor was recorded, the other 7 sensors are not recorded

Another site TUCA-Na13 has recovered from 2021-12-08 0:30 PST, and receiving steady stream of data.
however before that it has been losing data sporadically since 2021-12-01 5:30:00 AM
https://monitormywatershed.org/sites/TUCA-Na13/

[aufdenkampe]

@neilh10, we migrated from LimnoTech to AWS servers yesterday with our MonitorMW v0.12 release yesterday.

To make that switch, we had to change the IP addresses associated with our URLs, and those changes took a very long time to fully propagate through all the Domain Name System (DNS) servers around the country.

All the data loss issues you describe are related to the slow propagation of the DNS records among servers but also in the caches of local networks and devices.

Fortunately the data are not lost, but just showing up on our old servers. So we will be able to sync the data.

We expect the DNS to sort out in the coming days. I've described it in detail at https://www.envirodiy.org/topic/status-update-on-mmw/#post-16150

[neilh10]

thanks for the update. Fingers crossed. I thought I should just record what I was seeing.

[neilh10]

Nice to have https://monitormywatershed.org/sites/TUCA_PO03/ uploading data.

Performing a "Download Sensor Data" and looking at the records
TUCA_PO03 appears to operating correctly, as indicated by a steady incrementing sequence number. It has pushed data reliably since 2021-12-10 09:45 PST -(as of date of this update 2021-12-10 16:30 PST)

Between 2021-12-07 5:45:00 AM PST (-8HR)
and 2021-12-10 09:30:00 AM (PST)
some 304 records, taken at 15minute intervals appears to have been lost.

This node implements reliable data delivery, it gets a handshake from the HOST saying the data was received by the host.

A similar node https://monitormywatershed.org/sites/TUCA-Na13/ has had all its data delivered successfully from 2021-12-08 12:15:00 AM PST (-8HR). Before that it was also losing data.

This is only my reporting on what I'm seeing from these nodes , ideally a network stability test will be defined, and then a scientific calibrated test can characterize performance.

Generally its impossible to prove software works, however its useful to define expected data flows, and then test to those architectures across activities, and characterize the results.
#524

[neilh10]

amazing.. the missing readings have re-appeared, ported to the new database.
Practically, if the old system is stopped, then any lookups to it will fail - timeout 504 Modular Sensors inserted error, and then when the lookup hits the new working system they should transfer.
I'm trying to see when I can visit TUCA_PO03 to power cycle it, however its on private property and in the past I've asked to do it as part of a regular scheduled visit for the hydrology team.

[aufdenkampe]

@neilh10, I'm glad you now see all that missing data.

Our plan is to keep our old production server (on LimnoTech premises) running for a few more months, to continue forwarding data to our new AWS servers until everyone has time to power-cycle their devices during their normal maintenance schedules.