Cascading failures after spurious ORP sensor reading

[lymanepp]

Summary of Problem

A single, bogus ORP reading from Atlas Scientific industrial ORP/pH/RTD probe initiated a chain of events leading to a very high chlorine level. It would probably still be chlorinating if I hadn't noticed things were wonky.

Here's a view of interesting values stored in Home Assistant from a very interesting week.

Pool Equipment

Intelliflo VS (ca 2016) connected via RS485
Hayward W3AQR15 (new) connected via RS485
Pentair IntelliChem 3.5 gallon acid tank (diluted from 31.45% to 15.725%)
Aquacal Tropical T115 (ca 2016) connected via fireman's switch

Controller

Altelix 20x16x8 enclosure
Raspberry Pi 4 Model B Rev 1.2 4GB
Sequent Smart Fan v4.0
Sequent MegaBAS v4.2
Sequent Eight Relay v3
Whitebox T3 with Atlas Scientific EZO boards for ORP, pH and RTD

Inline sensors

Atlas Scientific Industrial ORP/pH/RTD probe
5 volt pressure sensor
Flow sensor
10K temperature sensor in thermowell

Software

nodejs-poolController

• configured to use flow sensor
• default pump speed of 750 rpm
• configured to only read pH and ORP when flow is detected
• configured to controllacid pump and chlorinator

nodejs-poolController-dashPanel
relayEquipmentManager
njsPC-HA custom integration for Home Assistant by @Crewski

Events

12 Feb 14:34:18 - ORP reading dropped from 797 mV to 79 mV for 5 seconds due to weird sensor reading in REM (see below for REM log).
12 Feb 14:34:21 to 16:12:36 - Chlorinator target was 100% for 52% of this time period
12 Feb 16:13:22 - Chlorinator target output stayed at 100% with 5 second pauses every 15 minutes
12 Feb 17:20:59 - ORP reading dropped from 641 mV to 64 mV for 5 seconds. ORP readings dropped by 156 mV in less than 3 hours. (see below for REM log)

13 Feb 03:09:46 - ORP dosing status went from mixing to monitoring where it remained even though chlorinator was still generating most of the time until I changed the ORP set point.

15 Feb 12:17:33 - Flow sensor stopped detecting flow, but the pressure sensor shows continued flow, so njsPC stopped reading ORP and pH levels. The final ORP reading was 590 mV.
15 Feb 14:30:00 - I just happened to notice that the flow sensor was off and I started investigating. Taylor test showed free chlorine was 20.5 ppm.

16 Feb 17:24:05 - Restarted Raspberry Pi to see if restarting would make flow sensor work. It didn't and Sense energy monitor showed that the chlorinator was still generating.
16 Feb 17:31:00 - Changed pump to 3,000 rpm to see if flow sensor would detect flow. It didn't.
16 Feb 18:00:00 (approx) - Changed njsPC to use the pressure sensor instead of flow sensor. Changed ORP set point to 500 ppm to stop chlorinator. Later changed to 400 ppm to be safe.
16 Feb 18:00:27 - njsPC reported flow detected for 1:33:20
16 Feb 18:10:05 - Chlorinator target output became zero and has stayed there since.

17 Feb 14:17:00 - Pulled the latest njsPC changes and restarted. njsPC is reporting continous flow since restart (and pool switched on).

Background

This is the second time this series of events has happened. I reported this to Atlas Scientific on 12/26 and they replied that "I have seen this issue before in swimming pools. Chlorinated water is highly reactive, you are detecting a local chemical reaction within the pipes. As an experiment, try temporarily moving the probe into the pool. Take readings near the intake port and the output port."

I replied that it wasn't really feasible due to the controller being mounted to the wall and the 10' cable not being long enough to reach. I finally recalibrated the ORP/pH/RTD probe on 4 Feb. ORP and pH were off by a lot since installation in early December.

Observations

1. I've observed spurious values from various njsPC sensors (in Home Assistant). Is it caused by the sensor or REM? Regardless, it would be ideal if some sanity checks filtered them out.
2. njsPC relentlessly overreacted to a single spurious ORP reading.
3. The chlorinator kept generating even when no flow was detected by njsPC. Even though values reported by njsPC indicate that it wasn't chlorinating. What the heck??
4. njsPC had bug(s) when using pressure sensor instead of flow sensor. These appear to be fixed with the latest commits. Fingers crossed.
5. Atlas Scientific's industrial ORP sensor readings drop when chlorine is actually rising rapidly. That seems very wrong.
6. My cheapo flow sensor appears to be flaky. I increased the pump speed to 850 rpm to hopefully prevent this. Even though I'm now using the pressure sensor to detect flow, this matters because I'm also using the same sensor for the chlorinator. Is there a way to control a REM relay to replace the chlorinator's flow sensor? That would ensure that they were always in-sync.
7. REM-out.log would be easier to analyze if it had a timestamp on every line.
8. There's something going on with the water temperature 10k readings. They magically rise 20 degrees F in a short time when it wasn't very warm. And then it magically dropped just as quickly. I'll investigate more and create a second discussion if warranted.

REM-out.log corresponding to bogus ORP reading at 12 Feb 14:34:18

debug: Atlas_EZO-ORP command R bytes written:1 result:79r.8
                                                      ^^^^^
verbose: Emitting: /chemController : {"orpLevel":79,"id":50}
verbose: Feed njspc/Pool Controller sending orpLevel: 79 to i2c:1:4

REM-out.log corresponding to bogus ORP reading at 12 Feb 17:20:59

debug: Atlas_EZO-ORP command R bytes written:1 result:64h.7
                                                      ^^^^^
verbose: Emitting: /chemController : {"orpLevel":64,"id":50}
verbose: Feed njspc/Pool Controller sending orpLevel: 64 to i2c:1:4

njsPC-out.log.gz
REM-out.log.gz
ORP-sensor-invalid-values.txt

[lymanepp]

I added primitive validation to my fork of REM. I'll create a PR after using it for a while.

diff --git a/i2c-bus/AtlasEZO.ts b/i2c-bus/AtlasEZO.ts
index 19984d3..96ff9fe 100644
--- a/i2c-bus/AtlasEZO.ts
+++ b/i2c-bus/AtlasEZO.ts
@@ -439,9 +439,12 @@ export class AtlasEZOorp extends AtlasEZO {
         finally { this.suspendPolling = false; }
     }
     public async readProbe(): Promise<number> {
+        const validResultPattern = /^\d+(\.\d+)?$/;
         try {
             this.suspendPolling = true;
             let result = await this.execCommand('R', 900);
+            if (!validResultPattern.test(result))
+                throw new Error('Invalid ORP probe value');
 
             let val = this.i2c.isMock ? 666.66 + (Math.floor(Math.random() * 10000) / 100): parseFloat(result);
             this.values.orp = val;

[rstrouse]

Huh, the I2c return is invalid? How is your flow sensor connected? It is on the I2c bus since that data in your text file kinda smells like an I2c reflection? You can smooth out the (especially with ORP) outlier values if you set REM up to use median averaging based on 5 or so samples but given the I2c responses in that file it looks like it still would end up with some goofball swings.

I did notice that the voltage from the ORP and pH sensors are reading under 4 volts. What power supply are you using?

[lymanepp]

Yes, my REM-out.log file contains 57 invalid ORP readings out of 840,403 total readings. While I was making changes a few days ago, I also changed ORP and pH sensors to average the last 6 values. That would reduce the size of the swing, but it would likely still be more than 100 mV for ORP. The filtering-out of bad values in REM takes care of it completely. 🙂

I'm powering the RPi stack via MegaBAS with 24V power from a Mean-Well NDR-240-24. I noticed that the HAT is reporting 22V with just over 5V of RPi power. I initially calibrated the power supply to 24V, but I'll need to check it again.

The ORP sensor is reporting 3.95V. Any ideas why this would be 23% lower than what MegaBAS is reporting?

The flow sensor is attached to the MegaBAS as a "dry contact". I was aware of the potential for reflections with RS485, but this is the first time I've heard of i2c reflections. Any ideas how to resolve that in the RPi stack?

FWIW, here's a picture of my RPi stack.

[rstrouse]

22vdc out of the rectifier on the mega bas is about right. Perhaps ask Atlas what the v2.13 vcc voltage should be. I2c can reflect if another device pulls the data line. Most of the time this is due to long lines that cause capacitance on the bus. I was referring to your pressure sensor not the flow switch.

[rstrouse]

Disconnect the 5v pressure sensor and check the voltage on the EZOs.

[lymanepp]

I disconnected the plug from the sensor. I also disabled the input in REM. Neither had any effect on the EZO voltage.

[rstrouse]

Contact Atlas and ask them what the vcc should be with v2.13 of that board.

[lymanepp]

The documentation for the EZO ORP and pH boards says they both support 3.3V to 5.5V.

[rstrouse]

I am not quite sure where 3.95v fits in that scenario. I have only seen it operate at the 5v level in the past but they did change some hardware at some point so this may be normal for your hardware. If it is not and the regulators are using an incorrect vcc reference voltage then this could be an issue. REM is pulling this from the I2c status. Theoretically, it should show a brown out if the voltage drops below its requisite.

[lymanepp]

I received the following from Atlas support.

Invalid characters is not a known issue of the EZO circuits, or if it does happen, something is causing it. The 3.95V is correct, that is what the isolator chip produces, this wouldn't create a brown out.

Another interesting thing I found is that the corrupt I2C responses only come from the EZO-ORP circuit and never from EZO-pH. I also get invalid STATUS responses from the ORP board. This seems like a bad ORP probe or more likely the EZO-ORP circuit.

[rstrouse]

It certainly does since the returns are reading a null terminated string that is corrupt off the I2c bus. My guess would be the circuit.

[lymanepp]

Resolved items:

1. The new EZO-ORP will be shipped soon.
2. I added filtering of invalid EZO-ORP, EZO-pH and EZO-RTD sensor values to my fork of REM. I also submitted a PR to @rstrouse.
3. My PR includes changes to add date & time stamp to all REM log messages.

[20/02/2024, 06:36:43] error: Atlas_EZO-ORP Received invalid value from I2C result:735j1
[21/02/2024, 04:05:01] error: Atlas_EZO-ORP Received invalid value from I2C result:50.8
[21/02/2024, 17:56:46] error: Atlas_EZO-ORP Received invalid value from I2C result:71b.6

Unresolved items:

1. njsPC relentlessly overreacted to a single spurious ORP reading.
2. The chlorinator kept generating even when no flow was detected by njsPC. Even though values reported by njsPC indicate that it wasn't chlorinating.

@tagyoureit, do you have any insights into these?

[tagyoureit]

Corrupt data can do weird things. I'd see if your problem is solved with the new hardware.