RTD bug in Excel beta and preview versions of 2409

[govert]

I have found a bug in the RTD-based features in the beta (as of 2024-08-30) versions of Excel version 2409.

RTD overview

Excel-DNA (and other add-in frameworks like PyXLL, JINX, XLL+, xlOil ), provides an RTD server base class implementation (called ExcelRtdServer), and then RTD-based implementations of async and streaming (IObservable) functions implemented with the same infrastructure.

There is a detailed description of the RTD mechanism with links to Microsoft source material in the Excel-DNA RTD Tutorial.

Affected Excel versions

In the current Beta channel version of Excel, there is a bug that in many cases prevents these features from updating correctly. This also affects RTD servers directly implemented in COM without Excel-DNA, and below we discuss reproducing the bug with a simple RTD server implemented in C#.

I've confirmed the problem with these versions:

• Microsoft® Excel® for Microsoft 365 MSO (Version 2409 Build 16.0.18015.20000) 64-bit [Beta Channel on 2024-08-27]
• Microsoft® Excel® for Microsoft 365 MSO (Version 2409 Build 16.0.18025.20006) 64-bit [Beta Channel on 2024-08-30]
• Microsoft® Excel® for Microsoft 365 MSO (Version 2409 Build 16.0.18025.20024) 64-bit [Current Channel (Preview) on 2024-09-05]

The last version I tested which had no problems was version 2408 (which was in the CurrentPreview channel as of 2024-09-01). So this version was fine:

• Microsoft® Excel® for Microsoft 365 MSO (Version 2408 Build 16.0.17928.20114) 64-bit

Instruction for changing the update channel are here:
Change the Microsoft 365 Apps update channel for devices in your organization - Microsoft 365 Apps | Microsoft Learn

Reverting to an exact earlier versions of Office can be done according to these instructions

Bug details

The RTD call contract indicates that a call to IRTDUpdateEvent.UpdateNotify() (or multiple such calls) must always be followed by a later callback from Excel to the IRtdServer.RefreshData(...) method.

With the affected Excel versions, an RTD server may call UpdateNotify, and not receive any subsequent RefreshData callback. In particular, there is an interaction between the interval between UpdateNotify calls and the RTD ThrottleInterval. When tracing the calls, the pattern might look like this:

[21:54:23.2] ServerStart
[21:54:23.2] ConnectData <<< New Topic created
[21:54:24.2] UpdateNotify <<< Excel is notified of updates
[21:54:24.2] RefreshData  <<< Excel calls back to fetch data
[21:54:25.2] UpdateNotify <<< Excel is notified of updates
>>> No RefreshData (or Heartbeat) callback 
[21:54:51.5] ServerTerminate

In this case the interval between UpdateNotify calls was 1 second, which is smaller than the RTD ThrottleInterval setting in this run, which was the default value of 2 seconds (2000).

• If the RTD Server update interval is made 2s (2000ms) or greater, updates will continue correctly.
• If the RTD.ThrottleInterval is set to a value lower than 1s (1000ms), then updates will also continue as expected.

(Note that for the sample RTD Server, UpdateNotify calls are made regularly, but for many RTD Servers UpdateNotify will only be called when new data arrives, with varying intervals.)

It seems if UpdateNotify calls are made in a period shorter than the ThrottleInterval, Excel will never subsequently call RefreshData. This violates the RTD call 'contract' and is a change of behavior from earlier Excel versions.

Test and reproduction instructions

The bug can be reproduced using the simple C#-based RTD server described here: https://weblogs.asp.net/kennykerr/Rtd6
The code from this blog writeup is available as a GitHub repository here: https://github.com/govert/KennyKerrRTD
(Note that the RTD Server must be 'Registered for COM Interop' and hence Visual Studio must run 'As Administrator' to rebuild the project.)

After building the project, the RTD server can be tested in Excel by calling from a workbook:

   =RTD("Kerr.Sample.RtdServer",,"HH:mm:ss")

With the unchanged RTD server project and default Excel RTD ThrottleInterval, updates work correctly.

To reproduce the bug we can either change the RTD server update interval to 1000 (from 2000ms) or change the Excel RTD ThrottleInterval to 3000 (to make it larger than the RTD server update interval. After this change, the updates will halt after the first refresh.

Impact

Excel-DNA add-ins using the RTD-based features (for streaming values or async functions) call UpdateNotify at irregular times. The Excel-DNA implementation takes care to only call UpdateNotify when needed (the first time there is an update after the last RefreshData call) and will suppress additional update calls. This means under versions with the bug, updates may also work for a while, and then stop unexpectedly.

[govert]

Addition - IRtdServer.Heartbeat not called when expected

According to the Microsoft documentation

The Heartbeat method is called by Microsoft Excel if the HeartbeatInterval property has elapsed since the last time Excel was called with the UpdateNotify() method.

However, we don't see this behavior in the beta versions where we have the above problem.
We do have in place code to make additional UpdateNotify calls from the Heartbeat callback if there are dirty topics.
In the Excel versions with the RTD bug we don't see these Heartbeat calls

I've not checked whether the Heartbeat calls were made as expected in earlier Excel versions.

[govert]

Microsoft has acknowledged the bug:

We're acknowledging the bug seems related to a recent perf optimization currently enabled in Beta release. Investigation started to find the root cause and the fix.

[govert]

Further feedback from Microsoft:

"The fix for the RTD issue has been checked in and should be available in Insiders Beta build 16.0.18108.20000 and later. Please let us know if the problem still persists. Recently we have made some performance improvement around RTD. If you see any other functional or performance issues in RTD, please let us know."

[govert]

Without an update, the RTD problem is no longer present in the Beta or Preview versions I was testing with. The Current version has been updated to 2409 and does not exhibit the problem either (even though it is not at the version indicated for a fix yet). It seems like the changes were reverted by an over-the-air feature flag change.
I don't know what to expect from the future version 16.0.18108.20000 indicated as having a fix for the issue - perhaps this will re-enable the intended performance changes, with a bugfix included.