Model Data Not Integrating - Production & Development API

[cloneofghosts]

Describe the bug

Looks like HRRR data has stopped integrating on the production endpoint with the last data being integrated at 14Z. The development endpoint is still integrating data without issue as it seems only the production endpoint is affected.

Expected behavior

HRRR data should be integrated every hour

Actual behavior

HRRR hasn't been updated since 14Z

API Endpoint

Production

Location

Ottawa, Ontario, Canada

Other details

No response

Troubleshooting steps

• I have searched this repository and Home Assistant Repository to see if the issue has already been reported.
• I have read through the API documentation before opening this issue.
• I have written an informative title.

[cloneofghosts]

Thought this was only isolated to the production API but it looks like the development API is having the same issue with models not integrating/updating.

@alexander0042

[cloneofghosts]

Whatever the issue was has been fixed.

[alexander0042]

This was actually a good test for the API! NWS had an outage that stopped model data from being disseminated, so there wasn't any data to ingest. However, the fact that nothing crashed and the API gracefully fell back on other sources meant that that side of things is going well!

[cloneofghosts]

Ah it was a NWS issue that caused the issue and nothing on PW's end. Is there an issue with NBM still? It hasn't been integrated since yesterday.

[ktrue]

Yes, they're still having issues
`2024-04-16 05:13:48 EDT - Iva Talley
Additional comments
College Park Datacenter Issue - Update # 4

Key Points

Several storage systems in the College Park data center remain down after Monday afternoon's temperature spike.
Systems have been automatically repairing themselves overnight, but NCO's NetApp Team are working to provide an estimate of when storage systems will be fully recovered.
Additional support was engaged overnight to keep IDP Applications running in Boulder healthy and to investigate mitigating impacts to WPC and OPC operations.
No ETR at this time.
Next update will be at 8am EDT.
Current Known Impacts include:
NCEP Centers' websites hosted in CP are currently inaccessible (including NCO, OPC, EMC, CPC, and NCWCP intranet sites)
WPC, OPC, and CPC's operational product suites are severely degraded
FTPPRD is inaccessible in CP (Customers are able to use nomads.ncep.noaa.gov as a viable backup in the meantime)
See the NCO IT Status Dashboard for the latest updates and more information

SDM Mary Beth`

Also unavailable is the NCO status page https://www.nco.ncep.noaa.gov/status/messages/

[cloneofghosts]

Good to know thanks. Is there a page or anything which has outage information?

[ktrue]

Normally, that's reported at https://www.nco.ncep.noaa.gov/status/messages/
but.. that's one of the servers impacted by the outage.

I'd reported the issue to [email protected] so was added to the mailing list for the problem.
That's where I got that status I'd posted above.
I've not received any status updates at this time, but will post on this issue when I do.

[cloneofghosts]

Okay thanks. Out of curiosity I checked AWS and it seems to be getting NBM data without issue. @alexander0042 Are you not using AWS anymore for model data? I thought you were based on the blog you had posted about the initial API setup.

[cloneofghosts]

This has turned into more of a discussion so I moved it over there.

[ktrue]

The NCEP sites are back up again.