Background updates eventually hang

[lwis]

iOS device model, version and app version

Model Name: iPhone 15 Pro
Software Version: 17.4
App version: 2024.2

Home Assistant Core Version
2024.3.0

Describe the bug
Eventually background sensor updates stop happening, running the "Update sensors" shortcut times out. Force closing the app brings everything back to life and the shortcut starts working again.

To Reproduce
Have iOS app pushing sensor updates in background for a ~period of time.

Expected behavior
App continues to function without repeatedly force closing.

[bgoncal]

When you open the app, does the sensors get updated correctly? I mean, when it is in foreground all works?
Do you have "local push" enabled?

[lwis]

It does not update when the app is in the foreground.

I do not have local push enabled, or an internal URL configured.

[bgoncal]

Did you observe a pattern for when does it stop reporting? In background I know for sure the sensor updates wont be live without local push enabled but foreground should be working properly.

[lwis]

Unfortunately there doesn't seem to be a pattern to it, I have no way to reproduce. Is there anything useful I can capture the next time it happens?

[bgoncal]

Your logs. You can export going in Debugging > export

[bgoncal]

Can you check the latest testflight beta? It may have fixed this

[lwis]

I'll update to the latest beta and continue to monitor! 🤞🏻

[lwis]

Still happening unfortunately, this is what I see when the shortcut fails;

[dmulcahey]

Home Assistant_2024-03-21_181706.txt
Home Assistant_2024-03-21_192239.txt
Home Assistant_2024-03-21_200921.txt
Home Assistant_2024-03-21_191428.txt

[dmulcahey]

POSIXErrorCode(rawValue: 28): No space left on device) Is thrown constantly by the app when the WS connection breaks

was something changed in the test flight version related to URLsessions or any kind of networking introspection libraries?

[dmulcahey]

@bgoncal let me know if there is anything you need to assist in debugging this.

[bgoncal]

https://stackoverflow.com/questions/67318867/error-domain-nsposixerrordomain-code-28-no-space-left-on-device-userinfo-kcf

This looks related, I'll investigate next week

[dmulcahey]

RC still has the issue. I’ll also note that when this happens and you open the app after it has been in the background it doesn’t connect to HA at all. It fails and HA logs the following:

Logger: homeassistant.components.websocket_api.http.connection
Source: components/websocket_api/http.py:461
integration: Home Assistant WebSocket API (documentation, issues)
First occurred: March 24, 2024 at 6:32:56 PM (6 occurrences)
Last logged: 7:21:09 AM

[281473247231552] from 192.168.1.68 (Home Assistant/2024.3 (io.robbie.HomeAssistant; build:2024.603; iOS 17.4.1)): Disconnected: Did not receive auth message within 10 seconds
[281472302263104] from fe80::1460:ed53:6ed7:34ed (Home Assistant/2024.3 (io.robbie.HomeAssistant; build:2024.603; iOS 17.4.1)): Disconnected: Did not receive auth message within 10 seconds
[281472222859200] from 192.168.1.68 (Home Assistant/2024.3 (io.robbie.HomeAssistant; build:2024.603; iOS 17.4.1)): Disconnected: Did not receive auth message within 10 seconds
[281471861314368] from fe80::1460:ed53:6ed7:34ed (Mozilla/5.0 (iPhone; CPU iPhone OS 17_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Home Assistant/2024.3 (io.robbie.HomeAssistant; build:2024.608; iOS 17.4.1) Mobile/HomeAssistant, like Safari): Disconnected: Did not receive auth message within 10 seconds
[281471753716160] from fe80::1460:ed53:6ed7:34ed (Mozilla/5.0 (iPhone; CPU iPhone OS 17_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Home Assistant/2024.3 (io.robbie.HomeAssistant; build:2024.608; iOS 17.4.1) Mobile/HomeAssistant, like Safari): Disconnected: Did not receive auth message within 10 seconds

hard closing the app and reopening it resolves the issue. I have been running test flight versions since they started getting published again and this just started with this cycle of work: after CarPlay release version so maybe 2 or 3 test flight pushes back.

[dmulcahey]

I just thought of something else. The timing could possibly coincide with the release of iOS 17.4

[bgoncal]

Can you consistently reproduce the issue in some way? Like rebooting HA with the App open for example. I would like to have the same issue happening on my device to investigate it better, for now I couldn't reproduce

[lwis]

Having a Shortcut automation which fires on power connect and disconnect to "Update Sensors" in the background is the way this manifests most often for me.

[dmulcahey]

Can you consistently reproduce the issue in some way? Like rebooting HA with the App open for example. I would like to have the same issue happening on my device to investigate it better, for now I couldn't reproduce

Just leaving the app alone in the background for like 30 mins is all it takes for me

[bgoncal]

@dmulcahey Im checking your logs, when they happened were you somewhere http://192.168.1.78:8123 was accessible? (your home probably) or were you using external URL?

[dmulcahey]

I work from home and am generally here. I just opened the app from the background again. Last open was around the time of my last reply and it happened again. I’ll get new logs for you in a few. Can you give me somewhere to send an iCloud link so I can get you the full bundle from the app?

[bgoncal]

Are you in discord? You can ping me there @bgoncal2

[bgoncal]

Im preparing a testflight build with a possible fix , when you see 2024.612 in testflight you can test it

[dmulcahey]

Im preparing a testflight build with a possible fix , when you see 2024.612 in testflight you can test it

Just saw this update and installed it. Will report back

[dmulcahey]

Im preparing a testflight build with a possible fix , when you see 2024.612 in testflight you can test it

Initial testing looks good. Let’s see how it does overnight

[dmulcahey]

Everything is still working. What was the issue?

[bgoncal]

I'm still investigating but initially I believe the issue is that StarScream lib we use for websocket interaction creates one URLSession per request, which in a retry loop goes over iOS limits. I believe this wasn't happening in some updates ago because we were still supporting iOS 12 at that time and this limit was not applied.
I created a PR to starscream daltoniam/Starscream#1011

[bgoncal]

Heads up that this fix is not on our main branch yet, so if you see new builds coming in testflight, they may revert this fix

[bgoncal]

@lwis can you also check if this works for you?

[lwis]

Updated this morning, so maybe too early to tell, but I've not run into any issues yet.

[bgoncal]

Feel free to reopen this issue if needed, I'm closing this issue for now and planning the next release to include the fix

[lwis]

Bad news; it happened again this morning. I'm unable to run the shortcut directly at all. The app is fully functional. Closing the app allows the shortcut to run again.

I grabbed logs before closing the app.

2024.624

[lwis]

This is still happening, please let me know if there's any way I can help!

[bgoncal]

Im doing some clean up around shortcuts, widgets and intents in general, I'll let you know when you can test it again

[Buddinski88]

@bgoncal I have the same behaviour with the current beta 2024.696 and iOS 17.5.1. Has there been any progress in fixing the bug?

[bgoncal]

Do you see anything in your logs? Like 503 error?
Also do you have any secondary Server that is not reachable when this error occurs?

[Buddinski88]

Do you see anything in your logs? Like 503 error? Also do you have any secondary Server that is not reachable when this error occurs?

Interesting. I've just done some research. If I connect the app directly locally via IP, the update works. So it has something to do with the external connection, which I set up via Cloudflare Tunnel.
When I look at the logs, I see various errors. I've restarted the plugin and so far it's working again. I'll get back to you as soon as the error occurs again.
But it definitely has something to do with the connection via Cloudflare Tunnel.

[bgoncal]

This is still happening, please let me know if there's any way I can help!

Do you have more than 1 server? All of them have external access enabled? Do you see webhook error 503 in your logs?

[alams154]

I’m experiencing the same issue where the update shortcut only works locally. The HA app is able to update the sensors periodically while it is completely closed and does an immediate push after opening the HA app.

I have 2 servers and only one server has external access enabled via Cloudflare Tunnel. I don’t see 503 in my logs but I do see POSIX Error 57.

I’ll try to upload my logs sometime soon if that will help.

[alams154]

Some more information about my setup:
Using cloudflared as tunnel to server
Using traefik as reverse proxy to home assistant
Related traefik router rules:

homeassistant-rtr:
  service: homeassistant-svc
  rule: "Host(`homeassistant.<domain>`)"
  entryPoints: https
  tls: true
  middlewares: authelia@docker
homeassistant-api-rtr:
  service: homeassistant-svc
  rule: 
    Host(`homeassistant.<domain>`) && 
    (PathPrefix(`/api/google_assistant`) || 
      PathPrefix(`/api/webhook`) || 
      PathPrefix(`/api/websocket`) || 
      PathPrefix(`/auth/authorize`) || 
      PathPrefix(`/auth/token`) || 
      (ClientIP(`<network range>`) && !ClientIP(`<specific ip>`))
    )
  entryPoints: https
  tls: true

Here are my logs
Home Assistant.txt
Home Assistant_2024-08-22_064119.txt
Home Assistant_2024-08-23_064242.txt
Home Assistant_2024-08-24_101522.txt
HomeAssistant-Extensions-PushProvider.txt