Greengrass IoT / Home Assistant / Ring-Mqtt dying with "Quota Exceeded" error

[irvintim]

I cannot keep the MQTT-service connected on my Home Assistant when the "ring-mqtt" service is turned on. The connection dies with this message:

2024-05-25 14:09:13.037 WARNING (MainThread) [homeassistant.components.mqtt.client] Disconnected from MQTT server endpoint-redacted-ats.iot.us-west-2.amazonaws.com:8883 (Quota exceeded)

The ring-mqtt configuration (config.json is:

{
  "mqtt_url": "mqtts://endpoint-redacted-ats.iot.us-west-2.amazonaws.com:8883",
  "mqtt_options": "--ca=/data/broker-certificate.pem,--cert=/data/client-certificate.pem.crt,--key=/data/client-private.pem.key,--clientId=Office-Ring-Thing",
  "livestream_user": "redacted",
  "livestream_pass": "redacted",
  "disarm_code": "",
  "enable_cameras": true,
  "enable_modes": false,
  "enable_panic": false,
  "hass_topic": "homeassistant/status",
  "ring_topic": "ring",
  "location_ids": [
    "ring-uuid-redacted"
  ]
}

When I set the HA MQTT to use homeassistant as the discovery prefix, then I see the ring devices being discovered but then the connection closes and everything shows up as "unavailable" in HA. The connection reopens, discovery happens again, and then the connection dies again. This repeats continuously.
If I stop the ring-mqtt container, the discovery logs continue with all the ring devices, so it seems something is cached somewhere and it keeps replaying all the discovery messages. This will continue for quite some time (days) before the log messages finally stop and the MQTT service stays connected on HA.

If I change the Home Assistant MQTT discovery prefix, then none of this happens. None of the ring devices are discovered, and the MQTT connection stays active; if I go to the HA MQTT config page and "Listen to a topic" and subscribe to "#" then I see all the ring MQTT messages going by (along with everything else. In the Greengrass IoT AWS console in the "MQTT test client" I also see all the messages if a subscribe to "#".

I have attached the logs from Home Assistant, which show a complete cycle from the MQTT connection, discovery, the Quota Exceeded message, and the disconnection.
ha-mqtt.log

My environment:

• AWS Greengrass IoT
• Raspberry PI 4
• EMQX MQTT Broker (aws.greengrass.clientdevices.mqtt.EMQX v 2.01)
• aws.greengrass.clientdevices.mqtt.Bridge v2.3.2
• ring-mqtt v 6.6.4 (tsightler/ring-mqtt:latest docker image)
• Home Assistant v. 2024.5.4 (homeassistant/home-assistant:stable docker image)

AWS MQTT Bridge config:

{
    "mqttTopicMapping": {
      "ClientDeviceEvents": {
        "topic": "uhppoted/#",
        "targetTopicPrefix": "office/",
        "source": "LocalMqtt",
        "target": "IotCore"
      },
      "RingDeviceEvents": {
        "topic": "ring/#",
        "targetTopicPrefix": "office/",
        "source": "LocalMqtt",
        "target": "IotCore"
      },
      "HADeviceEvents": {
        "topic": "homeassistant/#",
        "targetTopicPrefix": "office/",
        "source": "LocalMqtt",
        "target": "IotCore"
      }
    },
    "mqtt5RouteOptions": {
      "toIoTCore": {
        "noLocal": true
      }
   }
}   

[tsightler]

I feel like the error message is quite clear, you are exceeding the quota for some service on the MQTT broker. I'm not sure what is driving you to use such an unusual configuration, but I would suggest just using something more common for your MQTT broker rather than trying to use a cloud based MQTT server with many default quota limits that is likely completely untested with Home Assistant or ring-mqtt. Home Assistant itself is quite picky for the MQTT broker (for example ActiveMQ and RabbitMQ are both explicitly unsupported).

Nobody here is going to be able to help you troubleshoot problems with such a setup so you'll be left to your own efforts, but here's a link to various documented quota limits for AWS IoT Core:

https://docs.aws.amazon.com/general/latest/gr/iot-core.html#message-broker-limits

If I were to look at that list, I'd assume the fairly low subscribe per second might be the limit being hit (100 per second seems really low to me). As Home Assistant subscribes for each topic for each device, If you have a reasonable number of devices, it doesn't take many devices to have 100's of subscriptions and those will happen in 100s of milliseconds.

[tsightler]

Actually, looking at that list a little more, I think it might be the subscriptions per connection limit, as it is very low (50). Admittedly, this is all just a guess, but I'm theorizing here since I know that HA subscribes to each topic as the discovery information is sent, so I could see it subscribing, hitting a limit, getting disconnected with the "Quota exceeded" reason, and then the process starting over.

[irvintim]

Again, thanks for taking a look at this. The "subscriptions per connection" limit is something I hadn't seen. So that seems to be a great clue. I'll open a ticket with AWS and see if they can increase the limit.

[irvintim]

Thanks for taking a look! This has been killing me for weeks now.

Yes, I agree the error message "seems" clear; however, if I disable the auto-discovery by changing the HA discovery prefix, I see all the ring (and other mqtt messages by subscribing to "#" on the test clients on both sides (AWS and HA) -- and the HA MQTT connection doesn't drop, and we don't get a quota problem). So, that leads me to the assumption that the problem appears only to occur when HA is discovering the ring devices (and why that happens is I think the mystery I am trying to solve) -- it feels like some sort of loop occurs when discovery is happening and that it what overruns the quota. If I can figure out how to prevent that loop I think I'd be in business.

As to why this setup -- this is an AWS published configuration for Greengrass IoT managed devices:
https://github.com/awslabs/aws-greengrass-labs-component-for-home-assistant
It allows for local control of devices, along with integration with AWS cloud services, e.g. DynamoDB to track status and states, cloudwatch logs for log term logging. This also leverages the Greengrass deployment system for management, updates, and software deployments behind firewalls.

[tsightler]

I feel like even that diagram shows the preferred setup as using Moquette as the local broker for HA, and configuring the MQTT bridge to bridge data to the AWS IoT Core MQTT. I'm guessing the local Moquette service would not have such subscription limits. However your configuration makes it look like you are connecting HA directly to AWS IoT Core.

Regardless, this is all just guesses based on a super fast read of the page. I still think the message is really clear, quota has been exceeded and thus the HA client is being disconnected. What quota is specifically being exceeded is less clear, but something like the subscription limit seems to me that it would be the most likely culprit since subscriptions happen as devices are discovered. Of course disabling discovery would stop the issue, because then the HA MQTT client isn't doing anything and no subscriptions are being created, but, from my perspective, that's a little like saying that if you don't try to drive the car the check engine light doesn't come on.

Interesting setup, seems really complex for a simple home automation setup, but I wish you luck in figuring it out.

[irvintim]

Thank you! I think you hit the nail on the head. I think the config piece that I misinterpreted was the correct end point to use for the local MQTT broker. My assumption was that the magic DNS entry was pointing to the local broker not the core one. And in hindsight I'm not sure why I made that assumption but I think that was an assumption I made a long time ago that was incorrect. I definitely have a bunch to look at now. I appreciate having the extra set of eyes, that's exactly what I needed to get me unstuck. I really appreciate your help.