iOS & AWS iOT: QOS1 mqqt messages get queued and app never recovers

[ylexus]

Using OwnTrack iOS version 12.0.3. I am connected to AWS iOT via mqqt and set a region "Home" and enabled significant changes only. When I leave "Home" and go far further than 500m (a few miles) I can see the local notification that the phone has left the region. However no messages are received by the mqqt subscriber, ever (waited for > 40 mins, even started the app, saw it's successfully connected). There is always a subscriber connected to the topics - guaranteed. I also have an iBeacon at home and I can see the app's been publishing transition and location updates for the beacon's region (although also not without problems - sometimes the transition mqqt messages are lost too!). When I manually publish location from the app it's immediately received by the subscriber. So the question is, is there a way to debug this from the app to find out why the events are lost? Does the app have logs that are accessible somehow?

[ckrey]

As stated numerous times AWS IOT is not a complete MQTT 3.1 implementation. E.g. it does not support QOS2 or retained messages.

Which QOS do you use for publishing? If you use QOS0 you may loose messages

[ylexus]

@ckrey I was using QOS0. I have switched to QOS1 on both OwnTracks and the subscriber side (subscriber is a java application using official AWS IoT API). I am still experiencing intermittent message delivery. Sometimes the app gets completely "stuck", i.e all the location/transition updates simply stop, although local notifications continue as normal. For example, as I am writing this, the last update the subscriber received was 18 hours ago. During this time I received multiple local enter/leave notifications as I've been moving, but no messages on MQQT.

Sometimes (not always) when I launch the app I see this in the "status" field:

idle The operation couldn’t be completed. (OSStatus error -9825.) {
    "_kCFStreamErrorCodeKey" = "-9825";
    "_kCFStreamErrorDomainKey" = 3;
}

However if I tap settings and then back, the status gets changed to "connected". I suspect it's an intermittent SSL handshake error and if it happens I believe the app will queue the update and retry next time, is this correct?

Previously my subscriber was only subscribing to owntracks/+/+/event and owntracks/+/+. I have now changed tags subscription to "#" to see if I receive anything that would give me any clue as to why it happens.

What else can I do to debug the problem?

Here's the app's config (with sensitive data masked), which is based one your AWS IoT example but with QOS changed to 1.

{
  "host" : "a2h2hlgv95ahy1-ats.iot.eu-west-2.amazonaws.com",
  "locatorDisplacement" : 200,
  "subTopic" : "",
  "usepolicy" : false,
  "validatecertificatechain" : false,
  "ignoreStaleLocations" : 0,
  "cleanSession" : true,
  "locked" : false,
  "url" : "",
  "usePassword" : true,
  "positions" : 50,
  "mode" : 0,
  "tls" : true,
  "willRetain" : false,
  "ignoreInaccurateLocations" : 0,
  "deviceId" : "********",
  "extendedData" : true,
  "monitoring" : 1,
  "_type" : "configuration",
  "locatorInterval" : 180,
  "ws" : false,
  "waypoints" : [
    {
      "_type" : "waypoint",
      "tst" : 1561309520,
      "lat" : ********,
      "lon" : ********,
      "rad" : 150,
      "desc" : "Home"
    },
    {
      "_type" : "waypoint",
      "tst" : 1561310162,
      "lat" : ********,
      "lon" : ********,
      "rad" : 0,
      "desc" : "-Home Beacon:********"
    }
  ],
  "username" : "alexey",
  "tid" : "",
  "policymode" : 0,
  "willTopic" : "",
  "clientpkcs" : "ios_owntracks_alexey.otrp",
  "ranging" : false,
  "keepalive" : 60,
  "willQos" : 1,
  "pubTopicBase" : "",
  "auth" : false,
  "passphrase" : "********",
  "allowinvalidcerts" : false,
  "port" : 8883,
  "clientId" : "********",
  "mqttProtocolLevel" : 4,
  "sub" : false,
  "allowRemoteLocation" : true,
  "pubQos" : 1,
  "password" : "********",
  "validatedomainname" : false,
  "subQos" : 1,
  "servercer" : "",
  "cmd" : false,
  "pubRetain" : false
}

[ckrey]

@ylexus you are using "cleanSession" : true,. What is the reason for that?

If you are using this setting, MQTT will throw away all queued messages when you connect the next time.

Please use "cleanSession" : false, otherwise QOS1 does not make sense in a mobile environment with switches from WIFI to cell and disconnectes when the app is put to background, etc.

[ylexus]

@ckrey the reason I use it is this is what your booklet advises for AWS IoT specifically: https://owntracks.org/booklet/tech/mqtt/#broker. Maybe worth correcting it there. Will now change to false and see what happens. Thanks!

[ylexus]

@ckrey I have updated to cleanSession=false and this is what I observe now. The app works fine for a while, say a day, publishing all notifications. At some point it stops publishing and the number on the app badge icon starts growing. If you don’t launch the app it keeps growing, seen it go to 150 or something. During this time I am either at home with excellent Wi-Fi coverage or otherwise in areas with excellent 4G coverage. When you do launch the app you always see the error I mentioned above in the status field (_kCFStreamErrorCodeKey" = "-9825";). When you tap settings and then back (not changing anything), looks like it re-initialises, the status changes to “connected” and the badge icon finally disappears.

So I suspect what happens is there’s a connection error during one of the background refreshes which the app is not able to recover from for a significant amount of time (I have not tried longer than 36 hours) and so gets “stuck” accumulating pending updates. It is able to reconnect instantly when it is reinitialised by opening and closing the settings screen in the app. Any advice here please?

[ylexus]

Sypthoms are the same as in #525 but a slightly different error code: I have -9825 and that other issue was about -9829. I am happy to invest time to help debug this, including installing the app not from App Store if needed. I am a software developer too (although no iOS experience).

[ckrey]

In the meantime we started to replace the underlying network layer with NSURLSession 7ecb822

Pls re-test when the new version becomes available

[ckrey]

probably linked to #605

[ckrey]

@ylexus is the problem still there in versions >= 13.3.6?

[ylexus]

@ckrey I've given up on AWS long time ago in favour of cloudmqtt and deleted all accounts and credentials. Sorry, can't verify now.