Owntracks stops responding with a client certificate

[DConcord]

Hi, I believe I'm experiencing a similar/same issue to #525

Deployment details: MQTT over TLS (not http/websocket) on TCP 443 with username/password AND client certificates (Internal CA - Openssl). Static public IP assigned to the MQTT server in a DMZ via NAT, but the same public IP is used externally and internally

MQTT Server: Mosquitto 1.6.4
Relevant Mosquitto Configuration: "require_certificate true" and " use_identity_as_username false" in order use both username/password AND client certificate

Phone: XR on Owntracks 13.0.2 (Although I've seen the issue previously as well

Protocol: MQTT (not websocket) on TCP 443

Issue:
After about 8-12 hours, one of my two deployed s stops responding with a client certificate. Connection attempts continue, but in the Mosquitto logs I see:

OpenSSL Error: error:140360C7:SSL routines:ACCEPT_SR_CERT:peer did not return a certificate

In Owntracks, I open info (i) and see the following error:

idle The operation couldn't be completed. (OSStatus error -9829.) {
  "_kCFStreamErrorCodeKey" = "-9829";
  "_kCFStreamErrorDomainKey" = 3;
}

I just need to go further into settings and back out for everything to resume properly for another 8-12 hours

I do believe there's a correlation between switching between networks as the phone was working consistently from the same carrier IP address, then it switched to a wifi/terrestrial address, and after it tried to return to the same carrier IP address, the issue continued from then-on. All of these addresses are outside/external to my MQTT server:

(IPs and name sanitized for privacy)

!Cellular Carrier public IP - working properly
1571762440: New connection from 108.X.X.57 on port 443.
1571762440: New client connected from 108.X.X.57 as UserA (p2, c0, k60, u'UserA').
1571762455: Client UserA disconnected.
1571762695: New connection from 108.X.X.57 on port 443.
1571762695: New client connected from 108.X.X.57 as UserA (p2, c0, k60, u'UserA').
1571762710: Client UserA disconnected.
1571762995: New connection from 108.X.X.57 on port 443.
1571762995: New client connected from 108.X.X.57 as UserA (p2, c0, k60, u'UserA').
1571763010: Client UserA disconnected.
1571763051: New connection from 108.X.X.57 on port 443.
1571763052: New client connected from 108.X.X.57 as UserA (p2, c0, k60, u'UserA').
1571763066: Client UserA disconnected.

!Comcast/Terrestrial Wifi IP - last working connection
1571763362: New connection from 66.X.X.32 on port 443.
1571763362: New client connected from 66.X.X.32 as UserA (p2, c0, k60, u'UserA').
1571763377: Client UserA disconnected.

!Possibly unrelated phishing as this IP is from an unrecognized ISP in a different location
1571765490: New connection from 128.X.X.134 on port 443.
1571765490: OpenSSL Error: error:140360C7:SSL routines:ACCEPT_SR_CERT:peer did not return a certificate
1571765490: Socket error on client <unknown>, disconnecting.

!Back to the same Cellular Carrier public IP as before - no longer working properly from here, onward
1571770672: Outgoing messages are being dropped for client UserA.
1571771419: New connection from 108.X.X.57 on port 443.
1571771420: OpenSSL Error: error:140360C7:SSL routines:ACCEPT_SR_CERT:peer did not return a certificate
1571771420: Socket error on client <unknown>, disconnecting.
1571771502: New connection from 108.X.X.57 on port 443.
1571771503: OpenSSL Error: error:140360C7:SSL routines:ACCEPT_SR_CERT:peer did not return a certificate
1571771503: Socket error on client <unknown>, disconnecting.
1571771582: New connection from 108.X.X.57 on port 443.
1571771582: OpenSSL Error: error:140360C7:SSL routines:ACCEPT_SR_CERT:peer did not return a certificate
1571771582: Socket error on client <unknown>, disconnecting.
1571771802: New connection from 108.X.X.57 on port 443.
1571771802: OpenSSL Error: error:140360C7:SSL routines:ACCEPT_SR_CERT:peer did not return a certificate
1571771802: Socket error on client <unknown>, disconnecting.
1571771978: New connection from 108.X.X.57 on port 443.
1571771978: OpenSSL Error: error:140360C7:SSL routines:ACCEPT_SR_CERT:peer did not return a certificate
1571771978: Socket error on client <unknown>, disconnecting.

!Century Link ISP - I do believe this is the same client. Still external to the MQTT server
1571772103: New connection from 72.X.X.142 on port 443.
1571772103: OpenSSL Error: error:140360C7:SSL routines:ACCEPT_SR_CERT:peer did not return a certificate
1571772103: Socket error on client <unknown>, disconnecting.

Additional Info
I have two XRs deployed. One of them has this issue repeatedly, and other does not have the issue at all.

They both use Client certificates sequentially issued from the same CA/Issuing CA using the same cert template, and I actually used a variation of the same password on the PKCS certificate bundle that uses the same letters/symbols/length but in a different order. I've triple checked that both phones have imported and trusted the CA and Intermediate CA the same way into the Profile/certificate settings and I compared all standard and even advanced settings to make sure everything is identical. So its pretty puzzling why one works consistently and the other doesn't.

Everything works wonderfully when I disable "require_certificate" in Mosquitto, but I'd really love to be able to use this feature.

thanks!

[ckrey]

In the meantime we started to replace the underlying network layer with NSURLSession 7ecb822

Pls re-test when the new version becomes available

[DConcord]

Hey @ckrey Just revisited this and still having the same problem. Interestingly, I found this recent (2 mo old) thread reply by Apple support agent "eskimo" where he says that it's a known issue for NSURLSession but has been fixed WKWebView. Ignorance disclaimer alert!: is there any way WKWebView could be utilized rather than NSURLSession? Thanks!

developer question snippet:
Are client certificate authentication challenges in a background session still not supported as of iOS 13.3.1?

eskimo's answer:
Sadly, that’s still the case. This was fixed for a

WKWebView

a while back, but

NSURLSession

background sessions still suffer from this limitation (r. 17526855).

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

https://developer.apple.com/forums/thread/28713

[ckrey]

@DConcord thanks for following this up.

Via NSURLSession, NSURLSessionStreamTask / NSURLSessionWebSocketTask are used for MQTT / Websocket connections. There is no equivalent in WKWebView for that.

If you drop me your email, I will invite you to the beta testing channel to check if NSURLSession in iOS13 solves your problem.

[hanche]

Thanks for sending me the Testflight version (as you noted over on #642). Unfortunately, it never managed to connect. Status remained “connecting”, and on the server side, I got this in the log (real addresses replaced by IPv[46]:

20200824_201247: New connection from IPv4 on port 8883.
20200824_201247: New connection from IPv6 on port 8883.
20200824_201247: Socket error on client <unknown>, disconnecting.
20200824_201248: OpenSSL Error: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown
20200824_201248: Socket error on client <unknown>, disconnecting.

[ckrey]

I guess you are using a self signed certificate on the server. In the iOS13 versions, you will have to install the server CA file in iOS to establish the trust. The OwnTracks option to suppress domain name validation is no longer used.

https://owntracks.org/booklet/features/tls/#ios

[ckrey]

I fixed another related problem. Version 13.3.4 should be available through Testflight soon

[hanche]

Well no, unless I understood you incorrectly. I have created my own CA certificate, which is installed on my phone and marked as Verified in nice green letters in the Settings app. And then I have created server and client certificates using that CA certificate: The former is used by the mosquitto server, and (one of) the latter in the OwnTracks app. Pretty sure I followed the instructions, and it does work in the most recent version on the App Store, just not in the Testflight version. I'll by sure to try again when a new Testflight version comes out, though.

Edited to add: I did not redo the setup steps with the Testflight version. I just assumed it would inherit everything from the App Store version, and looked briefly into Settings and checked that it all looked right.

[hanche]

No change for me with the latest testflight version. But I have no time to experiment with different settings right now. Maybe later. (Except I don't know what to change.)

[ckrey]

[hanche]

With that setting, it connects. But I feel uneasy about the security implications. Not that I think it likely that someone will try to trick owntracks on my phone to connect to their MQTT server instead of my own, but I'd prefer if they can't.
Anyhow, for now I'll keep it this way and see if it connects more reliably.

[ckrey]

You don't need to enable Allow Untrusted Certificates if your server's certificate does adhere to the new checking policy Apple introduced with iOS 13:

https://support.apple.com/en-us/HT210176

[jpmens]

FWIW, we've fixed our certificate generation utility to adhere to those rules: server certificates are issued with a validity of 825 days.

[hanche]

@ckrey Thanks for the tip. I'll check it out.

[hanche]

Today, some time after leaving home, I saw that the OwnTracks icon had a badge with a count 5 on it. I assume that is the number of location updates it has stacked up locally while it could not connect to the server, is that correct?

Incidentally, I am getting messages about entering or leaving regions only when it seems unable to connect. Is that by design?

Byt anyhow, upon entering the info page of the app, I noticed that the status was flip-flopping between “idle” and “connecting”. As soon as I entered the Setup screen, though, it connected – and it showed “connected” back on the infor screen, and the icon badge was gone.

Looking at the mosquitto log, I once more see the “peer did not return a certificate” message at around that time. The log shows many such failed attempts, and a whole bunch of them before a connection finally succeeded. I suppose that was when I entered the Setup screen, but can't be sure, since I did not make note of the time. I'll try to remember that next time.

In any case, this is with the testflight version of the app. (Version 13.3.1.)

Oh wait, going into TestFlight, I see there's an update. (Oddly, the testflight icon showed no badge, nor did I get a notification about it.) So this was probably the version before the latest? But I figured it does not hurt to send this in anyhow.

[ckrey]

You need OwnTracks >= 13.3.4 from Testflight to solve the problem you saw

[hanche]

FWIW, I had the “peer did not return a certificate” problem again this morning. I noticed owntracks showed a badge count of 50. Status was shown as idle, periodically flipping to something else and back too quick for me to read it. It resolved itself when I entered Settings. Returning from Settings, status was now connected, and the badge count was gone. In the mosquitto log, I saw several failed attempts to connect, followed by successful connection, and a bunch of PUBLISH/PUBACK lines.

I am now on iOS 14, owntracks 13.3.5 (latest version from testflight).

[ckrey]

You may try 13.3.6 which is now both in TestFlight and on the Public App Store. But I think the problem is unrelated.

One question: when you are in your Wifi environment, does the broker name resolve to a different IP compared to when you are in Cellular enviroment?

[hanche]

The broker has both IPv4 and IPv6 connectivity. My cellular provider does not support IPv6, however. (I am considering switching for that reason.) On WiFi at home and at work I have both, but it appears always to prefer IPv6 in that case.

[jpmens]

it appears always to prefer IPv6

All applications prefer v6 and actually first do AAAA DNS lookups before falling back to A. :-)

[ckrey]

The failing clients don't seem to connect via IPv6 (see 108.X.X.57 below)

1571771419: New connection from 108.X.X.57 on port 443.
1571771420: OpenSSL Error: error:140360C7:SSL routines:ACCEPT_SR_CERT:peer did not return a certificate
1571771420: Socket error on client <unknown>, disconnecting.

[hanche]

The failed attempt this morning were IPv6. But one has to read the logs carefully; there is no shortage of unwelcome clients out there trying to connect now and then. Needless to say, they don't present a certificate.

[ckrey]

You may lower the number of unwelcome clients by choosing a port number other than 443, e.g. 8883

[hanche]

My server is already on port 8883. That is the default port for this kind of service, isn't it? Anyway, the number of unwelcome clients is not all that large. Only enough to make me need to check the logged errors with a bit more care. I suppose I could move to a totally random port, though. That should lower the number to near zero.

[ckrey]

Sorry you are right, I was looking at the log output of @DConcord