freeswitch randomly overwrites files that it has open in r/w mode

[themsley-voiceflex]

Describe the bug
Freeswitch randomly writes SSL data to files that it has open in r/w mode due to some sort of race condition.

To Reproduce
Steps to reproduce the behavior:

1. Set up a freeswitch profile to use WSS webrtc
2. Expose it to the internet and allow people to attempt to use it
3. after some time freeswitch will overwrite random filehandles that it has open with SSL data
4. FreeSWITCH crashes with sqlite "file is not a database" messages in the majority of cases as the filehandle that is most often used and in r/w mode is the one for /dev/shm/core.db. This bug could potentially overwrite any file that is open in r/w mode although the more often it is opened the more likely it is to be a victim which is probably why it seems to pick on /dev/shm/core.db

Expected behavior
freeswitch does not do random things!

Package version or git hash
1.10.10

This is the same issue previously reported in #420 which was mistakenly closed as a SQLite problem.

I am 100% certain that it not for the following reasons:

1. we have the patched version of SQLite installed and in use

2. the bug in SQLite pointed to from core db gets corrupted  #420 https://www.philipotoole.com/how-i-found-a-bug-in-sqlite/ is in a specific SLQite area that needs special action to invoke and freeswitch does not use this. This means the bug will probably never affect freeswitch so a SQLite upgrade is unnecessary. The SQLite bug as reported in that link affects only databases using this specifc set up https://www.sqlite.org/inmemorydb.html and to the best of my knowledge this way of using SQLite is not used by freeswitch. It is 100% definitely not used by our installation. And even if it was (which it isn't!) we have the patched version running.

3. our freeswitch instance is set up to use param name="core-db-name" value="/dev/shm/core.db" which is an ordinary file based SQLite database. It just happens to be on a filesystem that is 'in memory' but that is not the same thing as a SQLite 'in memory' database as described above. All these reasons rule out a SQLite problem.

In addition I have added debug code to libsofia-sip-ua/tport/ws.c in the ws_close() function, immediately prior to the SSL_write() that it uses. This debug code is

                /* check if no fatal error occurs on connection */
                char procbuf[1024];
                char fnbuf[1024];
                ssize_t fnlen;
                sprintf(procbuf,"/proc/self/fd/%d",wsh->sock);
                if ((fnlen = readlink(procbuf,fnbuf,sizeof(fnbuf)-1)) != -1) {
                        fnbuf[fnlen] = '\0';
                        printf("WS ws_close fd %d target %s\n",wsh->sock,fnbuf);
                        }
                else {
                        printf("WS ws_close fd %d readlink failed\n",wsh->sock);
                        }
                code = SSL_write(wsh->ssl, buf, 1);

and when the bug is hit it shows the following information

Oct 18 07:50:47 fstrtc01.voiceflex.com stdbuf[389441]: WS ws_close fd 97 target /dev/shm/core.db
Oct 18 12:20:39 fstrtc01.voiceflex.com stdbuf[395088]: WS ws_close fd 63 target /dev/shm/core.db
Oct 18 19:20:46 fstrtc01.voiceflex.com stdbuf[403552]: WS ws_close fd 58 target /dev/shm/core.db

The SSL_write in ws_close() is issued immediately after that debug message and is writing SSL data to random files as per the similar problem that Facebook engineers found and debugged in their code referenced https://engineering.fb.com/2014/08/12/ios/debugging-file-corruption-on-ios/

"The SSL layer was writing to a socket that was already closed and subsequently reassigned to our database file. "
and
"Using a hex analyzer, we found a common prefix across the attachments: 17 03 03 00 28"

Our overwrite is not identical but similar enough for it to be the same problem:

$ for dbfile in $(ls /dev/shm/core.db-*); do echo $dbfile; od -X $dbfile | head -1;done
/dev/shm/core.db-1697017847
0000000 00030317 2299b112 b8d06713 442802c5
/dev/shm/core.db-1697359841
0000000 00030317 10e68912 769c4a0c 587aac68
/dev/shm/core.db-1697457656
0000000 00030317 cdc06712 26b9d175 b91379ec
/dev/shm/core.db-1697474154
0000000 00030317 0298eb12 10f4a06d a38d8a70
/dev/shm/core.db-1697477136
0000000 00030317 39243212 f67342eb 4eedfd89
/dev/shm/core.db-1697537157
0000000 00030317 bbc1a212 0dd52644 ae8bd737
/dev/shm/core.db-1697547645
0000000 00030317 c42d2612 233c9109 1cb91ff7
/dev/shm/core.db-1697569857
0000000 00030317 9e3fe612 3b797957 df9c0fba
/dev/shm/core.db-1697604657
0000000 00030317 af282512 3c08939a ac980ecf
/dev/shm/core.db-1697615458
0000000 694c5153 03031774 dd7c1300 2223e3b0
/dev/shm/core.db-1697631649
0000000 00030317 21e31112 bad65ac4 f6609114
/dev/shm/core.db-1697656856
0000000 00030317 42088b12 b27bce2d f3ed235c

Only /dev/shm/core.db-1697615458 does not start with 0x1703030012 and that has 0x1703030013 at +5 into the file.

Within a few seconds of those debug messages being issued we start to get

2023-10-18 07:34:41.466490 99.60% [WARNING] sofia_reg.c:1842 SIP auth challenge (REGISTER) on sofia profile 'internaltcp' for [webrtc@webrtc.nebulaip.com] from ip x.179.208.228
2023-10-18 07:50:47.946480 99.60% [WARNING] sofia_reg.c:1842 SIP auth challenge (REGISTER) on sofia profile 'internaltcp' for [webrtc@webrtc.nebulaip.com] from ip x.8.18.233
2023-10-18 07:50:48.026471 99.60% [WARNING] sofia_reg.c:1842 SIP auth challenge (REGISTER) on sofia profile 'internaltcp' for [webrtc@webrtc.nebulaip.com] from ip x.8.18.233
2023-10-18 07:50:48.026471 99.60% [ERR] switch_core_sqldb.c:728 [db="/dev/shm/core.db",type="core_db"] NATIVE SQL ERR [file is not a database]
BEGIN EXCLUSIVE
2023-10-18 07:50:48.026471 99.60% [CRIT] switch_core_sqldb.c:2109 ERROR [file is not a database], [db="/dev/shm/core.db",type="core_db"]
2023-10-18 07:50:48.026471 99.60% [ERR] switch_core_sqldb.c:728 [db="/dev/shm/core.db",type="core_db"] NATIVE SQL ERR [cannot commit - no transaction is active]
COMMIT
2023-10-18 07:50:48.086485 99.60% [WARNING] sofia_reg.c:1842 SIP auth challenge (REGISTER) on sofia profile 'internaltcp' for [webrtc@webrtc.nebulaip.com] from ip x.8.18.233

There are identical messages in the logs for the core.db overwrite at 12:20:39 and 19:20:46

Once this database is overwritten in this way then freeswitch will refuse to start up as the core.db.dsn database is corrupted and cannot be opened. It has to be deleted/renamed for freeswitch to start up again.

I also see hangs in freeswitch where it stops responding to connection attempts and a gcore taken at the time of this shows that it is stuck in ws_close() in the middle of the SSL_write() call. I suspect this is related and we are sending to a socket that is not expecting us to write to it (wild abandoned guess!) but this can be stuck there waiting for sometimes hours. I suspect this is also involved in #1934 where people are reporting hangs when attempting to use WSS. This is from the most recent gcore taken for fd 49 which hung from Oct 19 15:23:25 to 16:26:43 when it woke up again.

(gdb) thread apply 18 bt full

Thread 18 (Thread 0x7f078bfff640 (LWP 450637)):
#0  0x00007f07bd53e91c in read () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f07bccdd091 in sock_read () from /lib64/libcrypto.so.3
No symbol table info available.
#2  0x00007f07bcccd44b in bread_conv () from /lib64/libcrypto.so.3
No symbol table info available.
#3  0x00007f07bccd0445 in bio_read_intern () from /lib64/libcrypto.so.3
No symbol table info available.
#4  0x00007f07bccd05c7 in BIO_read () from /lib64/libcrypto.so.3
No symbol table info available.
#5  0x00007f07bd130e8c in ssl3_read_n.part () from /lib64/libssl.so.3
No symbol table info available.
#6  0x00007f07bd132af9 in ssl3_read_bytes () from /lib64/libssl.so.3
No symbol table info available.
#7  0x00007f07bd1425c0 in state_machine.part () from /lib64/libssl.so.3
No symbol table info available.
#8  0x00007f07bd13076e in ssl3_write_bytes () from /lib64/libssl.so.3
No symbol table info available.
#9  0x00007f07bd1136e7 in SSL_write () from /lib64/libssl.so.3
No symbol table info available.
#10 0x00007f07bd2a5340 in ws_close (wsh=wsh@entry=0x7f0781f71240, reason=reason@entry=0) at tport/ws.c:900
        ssl_error = 0
        buf = 0x7f07bd2d41b6 "0"
        procbuf = "/proc/self/fd/49\000\000\316\201\a\177\000\000\320\347\377\213\a\177\000\000\310\347\377\213\a\177\000\000\204\236(\275\a\177\000\000\220y2\200\a\177\000\000\300\240\061\275\a\177\000\000P\303\316\201\a\177\000\000\200\350\377\213\a\177\000\000\000\000\000\000\000\000\000\000֢(\275\a\177\000\000\377\377\377\177\000\000\000\000\064Z)\275\a\177\000\000tag=ZNS5\337Q-\275\a\177\000\000\377\377\377\377\377\377\377\377?B\017\000\000\000\000\000P\303\316\201\a\177\000\000\000\326\354\367\376AN\245\377\377\377\177\000\000\000\000P\303\316\201\a\177\000\000\000\000\000\000\000\000\000\000\365\022*\275\a\177\000\000P\303\316\201\a\177\000\000\000\067a\202\a\177"...
        fnbuf = "socket:[9955738]\000\000\000\000\000\000\000\000d\002\000\000\000\000\000\000\001", '\000' <repeats 15 times>, "\307\002\000\000\000\000\000\000\325\371J\275\a\177\000\000\236\304\333\350\000\000\000\000\273\360\b\000\000\000\000\000@\345\377\213\a\177\000\000@\345\377\213\a\177\000\000\340\vy\202\a\177\000\000\361\001*\275\a\177\000\000\001\000\000\000\000\000\000\000\240\344\377\213\a\177\000\000\060\067a\202\a\177\000\000\001\201\001\200\a\177\000\000\312\326,\275\a\177\000\000\020\202\001\200\a\177\000\000d\002\000\000\000\000\000\000`.Ё\000\000\000\004\000\nV\022C\000\000\000\000\003\000\n]_|\006", '\000' <repeats 25 times>...
        fnlen = <optimized out>
        code = 0
#11 0x00007f07bd2a5bd3 in ws_destroy (wsh=0x7f0781f71240) at tport/ws.c:836
No locals.
#12 0x00007f07bd2a5ca3 in tport_ws_deinit_secondary (self=0x7f0781f71050) at tport/tport_type_ws.c:536
        wstp = <optimized out>
        wstp = <optimized out>
        __func__ = {<optimized out> <repeats 26 times>}
#13 tport_ws_deinit_secondary (self=0x7f0781f71050) at tport/tport_type_ws.c:530
        wstp = 0x7f0781f71050
        __func__ = "tport_ws_deinit_secondary"
#14 0x00007f07bd294df8 in tport_zap_secondary (self=0x7f0781f71050) at tport/tport.c:1101
        mr = <optimized out>
        __func__ = "tport_zap_secondary"
#15 0x00007f07bd287ed8 in su_timer_expire (timers=timers@entry=0x7f0780000ba8, timeout=timeout@entry=0x7f078bffec50, now=...) at su/su_timer.c:587
        t = 0x7f07836cc9e0
        f = 0x7f07bd296ed0 <tport_secondary_timer>
        n = 42
        __PRETTY_FUNCTION__ = "su_timer_expire"
#16 0x00007f07bd288165 in su_base_port_run (self=0x7f0780000b60) at su/su_base_port.c:339
        now = {tv_sec = <optimized out>, tv_usec = <optimized out>}
        tout = 15000
        tout2 = 0
        __PRETTY_FUNCTION__ = "su_base_port_run"
#17 0x00007f07bd28b1f3 in su_pthread_port_clone_main (varg=0x7f07b916b580) at su/su_pthread_port.c:343
        arg = 0x0
        task = {{sut_port = 0x7f0780000b60, sut_root = 0x7f07800013d0}}
        zap = 1
#18 0x00007f07bd49f812 in start_thread () from /lib64/libc.so.6
No symbol table info available.
#19 0x00007f07bd43f450 in clone3 () from /lib64/libc.so.6

[themsley-voiceflex]

A not overwritten core.db SQLite database should start with

$ od -X /dev/shm/core.db | head -1
0000000 694c5153 66206574 616d726f 00332074
or more usefully in characters "SQLite format 3"

[themsley-voiceflex]

I'm reading through the issues backlog and I suspect that the following may be related to this
#2247
#2035
#1934
#1920
#1871 maybe
#1558
#1513 possible but less likely
#922
#892 maybe but looks unlikely
and of course #420

[themsley-voiceflex]

Oh and using sofia-sip-1.13.16-1.el9.x86_64 but problem is also present with 1.13.9 and 1.13.14 and maybe versions going back to 1.13.0 (I've gone 1.13.0 then .3 .6 .9 .13 .14 and now .16)

[andywolk]

Thank you for the details. We will double check.

[themsley-voiceflex]

Thanks. FYI I have attached the current fairly nasty patch I have been using to try to debug these issues. I've put a bodge in there for now that detects when the fd is pointing to a file that starts with / and avoided the SSL_write so the core.db overwrite is now bypassed but not fixed. That patch is very noisy but might help to understand the issues. I've also put in a few extra checks that avoid it hanging in SSL_read and SSL_write to try to make the service more usable.

I suspect the real issue is higher up the tree, perhaps in tport_type_ws.c as that's the guy that calls ws_init() with what should be a socket but I have only briefly looked at that code so it could be elsewhere.
sofia-sip-ws-ipaddr.patch.txt

[themsley-voiceflex]

I do have a 40MB log file from that debug code available if it would be useful? It contains ip addresses so it's not something I can share publicly.

[andywolk]

need to find out where file descriptors are closed by a mistake so then two different things use them after that. WS keeps using fd thinking the descriptor belongs to ws but it was actually opened by sqlite after close.

[themsley-voiceflex]

Yeah but it's more than the sqlite fd affected I think. I have 19,000 log entries that hit the debug message that says "readlink failed" vs 110k that write "WS ws_raw_write fd %d target %s\n" and are pointing to open fd's for sockets/files.

I did see a bunch of places where ws.c calls ws_close() directly and I can see that when it returns to tport_type_ws.c that then calls ws_destroy() which then calls ws_close() again. That looks a bit odd but not sure if related.

[themsley-voiceflex]

[root@fstrtc01 ~]# journalctl -u freeswitch -S  "2023-10-20 03:42:33" | grep '/' | grep -v ill
Oct 20 03:59:09 fstrtc01.voiceflex.com stdbuf[483339]: WS ws_close fd 112 target /opt/freeswitch/sounds/music/16000/bleep/DialtoneCallWaiting16.wav
Oct 20 04:29:11 fstrtc01.voiceflex.com stdbuf[483339]: WS ws_close fd 112 target /opt/freeswitch/sounds/music/32000/bleep/DialtoneCallWaiting32.wav
Oct 20 04:31:11 fstrtc01.voiceflex.com stdbuf[483339]: WS ws_close fd 112 target /opt/freeswitch/sounds/music/8000/bleep/DialtoneCallWaiting8.wav
Oct 20 04:37:11 fstrtc01.voiceflex.com stdbuf[483339]: WS ws_close fd 116 target /opt/freeswitch/sounds/music/8000/bleep/DialtoneCallWaiting8.wav
Oct 20 04:39:11 fstrtc01.voiceflex.com stdbuf[483339]: WS ws_close fd 118 target /opt/freeswitch/sounds/music/16000/bleep/DialtoneCallWaiting16.wav
Oct 20 04:47:12 fstrtc01.voiceflex.com stdbuf[483339]: WS ws_close fd 118 target /opt/freeswitch/sounds/music/16000/bleep/DialtoneCallWaiting16.wav
Oct 20 05:59:15 fstrtc01.voiceflex.com stdbuf[483339]: WS ws_close fd 142 target /opt/freeswitch/sounds/music/16000/bleep/DialtoneCallWaiting16.wav
Oct 20 08:08:07 fstrtc01.voiceflex.com stdbuf[483339]: WS ws_close fd 214 target /opt/freeswitch/sounds/music/16000/bleep/DialtoneCallWaiting16.wav
Oct 20 09:04:45 fstrtc01.voiceflex.com stdbuf[483339]: WS ws_close fd 305 target /opt/freeswitch/sounds/music/16000/bleep/DialtoneCallWaiting16.wav
Oct 20 09:04:45 fstrtc01.voiceflex.com stdbuf[483339]: WS ws_close fd 226 target /opt/freeswitch/sounds/music/32000/bleep/DialtoneCallWaiting32.wav
Oct 20 09:04:45 fstrtc01.voiceflex.com stdbuf[483339]: WS ws_close fd 171 target /opt/freeswitch/sounds/music/8000/bleep/DialtoneCallWaiting8.wav
Oct 20 09:04:45 fstrtc01.voiceflex.com stdbuf[483339]: WS ws_close fd 305 target /opt/freeswitch/sounds/music/16000/bleep/DialtoneCallWaiting16.wav
Oct 20 09:04:50 fstrtc01.voiceflex.com stdbuf[483339]: WS ws_close fd 304 target /opt/freeswitch/sounds/music/16000/bleep/DialtoneCallWaiting16.wav
Oct 20 09:18:41 fstrtc01.voiceflex.com stdbuf[483339]: WS ws_close fd 297 target /opt/freeswitch/sounds/music/32000/bleep/DialtoneCallWaiting32.wav
Oct 20 09:18:41 fstrtc01.voiceflex.com stdbuf[483339]: WS ws_close fd 206 target /opt/freeswitch/sounds/music/8000/bleep/DialtoneCallWaiting8.wav
Oct 20 09:18:47 fstrtc01.voiceflex.com stdbuf[483339]: WS ws_close fd 266 target /opt/freeswitch/sounds/music/16000/bleep/DialtoneCallWaiting16.wav
Oct 20 10:58:29 fstrtc01.voiceflex.com stdbuf[483339]: WS ws_close fd 141 target /opt/freeswitch/sounds/music/32000/bleep/DialtoneCallWaiting32.wav
Oct 20 11:15:53 fstrtc01.voiceflex.com stdbuf[483339]: WS ws_close fd 196 target /opt/freeswitch/sounds/music/16000/bleep/DialtoneCallWaiting16.wav
Oct 20 11:53:32 fstrtc01.voiceflex.com stdbuf[483339]: WS ws_close fd 155 target /opt/freeswitch/sounds/music/8000/bleep/DialtoneCallWaiting8.wav
Oct 20 12:13:10 fstrtc01.voiceflex.com stdbuf[483339]: WS ws_close fd 256 target /opt/freeswitch/sounds/music/32000/bleep/DialtoneCallWaiting32.wav
Oct 20 12:13:10 fstrtc01.voiceflex.com stdbuf[483339]: WS ws_close fd 250 target /opt/freeswitch/sounds/music/8000/bleep/DialtoneCallWaiting8.wav
Oct 20 12:41:14 fstrtc01.voiceflex.com stdbuf[483339]: WS ws_close fd 257 target /opt/freeswitch/sounds/music/32000/bleep/DialtoneCallWaiting32.wav
Oct 20 12:41:14 fstrtc01.voiceflex.com stdbuf[483339]: WS ws_close fd 271 target /opt/freeswitch/sounds/music/8000/bleep/DialtoneCallWaiting8.wav
Oct 20 14:43:49 fstrtc01.voiceflex.com stdbuf[483339]: WS ws_close fd 212 target /opt/freeswitch/sounds/music/16000/bleep/DialtoneCallWaiting16.wav
Oct 20 15:11:19 fstrtc01.voiceflex.com stdbuf[483339]: WS ws_close fd 146 target /opt/freeswitch/sounds/music/8000/bleep/DialtoneCallWaiting8.wav
Oct 20 15:11:19 fstrtc01.voiceflex.com stdbuf[483339]: WS ws_close fd 231 target /opt/freeswitch/sounds/music/16000/bleep/DialtoneCallWaiting16.wav
Oct 20 15:28:38 fstrtc01.voiceflex.com stdbuf[483339]: WS ws_close fd 179 target /opt/freeswitch/sounds/music/8000/bleep/DialtoneCallWaiting8.wav
Oct 20 15:31:30 fstrtc01.voiceflex.com stdbuf[483339]: WS ws_close fd 215 target /opt/freeswitch/sounds/music/8000/bleep/DialtoneCallWaiting8.wav
Oct 20 15:31:30 fstrtc01.voiceflex.com stdbuf[483339]: WS ws_close fd 167 target /opt/freeswitch/sounds/music/8000/bleep/DialtoneCallWaiting8.wav
Oct 20 15:57:33 fstrtc01.voiceflex.com stdbuf[483339]: WS ws_close fd 222 target /opt/freeswitch/sounds/music/8000/bleep/DialtoneCallWaiting8.wav

[themsley-voiceflex]

If ever there was a reason to make sure you never run freeswitch as root...
Oct 21 14:13:47 fstrtc01.voiceflex.com stdbuf[517990]: WS ws_close fd 118 target /etc/passwd

[seven1240]

I use in memory db in FreeSWITCH and I never hit a problem like this. I'm not using wss btw.

The issue might be caused by ssl close like this one:

freeswitch/sofia-sip#212

some one said switch back to a earlier version might fix this problem, see: freeswitch/sofia-sip#212 (comment)

[themsley-voiceflex]

This issue is due to WSS so if you do not run it, you will not be hit by it.