Qt6: recent project paths are stored in QGIS.ini in UTF-8 (instead of hex-escaped Unicode), incompatible with Qt5 version

[justinbb]

What is the bug or the crash?

Using a Mac Qt6 build (thanks @m-kuhn!)
– the QGIS.ini file from regular (Qt5) releases is interpreted correctly
– but when it is written back out, recent project paths are encoded in UTF-8
The UTF-8 encoded QGIS.ini cannot be read correctly by Qt5 QGIS, which tries to interpret each octet of UTF-8 as if it were a separate Unicode character.
The UTF-8 QGIS.ini is (fortunately) interpreted correctly by Qt6 QGIS. Switching back and forth between Qt5 and Qt6 builds, however, leads to increasing problems.

Steps to reproduce the issue

1. In a Qt5 QGIS, create a project and save it under a name containing a non-ASCII character, e.g. • (U+2022). Quit QGIS
2. Examine QGIS.ini with a text editor. In the recentProjects section, find the newly saved project: its name will contain \x2022
3. Now open Qt6 QGIS. Note the (correct) filename in the recent projects lists. Quit QGIS
4. Examine QGIS.ini with a text editor. The project's name will have been rewritten in UTF-8, so it will display normally if the text editor is correctly interpreting UTF-8 text. The hex codes for the • character will be E2 80 A2.
5. Open Qt5 QGIS. In the recent files list, the file is greyed out and its name is displayed incorrectly with three characters in place of the •, the Unicode code points U+00E2 U+0080 U+00A2. Quit QGIS
6. Examine QGIS.ini with a text editor again, there will now be \xe2\x80\xa2, the misinterpretation is now hex-escaped and getting difficult to recover from.

Versions

QGIS version | 3.41.0-Master -- | -- QGIS code revision | 9f6d7bd   Libraries Qt version | 6.8.1 Python version | 3.11.10 GDAL/OGR version | 3.9.3 PROJ version | 9.5.1 EPSG Registry database version | v11.022 (2024-11-05) GEOS version | 3.13.0-CAPI-1.19.0 SQLite version | 3.47.2 PDAL version | 2.8.1 PostgreSQL client version | unknown SpatiaLite version | 5.1.0 QWT version | 6.3.0 QScintilla2 version | 2.14.1 OS version | macOS Sonoma (14.7)   Active Python plugins db_manager | 0.1.20 MetaSearch | 0.3.6 processing | 2.12.99

QGIS version
3.41.0-Master
QGIS code revision
9f6d7bd6

Libraries
Qt version
6.8.1
Python version
3.11.10
GDAL/OGR version
3.9.3
PROJ version
9.5.1
EPSG Registry database version
v11.022 (2024-11-05)
GEOS version
3.13.0-CAPI-1.19.0
SQLite version
3.47.2
PDAL version
2.8.1
PostgreSQL client version
unknown
SpatiaLite version
5.1.0
QWT version
6.3.0
QScintilla2 version
2.14.1
OS version
macOS Sonoma (14.7)

Active Python plugins
db_manager
0.1.20
MetaSearch
0.3.6
processing
2.12.99

Supported QGIS version

• I'm running a supported QGIS version according to the roadmap.

New profile

• I tried with a new QGIS profile

Additional context

Although this is not an issue if one sticks to Qt5 or Qt6, it makes switching back and forth difficult (unless one limits filenames to ASCII characters). Some sort of workaround would be helpful to avoid problems when the transition arrives.

[nicogodet]

Issue valid for windows too

[nicogodet]

Same for news feed

[nirvn]

https://forum.qt.io/topic/157293/qt-6-way-of-saving-a-unicode-string-to-qsettings-is-this-a-bug-or-a-feature/2

[justinbb]

Thank you for the extra information confirming that Qt created the situation. A few years ago, in one of the final Qt5 releases, they should have made Qt5 capable of reading the new Qt6 format – it would have been easy to do and effective. Now it's up to applications such as QGIS to work around their dropped ball.

The Qt6 choice of using UTF-8 for the ini file is sound, as well as being the (already established) way of the future. The most appropriate fix would therefore be to add code in QGIS Qt5 builds to handle UTF-8 correctly if it is present in the ini file. (There is no ambiguity, as the Qt5 format uses only USASCII characters to do its custom hex encoding of Unicode.) It would be a wrapper around whatever Qt call is used to read the ini file. No need to do anything when writing the ini file, since the Qt5 format is read correctly by Qt6.

[justinbb]

The job can probably be done (Qt5 builds only) with QSettings::setIniCodec("UTF-8"), called immediately after constructing a QSettings object. To be determined: can this call be used all the time (if it still allows hex-escaped Unicode to be read), or does one have to try to figure out which format is present and set the UTF-8 iniCodec only if necessary?
See

• https://doc.qt.io/qt-5/qsettings.html#setIniCodec-1
• https://doc.qt.io/qt-6/qsettings.html#Format-enum [scroll down to "Compatibility with older Qt versions"]

[justinbb]

Note that it will also be necessary to insert QSettings::setIniCodec(nullptr) after reading / before any attempt is made to write the ini file, so that Qt5 builds continue to write the ini file in the old format.

[m-kuhn]

We could also rename the ini file for qt6 builds (QGIS4.ini or QGIS-qt6.ini)

[justinbb]

We could also rename the ini file for qt6 builds (QGIS4.ini or QGIS-qt6.ini)

QGIS4.ini would be more user-friendly, I think. This solution may be simpler (and acceptable to users as long as the old ini file is migrated).

Advantages of a pair of calls to QSettings::setIniCodec() as I described above, if that does the job:

• elegant: restricted to Qt5 builds and therefore would disappear in the long term
• no change in the ini filename

Disadvantages:

• the problem would be solved only for people alternating between Qt6 and late Qt5 (i.e., after the patch); anyone opening a Qt6 ini with a current or older Qt5 build would provoke the issue
• would need to be tried out (and I am not really in a position to do the pull request myself)