Linux: (Optionally) Enforce XCB backend if QT_QPA_PLATFORM is not set

Especially on older distributions, Qt/Wayland works worse than Qt/
XCB(XWayland). On Ubuntu 20.04, starting Qt Creator with Wayland can
even crash the Wayland session
(https://bugs.launchpad.net/ubuntu/+source/mutter/+bug/1958684/).

We still want to ship the plugins, and allow forcing the usage of
Wayland, since on the other hand Wayland works better in HiDPI
environments for newer distributions.

Behavior is enabled when configuring with  -DQTC_FORCE_XCB=ON

Task-number: QTCREATORBUG-26867
Change-Id: I623f001412f809381208b6cb3ec5b3fcfdb86011
Reviewed-by: <github-actions-qt-creator@cristianadam.eu>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: hjk <hjk@qt.io>
Reviewed-by: Robert Löhning <robert.loehning@qt.io>

scripts/build.py

@@ -173,6 +173,7 @@ def build_qtcreator(args, paths):
                   '-DBUILD_QBS=' + cmake_option(build_qbs),
                   '-DBUILD_DEVELOPER_DOCS=' + cmake_option(not args.no_docs),
                   '-DBUILD_EXECUTABLE_SDKTOOL=OFF',
+                  '-DQTC_FORCE_XCB=ON',
                   '-DCMAKE_INSTALL_PREFIX=' + common.to_posix_path(paths.install),
                   '-DWITH_TESTS=' + cmake_option(args.with_tests)]
     cmake_args += common_cmake_arguments(args)

src/app/CMakeLists.txt

@@ -92,6 +92,13 @@ if (APPLE)
   )
 endif()
 
+option(QTC_FORCE_XCB "Enables that Qt Creator forces XCB on Linux if QT_QPA_PLATFORM is not set." OFF)
+extend_qtc_executable(qtcreator
+  CONDITION QTC_FORCE_XCB
+  DEFINES
+    QTC_FORCE_XCB
+)
+
 extend_qtc_executable(qtcreator
   CONDITION BUILD_WITH_CRASHPAD
   DEFINES

src/app/main.cpp

@@ -500,6 +500,26 @@ int main(int argc, char **argv)
         qputenv("QT_ENABLE_REGEXP_JIT", "0");
     }
 
+#if defined(QTC_FORCE_XCB)
+    if (Utils::HostOsInfo::isLinuxHost() && !qEnvironmentVariableIsSet("QT_QPA_PLATFORM")) {
+        // Enforce XCB on Linux/Gnome, if the user didn't override via QT_QPA_PLATFORM
+        // This was previously done in Qt, but removed in Qt 6.3. We found that bad things can still happen,
+        // like the Wayland session simply crashing when starting Qt Creator.
+        // TODO: Reconsider when Qt/Wayland is reliably working on the supported distributions
+        const bool hasWaylandDisplay = qEnvironmentVariableIsSet("WAYLAND_DISPLAY");
+        const bool isWaylandSessionType = qgetenv("XDG_SESSION_TYPE") == "wayland";
+        const QByteArray currentDesktop = qgetenv("XDG_CURRENT_DESKTOP").toLower();
+        const QByteArray sessionDesktop = qgetenv("XDG_SESSION_DESKTOP").toLower();
+        const bool isGnome = currentDesktop.contains("gnome") || sessionDesktop.contains("gnome");
+        const bool isWayland = hasWaylandDisplay || isWaylandSessionType;
+        if (isGnome && isWayland) {
+            qInfo() << "Warning: Ignoring WAYLAND_DISPLAY on Gnome."
+                    << "Use QT_QPA_PLATFORM=wayland to run on Wayland anyway.";
+            qputenv("QT_QPA_PLATFORM", "xcb");
+        }
+    }
+#endif
+
     Utils::TemporaryDirectory::setMasterTemporaryDirectory(QDir::tempPath() + "/" + Core::Constants::IDE_CASED_ID + "-XXXXXX");
 
 #ifdef Q_OS_MACOS