rendervulkan: Fix getting Vulkan DRM formats if backend doesn't use modifiers

[laurirasanen]

After commit 0110109 running gamescope nested SDL backend on top of x11 plasma causes assertion in wlroots. Related to issue #1218.

[gamescope] [Info]  console: gamescope version 3.15.9-10-g6497b0f (gcc 14.2.1)
No CAP_SYS_NICE, falling back to regular-priority compute and threads.
Performance will be affected.
[gamescope] [Info]  vulkan: selecting physical device 'AMD Radeon RX 6800 XT (RADV NAVI21)': queue family 1 (general queue family 0)
[gamescope] [Info]  vulkan: physical device supports DRM format modifiers
[gamescope] [Info]  wlserver: [backend/headless/backend.c:67] Creating headless backend
[gamescope] [Info]  vulkan: supported DRM formats for sampling usage:
[gamescope] [Info]  vulkan: Creating Gamescope nested swapchain with format 44 and colorspace 0
gamescope: types/wlr_linux_dmabuf_v1.c:532: feedback_compile: Assertion `table_len > 0' failed.
zsh: IOT instruction (core dumped)  ./build/src/gamescope -- glxgears

(gdb) bt
#0  0x00007ffff73b63f4 in ?? () from /usr/lib/libc.so.6
#1  0x00007ffff735d120 in raise () from /usr/lib/libc.so.6
#2  0x00007ffff73444c3 in abort () from /usr/lib/libc.so.6
#3  0x00007ffff73443df in ?? () from /usr/lib/libc.so.6
#4  0x00007ffff7355177 in __assert_fail () from /usr/lib/libc.so.6
#5  0x00005555557f0472 in feedback_compile (feedback=0x7fffffffda20) at ../subprojects/wlroots/types/wlr_linux_dmabuf_v1.c:532
#6  0x00005555557f131d in set_default_feedback (linux_dmabuf=0x555555b19770, feedback=0x7fffffffda20)
    at ../subprojects/wlroots/types/wlr_linux_dmabuf_v1.c:888
#7  0x00005555557f170e in wlr_linux_dmabuf_v1_create (display=0x7fffe44f2eb0, version=4, default_feedback=0x7fffffffda20)
    at ../subprojects/wlroots/types/wlr_linux_dmabuf_v1.c:971
#8  0x00005555557f1806 in wlr_linux_dmabuf_v1_create_with_renderer (display=0x7fffe44f2eb0, version=4, renderer=0x555555b19970)
    at ../subprojects/wlroots/types/wlr_linux_dmabuf_v1.c:1002
#9  0x00005555557c656a in wlr_renderer_init_wl_display (r=0x555555b19970, wl_display=0x7fffe44f2eb0) at ../subprojects/wlroots/render/wlr_renderer.c:87
#10 0x000055555563c225 in wlserver_init () at ../src/wlserver.cpp:1738
#11 0x00005555556344f5 in main (argc=1, argv=0x7fffffffdd28) at ../src/main.cpp:934

This PR returns the vulkan DRM format handling to what it was previously. i.e. they get added even if backend doesn't use modifiers.

[gamescope] [Info]  console: gamescope version 3.15.9-10-g6497b0f (gcc 14.2.1)
No CAP_SYS_NICE, falling back to regular-priority compute and threads.
Performance will be affected.
[gamescope] [Info]  vulkan: selecting physical device 'AMD Radeon RX 6800 XT (RADV NAVI21)': queue family 1 (general queue family 0)
[gamescope] [Info]  vulkan: physical device supports DRM format modifiers
[gamescope] [Info]  wlserver: [backend/headless/backend.c:67] Creating headless backend
[gamescope] [Info]  vulkan: supported DRM formats for sampling usage:
[gamescope] [Info]  vulkan:   AR24 (0x34325241)
[gamescope] [Info]  vulkan:   XR24 (0x34325258)
[gamescope] [Info]  vulkan:   AB24 (0x34324241)
[gamescope] [Info]  vulkan:   XB24 (0x34324258)
[gamescope] [Info]  vulkan:   RG16 (0x36314752)
[gamescope] [Info]  vulkan:   NV12 (0x3231564E)
[gamescope] [Info]  vulkan:   AB4H (0x48344241)
[gamescope] [Info]  vulkan:   XB4H (0x48344258)
[gamescope] [Info]  vulkan:   AB48 (0x38344241)
[gamescope] [Info]  vulkan:   XB48 (0x38344258)
[gamescope] [Info]  vulkan:   AB30 (0x30334241)
[gamescope] [Info]  vulkan:   XB30 (0x30334258)
[gamescope] [Info]  vulkan:   AR30 (0x30335241)
[gamescope] [Info]  vulkan:   XR30 (0x30335258)
[gamescope] [Info]  vulkan: Creating Gamescope nested swapchain with format 44 and colorspace 0
[gamescope] [Info]  wlserver: Using explicit sync when available
[gamescope] [Info]  wlserver: Running compositor on wayland display 'gamescope-0'
[gamescope] [Info]  wlserver: [backend/headless/backend.c:17] Starting headless backend
[gamescope] [Info]  wlserver: Successfully initialized libei for input emulation!
[gamescope] [Error] wlserver: [xwayland/sockets.c:64] Failed to bind socket @/tmp/.X11-unix/X0: Address already in use
[gamescope] [Info]  wlserver: [xwayland/server.c:107] Starting Xwayland on :1
[gamescope] [Info]  pipewire: stream state changed: connecting
[gamescope] [Info]  pipewire: stream state changed: paused
[gamescope] [Info]  pipewire: stream available on node ID: 104
[gamescope] [Info]  vblank: Using timerfd.
[gamescope] [Info]  vulkan: Creating Gamescope nested swapchain with format 44 and colorspace 0
[gamescope] [Warn]  xwm: got the same buffer committed twice, ignoring.
The XKEYBOARD keymap compiler (xkbcomp) reports:
> Warning:          Unsupported maximum keycode 708, clipping.
>                   X11 cannot support keycodes above 255.
> Warning:          Could not resolve keysym XF86KbdInputAssistPrevgrou
> Warning:          Could not resolve keysym XF86KbdInputAssistNextgrou
Errors from xkbcomp are not fatal to the X server
Running synchronized to the vertical refresh.  The framerate should be
approximately the same as the monitor refresh rate.
828 frames in 5.0 seconds = 165.470 FPS

[HanabishiRecca]

This patch does not work for me on Polaris.

Instead, the one proposed in #1218 (comment)

--- a/src/rendervulkan.cpp
+++ b/src/rendervulkan.cpp
@@ -2799,7 +2799,7 @@ bool vulkan_init_format(VkFormat format, uint32_t drmFormat)
 	}
 	else
 	{
-		if ( !GetBackend()->SupportsFormat( drmFormat ) )
+		if ( GetBackend()->UsesModifiers() && !GetBackend()->SupportsFormat( drmFormat ) )
 			return false;
 
 		wlr_drm_format_set_add( &sampledDRMFormats, drmFormat, DRM_FORMAT_MOD_INVALID );

does work.

[laurirasanen]

@HanabishiRecca That path only gets hit if !g_device.supportsModifiers(). I didn't have an old enough GPU to test.

[HanabishiRecca]

Other reporters also confirm that: #1218 (comment).
So I don't see a problem combining both patches here.

[laurirasanen]

Updated to include that check too.

[HanabishiRecca]

Thank you.