Vulkan driver fails to start in a SteamDeck (and Linux under Wine in general)

[10-p]

Unreal Tournament using the Vulkan driver fails to start and only gives this error in the logs (I think is the following):
Log: Could not create vulkan renderer: Could not allocate descriptor sets: out of pool memory

This is the full Unreal Tournament log.
UnrealTournament.log

In the Steam Deck, I'm starting UT through Steam in desktop mode.

I've also tried running the game in my desktop using straight Wine with a RX6900. Same error.

I've only tried AMD cards, but maybe some incompatibility with mesa?

[dpjudas]

Unfortunately I'm unable to reproduce this issue on my computer, even with the vulkan validation layers enabled. I don't own a Steam Deck, so I can't test it on that specific hardware.

If you're not afraid of dev tools, you can try install the Vulkan SDK on the system and then enable VkDebug in UnrealTournament.ini. Hopefully it will write some more detailed messages into the log that I can use to trace down what the issue could be.

[10-p]

It never run so I had to add the VulkanDrv.VulkanRenderDevice section to the ini and set the VkDebug to true.

Running it with the VulkanSDK get no additional logs. Maybe I'm running it wrong?

I've tried enabling as many debugging options in the configurator but nothing else appears in the configurator logs either.

On a side note I've also tried the D3D11 renderer but it probably requires proton with dxvk as it fails with;

Log: Bound to D3D11Drv.dll
Log: Could not create d3d11 renderer: Could not compile shader 'shaders/Scene.vert':shaders/Scene.vert:23:7: E5013: Invalid semantic 'SV_ClipDistance'.
Log: 3D hardware initialization failed

[dpjudas]

You don't need to do anything in that Vulkan Configurator. VulkanDrv will use the validation layer if you set VkDebug to True in UnrealTournament.ini and then any validation errors will be written into UnrealTournament.log. Look for any lines containing the word error.

About D3D11Drv, it would appear that wine does not support SV_ClipDistance. You may be able to workaround that if you bring a D3DCompiler_47.dll from a Windows machine and place it next to D3D11Drv.dll.

[Barktooth]

I get the exact same error as well, Log: Could not create vulkan renderer: Could not allocate descriptor sets: out of pool memory

I installed the Vulkan SDK but even with that and VkDebug=True in the .ini, I don't get any additional errors in UnrealTournament.log. If I launch the game through the Vulkan Configurator it does produce its own log file (UnrealTournament.txt) but I'm not sure if any of it is relevant to UT.

I tried some of the older releases and it seems this problem appeared in 2.12, that version still gives the same error. Meanwhile, 2.11 starts up fine but has some artifacting:

Artifacting appears to have been introduced in 2.6, as 2.5 does not have it (and overall seems to work great from my brief testing, except for the lack of gamma setting). Here is also the log file from v.2.11 with VkDebug=True, as it seems to output more information than 2.20:
UnrealTournamentV2.11.log

I, too, am using an AMD card by the way, the Radeon R9 380.

[dpjudas]

I'm afraid I have no way of fixing this. I don't own a Steam Deck.

The error appears to be caused by the Windows emulation layer on the Steam Deck (wine/mesa). The Vulkan validation layer is saying everything is completely valid. Either there's a bug in wine/mesa or the validation layer isn't actually installed inside wine. It may be that installing the validation layer on the Deck only installed it for Linux, but not the wine emulation environment. You'd have to ask someone with knowledge about the Deck as I don't know the details of this platform.

If you have access to a Windows machine, I suggest you try use the D3D11 or D3D12 render devices instead. Wine generally supports Direct3D better than Vulkan. However, due to wine not having implemented SV_ClipDistance, you need to copy the Windows version of D3DCompiler_43.dll into the Unreal Tournament System folder.

[Barktooth]

The error appears to be caused by the Windows emulation layer on the Steam Deck (wine/mesa).

Oh, sorry for the misunderstanding, but I'm actually using Windows 10 on a regular desktop machine, not a Steam Deck, and running UT natively, with no Linux or WINE trickery. I kind of forgot this topic was for the Steam Deck specifically, I probably should have opened a separate one. I just saw that the error was the same one that I was getting so my brain kind of dismissed the other details; apologies for the confusion.

Either way, Vulkan release 2.5 seems to work well enough for me so it's okay if you can't fix any of this. It is still probably the best renderer in terms of game feel for me, the mouse is more snappy with it compared to D3D9 and D3D11. D3D12 might be able to match it if I could get it to turn off desktop compositing properly (by which I mean use "Flip fullscreen" or whatever the official term for that is, which AFAIK makes the compositor sort of take a back seat) when going into fullscreen, but for some reason it refuses to, even if I Alt-Tab or Alt-Enter a few times. Or maybe it's V-Sync and not compositing, because my FPS is locked to refresh rate in fullscreen, but changing the V-Sync setting in the .ini makes no difference to this behavior so it's rather strange.

[dpjudas]

Oh, I see. Yes that makes it quite interesting since then clearly the entire Steam Deck wine/mono thing is out of the picture.

I took a closer look at the UnrealTournament.txt log file from the vulkan validation layer and I'm seeing two interesting things. First is that it does not detect any descriptor set allocation error. According to the validation layer everything is fine.

The second thing is that there are several implicit layers active while the game runs: SteamOverlayVulkanLayer, SteamFossilizeVulkanLayer, EOSOverlayVkLayer and obs-vulkan. Those layers come from Steam, Epic Online Store and OBS. What they do is that they inject their own layers on top of the output of the Vulkan application. I.e. for Steam its that overlay you can see with your friends and so on. There have been instances in the past where such layers have had bugs in them that triggered vulkan crashes. It is a possibility this is what is going on here. Could also be completely unrelated, of course.