The same semaphore is used for consecutive frames by RecordAndSubmitTask

[AnyOldName3]

vsg::RecordAndSubmitTask doesn't have separate semaphores per frame/swapchain image like vsg::Window does for image acquisition, so every frame's queued submissions signal the same semaphore, and every frame's presentation waits for the same semaphore.

https://docs.vulkan.org/guide/latest/swapchain_semaphore_reuse.html has a writeup on this being a bad thing, and with a new enough version of the Vulkan SDK (i.e. 1.4.313 or later), this will trigger a validation error in any VSG-based application:

Validation Error: [ VUID-vkQueueSubmit-pSignalSemaphores-00067 ] | MessageID = 0x539277af
vkQueueSubmit(): pSubmits[0].pSignalSemaphores[0] (VkSemaphore 0x30000000003) is being signaled by VkQueue 0x1b6621abd80, but it may still be in use by VkSwapchainKHR 0x220000000022.
Here are the most recently acquired image indices: 0, [1], 2.
(brackets mark the last use of VkSemaphore 0x30000000003 in a presentation operation)
Swapchain image 1 was presented but was not re-acquired, so VkSemaphore 0x30000000003 may still be in use and cannot be safely reused with image index 2.
Vulkan insight: One solution is to assign each image its own semaphore. Here are some common methods to ensure that a semaphore passed to vkQueuePresentKHR is not in use and can be safely reused:
        a) Use a separate semaphore per swapchain image. Index these semaphores using the index of the acquired image.
        b) Consider the VK_KHR_swapchain_maintenance1 extension. It allows using a VkFence with the presentation operation.
The Vulkan spec states: Each binary semaphore element of the pSignalSemaphores member of any element of pSubmits must be unsignaled when the semaphore signal operation it defines is executed on the device (https://vulkan.lunarg.com/doc/view/1.4.321.1/windows/antora/spec/latest/chapters/cmdbuffers.html#VUID-vkQueueSubmit-pSignalSemaphores-00067)
Objects: 2
    [0] VkSemaphore 0x30000000003
    [1] VkQueue 0x1b6621abd80

As far as I've been able to tell, this is the cause for the much lower performance with AMD graphics on Windows. As we submit several frames and reuse the same semaphore several times before first waiting on a fence, it's ambiguous which frame's present should wait for which frame's rendering, and we can end up with the first frame's present waiting for the third frame's rendering. For whatever reason, this has ended up being symptomatic with this one driver and hardware combination, but not with others anyone's tested.

I'll take a look at potential fixes for this over the next few days. As the validation error suggests, just using one rendering semaphore per swapchain image should be pretty straightforward, although we could simplify all the synchronisation in the VSG by switching from fences and binary semaphores to timeline semaphores (using LunarG's timeline semaphore emulation layer implemented on top of fences and binary semaphores when running on systems that don't support Vulkan 1.2).

[timoore]

I'll take a look at potential fixes for this over the next few days. As the validation error suggests, just using one rendering semaphore per swapchain image should be pretty straightforward, although we could simplify all the synchronisation in the VSG by switching from fences and binary semaphores to timeline semaphores (using LunarG's timeline semaphore emulation layer implemented on top of fences and binary semaphores when running on systems that don't support Vulkan 1.2).

I'm all for using timeline semaphores, and I think their use would simplify multi-queue synchronization. As @AnyOldName3 mentions, there is a solution for systems that don't support them, mostly Android. I don't know if this is a gateway to discussing the utility of features in Vulkan 1.2 and beyond.

[AnyOldName3]

Other uses of semaphores get registered with a fence, but not as reliably as command buffers do (i.e. lots of places are missed, and the _numDependentSubmissions value isn't incremented even when it's done), and it's never checked (i.e. _numDependentSubmissions isn't read). If that were made more sound, then we could have checks that _numDependentSubmissions was zero when semaphores were used and therefore that a fence had told us that the GPU was finished with any previous usage of the same semaphore. I'd be unsurprised if that found further problems.

This would be a bit of a nuisance at the moment, though. We don't always have access to the relevant fence (if there even is one) when we're using semaphores. While it would be pretty doable to ensure that queue submission associates the wait semaphores with the signal fence, presentation (without VK_KHR_swapchain_maintenance1) doesn't signal a fence, and you're expected to infer that the semaphores are available once you successfully acquire the same swapchain image again.

It's pretty annoying that they didn't add this validation error right from the initial release, as it's not the clearest part of the specification, so I don't imagine that the VSG's the only thing affected unless everyone's decided it was too hard to think about and just copied vulkan-tutorial.com and/or the Vulkan samples exactly.

[AnyOldName3]

With a preliminary fix, I'm no longer seeing the validation error and the framerate's at least five times higher.

[robertosfield]

I believe there are some users with embedded hardware that is stuck on Vulkan 1.1. It will be worth having a wider discussion about minimum versions.

[AnyOldName3]

VK_LAYER_KHRONOS_timeline_semaphore is the layer that emulates timeline semaphores without requiring Vulkan 1.2, and as far as I can tell, it's a really simple C implementation, so should be portable to nearly any platform.

Either way, the immediate priority is fixing the validation error (and thereby fixing the performance problem), which only requires splitting up the rendering-finished semaphores by swapchain image. While that doesn't let us strip out all the existing fence and binary semaphore usage and replace it with something simpler, the size of the patch and therefore the amount we need to think about right now is much smaller, so that's what I've been experimenting with today.

[AnyOldName3]

Further testing has revealed that when using a slightly newer version of AMD's Windows driver, the higher performance is happening without the fix, too. It seems that something has changed about the timing, and that's allowing us to get away with the semaphore dependencies as they were intended instead of reliably hitting the worst-case interpretation. I've confirmed this with Radeon Performance Profiler, too.

This isn't to say we don't need the fix - plenty of people don't run bleeding-edge drivers, and as the validation error exists, it's legitimate for drivers to respond to what the VSG currently does by being really slow or crashing. It's just harder to produce an impressive before-and-after comparison.

That said, as I have RGP profiles from the older driver, I can still show something:

Before

Presents depend on rendering from 1-3 frames ago

After

And now they don't

[AnyOldName3]

I've opened #1559 with a fix so it can get some feedback.

[reedev]

I believe there are some users with embedded hardware that is stuck on Vulkan 1.1. It will be worth having a wider discussion about minimum versions.

Yes, indeed, I am using the VSG on Jetson Nano a lot. I depend on it actually.

I am checking things on the Jetson and this is some info from vulkaninfo:
Vulkan Instance Version: 1.2.70
The layers says 'Vulkan version 1.1.70'

I am still a bit confused how I should interpret these, maybe one of you can shine some light on it?

Thanks

[robertosfield]

To help with testing I've merged @AnyOldName3's changes as a AnyOldName3-fix-semaphore-reuse branch:

https://github.com/vsg-dev/VulkanSceneGraph/tree/AnyOldName3-fix-semaphore-reuse

@reedev et. al. could you test this branch?

[AnyOldName3]

As I said, I've not done the switch to timeline semaphores in that branch, I've gone for a minimal fix that still relies on binary semaphores and fences. Switching to timeline semaphores would be a separate task as although we'd be replacing complicated code with simpler code, it would mean replacing all the synchronisation in the VSG (except things done with pipeline barriers), so would need much more thorough testing.

[AnyOldName3]

I don't understand how, but today, even with the fixed branch that isn't triggering the validation error, the performance is back to being terrible and the Radeon profiler is back to showing the unwanted cross-frame semaphore dependency.

[AnyOldName3]

There doesn't appear to be an ambiguous or incorrect sequence if I log the semaphores. While my fix does eliminate a problem, it doesn't seem to be the same one as I was investigating, even though the theoretical symptoms were the same.