Support output to HDR monitors #94496
Conversation
|
I gave this a quick test locally (on Windows 11 23H2 + NVIDIA 560.80 + LG C2 42"), it works as expected. This is encouraging to see, I've been wanting this for a while 🙂 I'll need to look into building more extensive scenes and getting tonemapped screenshots/videos out of this. 2D HDR also needs to be tested thoroughly. Remember that JPEG XL or AVIF for images and AV1 for videos are a must for HDR, as other formats can only store SDR data. You may need to embed those in ZIP archives and ask users to preview them in a local media player, as GitHub doesn't allow uploading those formats and browsers often struggle displaying HDR correctly. I noticed some issues for now:
See the settings exposed by the Control HDR mod for an example of a best-in-class HDR implementation (related video): control_hdr_mod_settings.mp4Interesting, that UI seems to use the term "paperwhite" in a different way, and has a dedicated setting for the brightness of UI and HUD elements. |
| Sets the maximum luminance of the display in nits (cd/m²) when HDR is enabled. | ||
| This is used to scale the HDR effect to avoid clipping. |
There was a problem hiding this comment.
| Sets the maximum luminance of the display in nits (cd/m²) when HDR is enabled. | |
| This is used to scale the HDR effect to avoid clipping. | |
| Sets the maximum luminance of the display in nits (cd/m²) when HDR is enabled. If set to [code]0.0[/code], luminance is not limited, which may look worse than setting a max luminance value suited to the display currently in use. | |
| This is used to scale the HDR effect to avoid clipping. |
There was a problem hiding this comment.
I'm not so sure about this change, the max value allowed by the spec for ST2084 is 10,000 nits, which always looks blown out on any consumer display (and most of the professional ones too). Perhaps a more reasonable default value would make more sense?
There was a problem hiding this comment.
Indeed, in real world scenarios, you'll always want luminance to be limited to a reasonable value. That said, as I understand the code, no limitation is applied if the luminance cap is set to 0 nits (the project setting defaults to that value).
That reminds me, should the default value for the HDR luminance cap be changed? The demo project uses 600 nits. We should probably see what modern games typically use as their default luminance cap value and use a value similar to that.
| Only available on platforms that support HDR output, have HDR enabled in the system settings, and have a compatible display connected. | ||
| </member> | ||
| <member name="display/window/hdr/max_luminance" type="float" setter="" getter="" default="0.0"> | ||
| Sets the maximum luminance of the display in nits (cd/m²) when HDR is enabled. |
There was a problem hiding this comment.
| Sets the maximum luminance of the display in nits (cd/m²) when HDR is enabled. | |
| Sets the maximum luminance of the display in nits (cd/m²) when HDR is enabled. If set to [code]0.0[/code], luminance is not limited, which may look worse than setting a max luminance value suited to the display currently in use. |
There was a problem hiding this comment.
Also not so sure about this, for the reasons in the other related comment.
DarkKilauea
force-pushed
the
rendering/hdr-output
branch
2 times, most recently
from
88beb60
to
8df131d
Compare
Thanks for taking a look!
Odd that NVidia's RTX HDR doesn't detect the HDR color space and avoid messing with the final swap chain buffer. Auto-HDR in Windows 11 appears to avoid messing with Godot when HDR is enabled. Updating the NVidia Profile may be outside the scope of this PR and be best done with a more focused PR.
For the initial draft, yes, everything is mapped using the same tonemapper. However, we should map UI elements to a different brightness to avoid them being too bright. For now, that can be worked around with dimming the brightness of any UI elements via the theme, but I would like to fix that in this PR.
I haven't looked into configuring the editor to use HDR yet. Will do after I figure out how to properly tone map UI elements, if you enable HDR on the editor now, the UI is a little unpleasant.
Agreed, UI elements and other 2D elements should probably be mapped to a different brightness curve. I'll probably have to figure out where in the engine 3D and 2D elements are composited together and perform the tone mapping there.
That might be outside of the scope of this PR. I'm not sure how I would indicate that certain 3D elements need to be mapped using a different brightness curve once they are all combined into the same buffer. It would be similar to trying to avoid sRGB mapping certain rendered elements. For now, this can be worked around by decreasing the brightness of the color of these elements.
Baldur's Gate 3 and Cyberpunk 2077 also have really nice HDR settings menus. I've been basing some of this work off their approach, though modifying contrast and brightness I'm leaving up to Environment since those effects are already there. Thanks again for your comments! I'll add some TODO items to the description for tracking. |
DarkKilauea
force-pushed
the
rendering/hdr-output
branch
3 times, most recently
from
b89985a
to
e9742ba
Compare
DarkKilauea
force-pushed
the
rendering/hdr-output
branch
from
e9742ba
to
b2bd1a1
Compare
|
Can you use any Godot project to test this PR? Bistro-Demo-Tweaked and Crater-Province-Level both use physical light units, and use as close to reference values for luminosity on light sources. (i.e. the sun at noon is 100000 lux, the moon at midnight is 0.3 lux) I'd love to help test this PR but unfortunately I don't have HDR hardware |
|
I recently got a monitor that supports Anyway, adding HDR output to D3D12 should be trivial and I might give it a try. (No promises!) Shall we also consider implementing HDR display for the compatibility renderer? I am not sure if native OpenGL can do HDR, but it is very possible to implement on Windows with the help of ANGLE and some manual setting up. |
|
This needs a rebase on master, but I have a https://www.dell.com/en-ca/shop/alienware-34-curved-qd-oled-gaming-monitor-aw3423dw/apd/210-bcye/monitors-monitor-accessories HDR display. I can help test. |
You should be able to test with any scene, though keep in mind that the realistic light units will not map directly to the brightness of the display. Consumer desktop displays typically don't go much above 1000 nits on the high end, which is far too dim to simulate sunlight. Values from the scene will be mapped to a range fitting within the max luminosity set for the window. |
DarkKilauea
force-pushed
the
rendering/hdr-output
branch
from
b2bd1a1
to
728912f
Compare
|
Here are the changes to get Rec. 2020 HDR output on D3D12: master...alvinhochun:godot:hdr-output-d3d12 |
Quote
The over-exposure in your screenshot is expected, but the colours are oversaturated because it is missing a colour space conversion. The colours need to be converted from BT.709 primaries to BT.2020 primaries. This is how it should look with the correct colours:
The conversion may be done with something like this: diff --git a/servers/rendering/renderer_rd/shaders/color_space_inc.glsl b/servers/rendering/renderer_rd/shaders/color_space_inc.glsl
index 3583ee8365..76305a8a3c 100644
--- a/servers/rendering/renderer_rd/shaders/color_space_inc.glsl
+++ b/servers/rendering/renderer_rd/shaders/color_space_inc.glsl
@@ -19,6 +19,15 @@ vec3 linear_to_st2084(vec3 color, float max_luminance) {
// max_luminance is the display's peak luminance in nits
// we map it here to the native 10000 nits range of ST2084
float adjustment = max_luminance * (1.0f / 10000.0f);
+ color = color * adjustment;
+
+ // Color transformation matrix values taken from DirectXTK, may need verification.
+ const mat3 from709to2020 = mat3(
+ 0.6274040f, 0.0690970f, 0.0163916f,
+ 0.3292820f, 0.9195400f, 0.0880132f,
+ 0.0433136f, 0.0113612f, 0.8955950f
+ );
+ color = from709to2020 * color;
// Apply ST2084 curve
const float c1 = 0.8359375;
@@ -26,7 +35,7 @@ vec3 linear_to_st2084(vec3 color, float max_luminance) {
const float c3 = 18.6875;
const float m1 = 0.1593017578125;
const float m2 = 78.84375;
- vec3 cp = pow(abs(color.rgb * adjustment), vec3(m1));
+ vec3 cp = pow(abs(color.rgb), vec3(m1));
return pow((c1 + c2 * cp) / (1 + c3 * cp), vec3(m2));
}
|
DarkKilauea
force-pushed
the
rendering/hdr-output
branch
from
728912f
to
56d27a6
Compare
DarkKilauea
force-pushed
the
rendering/hdr-output
branch
from
37d8407
to
d5a7579
Compare
|
Update: I'm currently working on tone-mapping 2D elements differently from 3D, but I'm running into some issues with how Godot renders scenes in its render targets. Godot will render the 3D scene, tonemap that scene, then proceed to render any 2D elements directly into the same render target. Then, any render targets (from different viewports) are blitted together into the final framebuffer. I'm currently performing the colorspace conversion from sRGB/Linear to HDR 10 at this blitter, which cannot distinguish between the 2D and 3D elements. I figured I would then update the 3D tonemap shader and canvas shader to perform the colorspace conversion themselves, but the engine makes assumptions (which are invalidated by this PR) in various different parts of the renderer that only sRGB and Linear colorspaces exist, which is making it difficult to ensure that I don't accidentally perform a conversion that has already occurred. I'm also trying to make sure any changes made are as local and limited as possible to avoid making this PR harder to merge. I'm working my way through all of the sites where sRGB conversion takes place and trying to see if there is a clean way to track what conversions have occurred, or at least determine if there is a limited subset I can touch and assume the correct space later on. I'm assuming it would not be acceptable to have the canvas renderer render into its own render target and have the blitter combine them later. Not only would that cost more VRAM, but there is a computational cost as well. There would have to be more of a benefit than just making my life easier. :) |
DarkKilauea
force-pushed
the
rendering/hdr-output
branch
2 times, most recently
from
5f5f917
to
52059de
Compare
DarkKilauea
force-pushed
the
rendering/hdr-output
branch
2 times, most recently
from
5abaebf
to
4e94080
Compare
|
This is really neat, I've been waiting for a proper HDR10 implementation in Godot for awhile. A couple comments: I think this PR uses a Secondly, I've pushed an old HDR10 prototype for Godot I did awhile back, just in case it might help somehow. If you see anything useful, feel free to use it here. Edit: NVIDIA just released |
|
HDR with raw Vulkan on Windows (or anywhere TBH, even on Linux for example) is notoriously finnicky, it's yet another reason why the best Vulkan games on Windows tend to present through DXGI. As well as the fact that Windows and most modern compositors are moving away from "exclusive fullscreen", and expect games to present to the compositor now. (and leave the resposibility of making present low latency and high performance to the compositor) IIRC the latest versions of Windows actually don't have true exclusive fullscreen anymore, the compositor just "fakes" it when an app requests it now. |
DarkKilauea
force-pushed
the
rendering/hdr-output
branch
from
4e94080
to
1f2daf6
Compare
|
@Jamsers It's a frequent misconception that games should present through the compositor. Using the compositor means that the application is no longer in control of Vsync, meaning you can't disable Vsync or use VRR (it might also prevent HDR), and you'll either have terrible framepacing or added input latency (because the compositor's Vsync isn't even the same Vsync as the app would have been using). I bring this up because I often see advice floating around claiming that nVidia users should, for example, enable the "full composition pipeline" on X11. This is bad advice and will only cause more problems than it tries to solve. What DXGI does (particularly in newer versions of Windows) is it allows the application to bypass the compositor and send the image directly to scanout, which is effectively the same as exclusive fullscreen, except without the downsides (namely, lack of multitasking). It's functionally similar to what Gamescope does on Linux. If your version of Windows is new enough, the direct scanout even works in windowed mode. The reason for moving Godot to using DXGI is to allow the use of direct scanout, with all the benefits it provides, regardless of whether you're fullscreen or windowed. Without DXGI, the graphics driver is performing additional copies, sending the image through the compositor, etc. The move away from "exclusive fullscreen" is more to do with the fact that, at least on Windows, it was designed back in the Windows 95 era where "exclusive" meant "the app has full control over VRAM and the display" meaning VRAM gets cleared the moment you hit Alt + Tab, in addition to any sort of resolution/color depth/refresh rate change. Exclusive fullscreen (as it was originally implemented on Windows) and presenting to direct scanout are not one and the same. You can have direct scanout without exclusive fullscreen. |
Ah see but that's the thing, in an ideal world (and with modern compositors), you should not need exclusive fullscreen to get all these benefits - the compositor should be able to properly handle vsync, VRR, HDR, framepacing, and provide zero additional input latency, not just for games, but for any application that requests any of these features. (A lot of these are also useful for video player apps, for example) On the latest Windows 11, if the app presents through DXGI (guaranteed if your app is a DX12 app), this is indeed the case. The only reason why this is still bad advice on Linux is because X11 is so horrifically old and yet still so horrifically widespread, and quite frankly, because Wayland developers are suprisingly dismissive and slow to respond to gaming needs. (The notorious discussion where a Wayland engineer confidently claimed "[competitive] gamers can live with one frame of latency, tearing isn't worth it" comes to mind) |
|
Now Linux is as Linux does - if it takes Wayland a decade to catch up to Window's DWM, then so be it. But in the here and now, it's very doable to do things properly in Windows land and make exclusive fullscreen (or at least needing exclusive fullscreen) a thing of the past, and here's the kicker - for HDR in Windows specifically, presenting through the compositor gets you better results than exclusive fullscreen anyway. There is notoriously a whole bunch of games where HDR is just janky and broken if you're in exclusive fullscreen, that works just fine when you're in Windowed or Borderless Windowed mode. This is especially true if you have HDR enabled in the Windows settings because you need your desktop apps to render in HDR as well (i.e. HDR Youtube in a browser, or editing videos in HDR with Premiere Pro). Auto-HDR also doesn't work in exclusive fullscreen. |
Compositing is an indirection, and so is inherently incapable of providing those things, unless it's done at the hardware plane level (as in, the planes that are sent directly to scanout), and that comes with its own limitations.
The reason DXGI works so well is because it's using direct scanout via multi-plane overlay (MPO). This means it bypasses the compositor and uses the hardware planes directly as if it were exclusive fullscreen. In fact, once direct scanout kicks in, the DWM compositor may even go to sleep as it doesn't need to update the screen (direct scanout takes care of that) unless some other application in the background needs to update and is using compositing. Here's Microsoft's article on the subject: https://devblogs.microsoft.com/directx/dxgi-flip-model/ Edit: The DXGI API and documentation does make reference to "hardware compositing" but this might be a weird nomenclature thing. It's just going through the hardware planes instead of the traditional DWM compositor. Those are quite different things. Gamescope on Linux works similarly. It takes advantage of the hardware planes where possible.
What's funny is that at one point in time this actually did work on X11, even in windowed mode (referred to as "unredirection"). But for whatever reason WM devs scrapped it because of driver issues.
I share your frustration. But also IIRC, the latest KDE (and possibly GNOME) supports direct scanout on Wayland, and should this not be available, there's always Gamescope. Honestly, Wayland with direct scanout to MPO is the future. |
I'm getting some mixed impressions about this online. I'm seeing some folks say it only works in full-screen, others saying it works in windowed if you're on nVidia. One poster on Reddit says you need to uncheck "disable fullscreen optimizations" (i.e. let DXGI use direct scanout) for AutoHDR to work. Which makes me wonder how much of this mess is caused by users changing tweakables they probably shouldn't instead of just leaving things at the default. Or it could just be bugs in Windows and/or graphics drivers. |
But DWM on modern Windows does provide these things. (Although only to DXGI swapchains) Of course, I understand what you really mean. The compositor being able to do these things is due to tight cooperation between the hardware, drivers, OS, and compositor itself, and is fundamentally impossible to achieve with just the compositor. But what I mean is that from the API perspective of the application, you really are just presenting to the compositor. You're not "piercing the veil" anymore like you would with exclusive fullscreen. You just provide DWM with a pointer to your swapchain, the appropriate feature flags, and from the app's perspective the compositor just takes care of everything. |
The confusion is understandable, it's quite a mess. The fundamental thing to understand about Auto-HDR is that it actually only has one requirement - flip-model present, which of course also means a DXGI swapchain. For DX12 games, Auto-HDR will just work. For DX11 and DX10 games, it depends, but for simplicity's sake let's just assume the game's devs didn't do flip present. In Windowed and Borderless Windowed mode, Auto-HDR won't work. But in Exclusive Fullscreen mode, if you have "disable fullscreen optimizations" unchecked, modern Windows actually won't do true exclusive fullscreen anymore, and instead "fakes" it through a DXGI swapchain that emulates the exclusive fullscreen characteristics the game is requesting. So in a roundabout way you ended up with a DXGI swapchain, and hence Auto-HDR now works. If you have "disable fullscreen optimizations" checked, Windows will actually do true exclusive fullscreen, which means you don't get the DXGI swapchain, and hence Auto-HDR doesn't work. HOWEVER, Windows 11 updates added a new option called "Optimizations for windowed games". It's a completely separate option by the way, despite the very similar wording. When you turn on this feature, all DX11 and DX10 games get forced to flip present. So now Auto-HDR will work with all DX11 and DX10 games, in Windowed and Borderless Windowed mode. BUT, whether Auto-HDR works with exclusive fullscreen is still dependent on the "disable fullscreen optimizations" checkbox, with all the conditions outlined earlier. For OpenGL and Vulkan games, Auto-HDR doesn't work, unless the developer explicitly presents through a DXGI swapchain. But again for simplicity's sake let's just assume the developer didn't. HOWEVER... NVIDIA has a compatibility option that forces OpenGL and Vulkan games to use a DXGI swapchain. This option is set to auto by default, and auto relies on a whitelist. So if your OpenGL or Vulkan game is on said whitelist, then you get Auto-HDR in Windowed or Borderless Windowed mode. But if it's not, then you don't get Auto-HDR. But if you set the game to Exclusive Fullscreen, you get no Auto-HDR regardless of whether the game's on the whitelist or not. AMD apparantly also has forced DXGI related compatibility options for OpenGL and Vulkan games but they're even more opaque and incomprehensible than NVIDIA's so I can't explain them at all. Yeah. What an absolute cluster****. |
|
So one thing I'm curious about is what happens if multiple windowed + flip model games try to control HDR at the same time? I know that for VRR, strange things can happen, like one game capping the framerate also influencing the other. Even just having the Godot editor window open with windowed optimizations affects cursor smoothness as the editor only refreshes when it needs to (and with VRR the mouse cursor movement is tied to the current refresh rate, bottoming out at around 24-40hz depending on the monitor). |
|
I believe flip present HDR fundamentally cannot control the monitor's HDR and as such, require you to turn on HDR in the Windows settings (so that HDR is on for the desktop). HDR flip present windows can then be composited normally on a HDR desktop. (That's what I recall, but I could be wrong on that since I don't have a HDR monitor and only get to try out Window's HDR on the TV occasionally.) VRR is a different thing - flip present windows still do not wrest control from the OS, just like HDR, but due to the very nature of VRR fundamentally only one app can "call the shots" so to speak. The OS decides which app that is (a vast majority of the time it's just the foreground app). In the case of the Godot Editor, Windows is treating it like a game and letting it "call the shots" when it really should be treating it like a standard app window. Of course, this is complicated by the fact that by design, the editor acts a lot like a game because the editor executable and an exported game's executable is practically the same... the Godot editor is just a Godot game, after all. So the Godot editor is reporting itself as a game to Windows and of course, Windows treats it accordingly. In fact for me, every time I launch the Godot editor I get the NVIDIA Geforce Experience overlay pop up in the upper right that you normally only get when you launch games. 😅 |
DarkKilauea
force-pushed
the
rendering/hdr-output
branch
3 times, most recently
from
34fcfc8
to
a27598d
Compare
|
I think I missed these comments in the discussion above, sorry about that.
I noticed that several laptops I have don't support the R16G16B16A16 format (or equivalent BGRA version) for Vulkan, but it shows up fine in DirectX due to the requirements D3D12 sets. I can see an argument for choosing the larger format in case of alpha blending, since 2 bits of alpha is not really useful for anything other than masking. I'm still not sure the memory cost of the larger buffer is worth it in other cases though. For Vulkan, it seems like support for the extended sRGB colorspace is far more limited than HDR10, which is why I'm starting there. HDR 10 (PQ ST2084) seems to have the widest support. I'm not sure if it makes sense to allow the user to pick the color space or not. I haven't seen any games on the market that give you that choice, usually it is a binary toggle for HDR output, along with settings to provide the max and min luminance of the target display.
Thanks, I'll take a look and see if there is anything I should add to this implementation. |
|
Update: I've been doing a deep dive into the rendering architecture of Godot trying to figure out the best way to implement the color space conversion and luminance limiter. I think I have a framework for going forward: There are two primary tasks that need to be done to enable HDR output:
While rendering, Godot will create a color buffer for a viewport, then render the 3D elements to it, tone map those, then render any 2D elements to the buffer. This buffer is then blitted, along with any other viewports, to the swap chain. My current approach waits till the blit in order to do the conversion to the correct color space. I believe this is too late in order to control the luminance of 2D and 3D elements separately. So, the plan is to move all the color space logic to the 3D tone mapper and canvas renderers and perform that conversion as elements are rendered to the viewport's color buffer. This buffer would then be blit to the swap chain with no conversion done there (with the exception of the boot image). The downside of this approach is that, if the user has multiple viewports and forgets to setup a viewport with the correct color space and luminance information, then it will look bad on the final output. I'm not sure how to avoid that without a magic project wide setting that all viewports destined for the final swap chain obey. That doesn't feel like it meshes well with Godot's design philosophy. |
|
@DarkKilauea To add to the complexity, you also have to consider the following scenarios
Accordingly, I don't think it is a realistic goal to transform 2D and 3D into the final display color space separately. The current design was intentionally decided on considering that we would eventually be adding HDR10 support. IMO the user should be able to scale the overall luminance of the Viewport only. The brightness of the 3D scene can already be scaled independently of 2D using the |
|
@clayjohn Thanks for the input. This suggests a new approach that I hadn't considered. If I consider a color value of 1.0, 1.0, 1.0 in the framebuffer as the "paper white" point, instead of the maximum value, I can map values over 1.0 into the extended range for the display. This would allow devs to control how bright the UI appears by setting that "paper white" point in terms of nits. Then, both 2D and 3D elements can be brighter than that in terms of multiples of the "paper white" point. For example, a value of 2.0 would be roughly twice as bright as the "paper white" point (I'm hand waving away the logarithmic scaling applied by PQ ST.2084 in this example). I think this also gives more artistic control, since you have the ability to make an element in the scene exactly 400 nits on the display. The trick is trying to avoid clipping for displays that aren't that capable. I'm not sure if I should allow it to clip for brightness values over the capability of the display, or attempt to scale them into the supported range as the current method does. Either way, I think I will give it a try and see how it works out. It would massively simplify the effort. |
|
@DarkKilauea Awesome! I'm looking forward to seeing what you come up with |
DarkKilauea
force-pushed
the
rendering/hdr-output
branch
2 times, most recently
from
aa228e4
to
cb39952
Compare
Co-authored-by: Alvin Wong <[email protected]>
DarkKilauea
force-pushed
the
rendering/hdr-output
branch
from
cb39952
to
28ef190
Compare
This PR enables the ability for Godot to output to HDR capable displays. This allows Godot to output brighter luminance than allowed in SDR mode and with more vibrant colors.
Testing project: https://github.com/DarkKilauea/godot-hdr-output
HDR (blown out a bit, looks better on an HDR display):
SDR:
Supported Platforms:
Supported Graphics APIs:
Supported HDR Formats:
Work to do:
Help Needed:
Adding support for DirectX, I'm not currently sure how to update the swap chain to output an HDR format.Technical Details:
I updated the Blit shader to convert the viewport buffer into the ST2084 color space when an HDR format is detected. The max supported nits for the ST2084 color space is 10,000 nits, which consumer monitors are not capable of achieving, so some adjustment is required. I looked at how AMD did it with FidelityFX and took a similar approach to adjusting the curve based on the max luminance of the display.
Here you can see the curves at several peek luminance values:
Plotting the derivative, you can see the error amongst the adjusted values is small, which should result in images looking similar on different displays with different max luminance capabilities:
This is an approximation though, AMD's FidelityFX HDR Mapper does a lot of fancy logic with color spaces and likely does a better job of mapping colors from the source format to the display. However, this approximation looks good to me and may be good enough for now.