[SDL 3] GPU feature request: Support upscalers (FSR, DLSS, Xess, MetalFX)

[savant117]

It would be nice if it were possible to support one or more of the various upscaling techniques. The way I see it, there are a couple paths here:

1. Abstract away the upscaling process behind some new SDL GPU APIs. The API would choose the upscaler behind the scenes (DLSS on supported nvidia, Xess on supported intel, MetalFX on supported mac/ios, FSR otherwise). This is probably not the preferred route since these will bloat SDL (especially Xess, which is huge). You could maybe drop Xess as an option and just use FSR on intel, then it wouldn't bloat so much since DLSS can use the DLL shipped with the driver, and FSR is not too large on its own.

2. Add functions to get pointers to native device objects of the underlying backend (command queue, texture, buffer, etc). Then the user could still use SDL GPU and just pass the native objects to the upscaler API of their choice. This has a side benefit of allowing users to tap into other features available on the executing platform which are not currently exposed via SDL GPU APIs (like timers, raytracing, etc).

Personally I love the new SDL GPU API, but lack of ability to use upscalers and timer queries in particular are a bit of a road block. I'd be happy to contribute in these regards if a plan could be formulated and approved. Thoughts?

[thatcosmonaut]

I do not want to expose object handles. We do this the portable way or not at all.

Hooking into vendor-specific solutions is going to be ugly and painful. I don't think it fits with the SDL philosophy of minimal dependencies.

There are some API-level upscaling solutions like DirectSR and MetalFX. Since we already depend on these APIs creating an upscaling API that calls these might be workable. The fact that Vulkan doesn't seem to have an API like this is rather unfortunate though.

I'm in favor of queries, someone just has to do it: #11696

[savant117]

Ok so the only acceptable way forward is having SDL apis that handle the upscaling.

If we used DirectSR and MetalFX with integrated FSR as a fallback, that would cover all SDL GPU supported platforms, backends and hardware, with the only 3rd party library included being FSR (which is pretty small). Is including FSR (or even some custom fallback) in the core SDL library a dealbreaker? If so, maybe it could be done as an add on package (similar to SDL_shadercross)?

I guess another option would be to use DirectSR and MetalFX when available, but just do a simple blit or something like TAAU if neither is supported (ie vulkan, webgpu backends)

I'll take a look at queries, I think that shouldn't be too difficult.

[thatcosmonaut]

A satellite library isn't really going to work because it would need access to the implementation details.

@icculus what do you think about linking to FSR dynamically? I have some hesitations about it. There have been four versions of FSR in four years. Keeping up with this treadmill is probably going to be irritating and it's likely that supporting new versions would require new APIs.

[icculus]

@icculus what do you think about linking to FSR dynamically?

I think I want to not add this feature at all until there's an extremely good reason to do so.