AMF support on linux

[cupown]

Is there an existing issue for this?

• I have searched the existing issues

Is your issue described in the documentation?

• I have read the documentation

Is your issue present in the nightly release?

• This issue is present in the nightly release

Describe the Bug

Currently, sunshine is not built with AMF support on linux. I didn't see any mention of that in the doc, I had to look at the code to understand why AMF wasn't working.

AMF should be supported on linux, at least until vulkan video encoding is done. AMF is faster than VAAPI.

I don't believe AMF need that much work to make it work on linux, someone that know a bit more than me about ffmpeg should be able to implement that quickly. For example, obs pr that implement AMF video encoding for linux: obsproject/obs-studio#7206

I tried to implement it myself with a simple patch:

diff --git a/src/video.cpp b/src/video.cpp
index 0ee04ff..9314046 100644
--- a/src/video.cpp
+++ b/src/video.cpp
@@ -22,6 +21,10 @@ extern "C" {
 extern "C" {
   #include <libavutil/hwcontext_d3d11va.h>
 }
+#else
+extern "C" {
+  #include <libavutil/hwcontext_vulkan.h>
+}
 #endif
 
 using namespace std::literals;
@@ -92,6 +95,8 @@ namespace video {
   vaapi_make_hwdevice_ctx(platf::hwdevice_t *hwdevice_ctx);
   util::Either<buffer_t, int>
   cuda_make_hwdevice_ctx(platf::hwdevice_t *hwdevice_ctx);
+  util::Either<buffer_t, int>
+  vulkan_make_hwdevice_ctx(platf::hwdevice_t *hwdevice_ctx);
 
   int
   hwframe_ctx(ctx_t &ctx, platf::hwdevice_t *hwdevice, buffer_t &hwdevice_ctx, AVPixelFormat format);
@@ -553,12 +553,18 @@ namespace video {
     PARALLEL_ENCODING | CBR_WITH_VBR | RELAXED_COMPLIANCE | NO_RC_BUF_LIMIT,
     dxgi_make_hwdevice_ctx,
   };
-
+#endif
   static encoder_t amdvce {
     "amdvce"sv,
+#ifdef _WIN32
     AV_HWDEVICE_TYPE_D3D11VA, AV_HWDEVICE_TYPE_NONE,
     AV_PIX_FMT_D3D11,
     AV_PIX_FMT_NV12, AV_PIX_FMT_P010,
+#else
+    AV_HWDEVICE_TYPE_VULKAN, AV_HWDEVICE_TYPE_NONE,
+    AV_PIX_FMT_NONE,
+    AV_PIX_FMT_NV12, AV_PIX_FMT_YUV420P10,
+#endif
     {
       // Common options
       {
@@ -597,9 +603,12 @@ namespace video {
       "h264_amf"s,
     },
     PARALLEL_ENCODING,
+#ifdef _WIN32
     dxgi_make_hwdevice_ctx
-  };
+#else
+    vulkan_make_hwdevice_ctx
 #endif
+  };
 
   static encoder_t software {
     "software"sv,
@@ -716,10 +725,10 @@ namespace video {
 #endif
 #ifdef _WIN32
     quicksync,
-    amdvce,
 #endif
 #ifdef __linux__
     vaapi,
+    amdvce,
 #endif
 #ifdef __APPLE__
     videotoolbox,
@@ -2031,6 +2010,8 @@ namespace video {
     hwdevice->init_hwframes(frame_ctx);
 
     if (auto err = av_hwframe_ctx_init(frame_ref.get()); err < 0) {
+      char string[AV_ERROR_MAX_STRING_SIZE];
+      BOOST_LOG(error) << "Failed to finalize context: "sv << av_make_error_string(string, AV_ERROR_MAX_STRING_SIZE, err);
       return err;
     }
 
@@ -2067,6 +2048,69 @@ namespace video {
     return hw_device_buf;
   }
 
+  util::Either<buffer_t, int>
+  vulkan_make_hwdevice_ctx(platf::hwdevice_t *base) {
+    buffer_t hw_device_buf;
+
+    auto status = av_hwdevice_ctx_create(&hw_device_buf, AV_HWDEVICE_TYPE_VULKAN, nullptr, nullptr, 0);
+    if (status < 0) {
+      char string[AV_ERROR_MAX_STRING_SIZE];
+      BOOST_LOG(error) << "Failed to create a Vulkan device: "sv << av_make_error_string(string, AV_ERROR_MAX_STRING_SIZE, status);
+      return -1;
+    }
+
+    return hw_device_buf;
+  }
+
   util::Either<buffer_t, int>
   cuda_make_hwdevice_ctx(platf::hwdevice_t *base) {
     buffer_t hw_device_buf;
@@ -2153,6 +2197,8 @@ namespace video {
     switch (type) {
       case AV_HWDEVICE_TYPE_D3D11VA:
         return platf::mem_type_e::dxgi;
+      case AV_HWDEVICE_TYPE_VULKAN:
+        return platf::mem_type_e::system;
       case AV_HWDEVICE_TYPE_VAAPI:
         return platf::mem_type_e::vaapi;
       case AV_HWDEVICE_TYPE_CUDA:

But sadly, it didn't work with the bundled ffmpeg, it error on av_hwdevice_ctx_create, with "cannot allocate memory", and with my system ffmpeg, it error on av_hwframe_ctx_init with "function not implemented".

Expected Behavior

No response

Additional Context

No response

Host Operating System

Linux

Operating System Version

n/a

Architecture

64 bit

Sunshine commit or version

0.20.0

Package

n/a

GPU Type

AMD

GPU Model

n/a

GPU Driver/Mesa Version

n/a

Capture Method (Linux Only)

No response

Config

n/a

Apps

No response

Relevant log output

n/a

[KuleRucket]

Do you have any benchmarks/evidence to back up "AMF is faster than VAAPI"?

vappi and amf are just different APIs to the same hardware I'm not sure how much faster amf could be.

[cupown]

It's possible that the problem is with the implementation of VAAPI in FFmpeg and not with AMD VAAPI implementation.

The problem have been known for a while with OBS, the AMF fork that include the patch that I linked is quite popular, even being packaged by default in some distro. With OBS, the default VAAPI implementation either have a lot of compression artifacts at low bitrate or start to drop frames at high bitrate. I haven't tested but some people are also satisfied with using a GStreamer VAAPI implementation to encode with OBS.

I can observe the same thing in sunshine, trying to stream 4k with 60 fps, 60-70 Mbps have nearly no dropped frames but lot of compression artifacts, going higher solve some of those problems, but for example using 150 Mbps, nearly half of the frames are dropped. On OBS, it's the exact same with default encoder but using AMF, allow to do the same things with way less compression artifacts but most importantly not a single dropped frames at high bitrate.

I hope that by using AMF with sunshine, it will solve my current streaming problem. As currently, it's not usable on my configuration.

[MatthiasGrandl]

I can also confirm. 4k@60 is unusable using VAAPI, even though my GPU should absolutely be sufficient (7900xt)

[cwegener]

I tried to implement it myself with a simple patch:

You would need to actually write an implementation to talk to AMF on Linux (just like it exists for NVENC)

Linux does not have DirectX, so access to the AMF hardware is not abstracted through a unified interface like it is on Windows.

[Arbitrate3280]

Any updates on this? FFMPEG has the capacity to offer AMF encoding on Linux, it would be great if this was enabled in Sunshine.

I too hope that by having the option to use AMF and compare it with Vaapi I can work out some issues I'm currently having.

[Smoukus]

VAAPI Encoding on Linux curently is just bad experience (@2160p resolution).
Please consider adding AMF encoding.

Here is the idea

[vgraev]

it would be great, if developers add this feature to sunshine) life is not fair)) amd open linux drivers are great, but i use sunshine to stream and unfortunately nvidia codecs are better then amd

[davigamer987]

+1 on this, VAAPI Looks really bad for me at 1440P even at high bitrates, AMF on windows looks really good when I tried it

[Milor123]

AMF should be enabled as optional, in some linux system VAAPI works really bad and slow comparing with AMF.
For example in my old ubuntu 22.04 i get 4-5x of speed in FFMPEG using h.264 vaapi and cant run AMF, but in Arch linux, i get 12x of speed doing the same process with h.264 AMF.

How can I force to use AMF instead of Vaapi in sunshine ?
Thank u very much

[ReenigneArcher]

How can I force to use AMF instead of Vaapi in sunshine ?

You can't... that's why this is a feature request.

[Altokos]

Same issue here, I have better perfs with software encoding with my 5700XT.

[tomasdeltell]

Any updates fro this request?

[cgutman]

We cannot implement this until FFmpeg implements support for encoding GPU textures via the AMF encoding APIs. FFmpeg AMF encoding supports this on Windows via DX11 but there is no support on Linux.

[tomasdeltell]

But FFmpeg supports AMF encoding on Linux via Vulkan, if I'm not mistaken.
For example, OBS can do it via FFmpeg on Linux installing the AMD propietary drivrs (amdgpu-pro)

[cgutman]

AFAICT you can initialize it using a Vulkan context, but you can't actually pass frames to it without each of them being read back GPU->CPU->GPU for encoding. The only hwaccel backends it supports are Windows-specific.

https://github.com/FFmpeg/FFmpeg/blob/65739691b90012c9d93b2e5e0e89a55d8de9eb7b/libavcodec/amfenc.c#L788

It's possible that we can write standalone AMF support similar to what we did for NVENC on Windows (using the encoding APIs directly rather than going through FFmpeg), but I'm not sure how much work that would be. I'd certainly accept PRs for that.

[Danik1601]

OBS added AV1 via VA API on Radeon RX 7000 and Intel Arc. Will be available with next OBS Studio release.
(I understand that most people now limited to h.264 and h.265 hardware, just keeping you updated)

https://www.phoronix.com/news/OBS-Lands-AV1-VA-API

obsproject/obs-studio#8832

[cgutman]

We've had AV1 VAAPI support for several months already. It was included in the v0.21.0 release.

[Danik1601]

FYI:

AMF 1.4.33 released today, changelog mentions "experimental" RADV support:

• AMF on Linux can now be used with AMD Pro Vulkan, and experimentally with RADV drivers

https://github.com/GPUOpen-LibrariesAndSDKs/AMF/releases/tag/v1.4.33

UPD:
This might be important too:

• Vulkan encoder became independent from Vulkan driver
• Switched to public Vulkan Khronos extensions for decoder

[olivierDelierre]

Any news regarding this? The gap between Windows using AMF and Linux not using it is impressively massive for 4K gaming, which is the only viable option with 1440 for 55 inches screen.

[gschintgen]

You might want to give the workarounds using VAAPI a try. See #2193
Putting more load on the encoder seems to help. (I'm not saying this is a substitute for AMF.)

[phamaralbr]

Any updates on this? Is it possible to use amdgpu-pro drivers to get AMF working on Sunshine?

[gschintgen]

No.

[gschintgen]

Hi all, I'm convinced that most of the issues you're experiencing is due to VA-API not exposing AMD's low latency mode.

This can be fixed (in a slightly hackish way) by recompiling mesa with a patch. All the feedback thay I've seen has been positive and the encoding times reported by Windows users are the same as those reported by Linux users with a patched mesa (10-14ms at 4k60).

This is now documented in Sunshine's docs: https://docs.lizardbyte.dev/projects/sunshine/en/master/troubleshooting/linux.html#amd-encoding-latency-issues

Without the patch there are definitely issues, 4k being particularly affected: #2193

If you go this route and recompile Mesa, please consider sharing instructions for your distribution.

I'll go first, here's for Ubuntu: https://gist.github.com/gschintgen/ea148962f024b1c75e3c9ad5a114afa5

(I'm not saying AMF should not be supported, but priority should lie in fixing the actual issues, independently of the api.)

[entropicdrifter]

I was able to use this same script (minus one line because I turned on source code repos manually) in order to patch my kisak-mesa on Linux Mint

[nolan-at-shippo]

Just wanted to chime in and mention that with the latest release of AMF on Linux, the RADV driver is now fully supported for encoding. In theory at least, this makes it seem to me that we could now have our cake and eat it too