Why isn't .NET used for multimedia production?

[winscripter]

I've been wanting to implement some multimedia-related apps in C# recently, like video conferencing apps or video editors, but I ran into a problem: there's no library for that in the ecosystem. So I have a question:

Why doesn't anyone implement video and audio libraries in .NET? How come .NET doesn't have any codecs for popular video and audio codecs H.264, AAC or MP3 in its entire ecosystem? In the entire .NET ecosystem (NuGet and GitHub), there's no codec for the aforementioned codecs/file formats that are used everywhere in the multimedia world, there are no such libraries. There are only the wrappers for native libraries (FFmpeg, OpenH264, Windows Media Foundation, ...) but no direct C# implementations.

At first, I'd think it's because it's just too complex - but theoretically, it's more complex to implement such codecs in C than in C# because in C# there's less boilerplate code - and would you look at that, almost all implementations are written in C. Not to mention, there are more C# developers than there are C developers.

Then I'd think performance, but C#/.NET is actually quite competitive to C in terms of performance when used correctly. It's sometimes even been proven to be faster, and generally speaking, it can be tuned further. You've got structs, parallelism, vectorization, intrinsics, threading - and you can even tune the performance of the application by putting it into release mode or using NativeAOT, ReadyToRun, or PGO. And it's also memory efficient thanks to the Garbage Collector. Also, there are even codec implementations in Java (mp4parser for MP4, jaad for AAC, jcodec for H.264) which actually isn't as fast as .NET, which makes this question even more interesting.

Then, .NET developers actually seem to be interested in this happening. Take a look at libraries such as RtspClientSharp - it's an RTSP client written in C#, and even though there's only about 150k NuGet downloads, there's already ~750 stars. SixLabors.ImageSharp has 8k stars on GitHub (more information below), SIPSorcery for VoIP has 1.7k stars, GroovyCodecs for G.729 and G.711 and MP3 audio codecs has 27 stars despite only 8 commits, Concentus, an Opus codec written in C#, has 300 stars, and SharpAvi, an AVI reader/writer in .NET has 170 stars.

I'd then think licensing and patents, but:

• Some codecs like VP8, VP9, AV1, or H.262 are royalty-free (with H.262 and VP8 being relatively simple, and H.262 is used everywhere in Blu-ray discs and is also simple and its patents expired/expiring)
• For H.264 and H.265, I believe developers can just use delegate the need to pay for royalties and obtain licenses to the user (in contexts where it is required to do so legally). This might be a bit off-topic, so it probably doesn't need much conversation, but just in general: if it's a thing in other languages, it should probably also be a thing in .NET - there's nothing in .NET itself that changes how this licensing is applied.
• For AAC, the LC channel has had patents expired in 2017, and it itself is used almost in all MP4 video file, so theoretically there's nothing stopping from people implementing it

.NET has also been cross-platform for a while now, which makes this question further more interesting.

Let's also take a look at SixLabors.ImageSharp for a moment. It's one of those multimedia-related libraries that is actually written directly in C# - it's a popular cross-platform imaging library, it implements image formats like PNG, BMP, JPEG, TIFF, TGA, QOI, WebP, GIF, PBM. And what do we learn from this? Well, if we use it in action, we can see that it is incredibly fast and memory efficient - so that just proves that .NET can be used to create multimedia libraries - so why hasn't anyone done this yet? Even the developers like it - 200M downloads on NuGet, nearly 8k stars on GitHub.

So, are there any factors that ultimately result in .NET not having any video/audio/multimedia libraries in its ecosystem (specifically NuGet, GitHub)? Developers seem to be interested, and .NET itself might actually be fast enough for writing this.

Why even have a multimedia library in .NET in the first place?

Theoretically, this is a must-have in many languages, but in .NET, it's especially a must-have rather than using native tools like FFmpeg.

First of all, deployment and portability. If we use an FFmpeg binding, well, since FFmpeg is unmanaged, we will need binaries for all architectures (x64, ARM64, RISC-V, ...) and platforms (Windows, Linux, WebAssembly, ...) - not to mention, FFmpeg is a large binary (1xyMB where x >= 2) per platform and architecture. In .NET, it's one single binary. Also, if I want to use it in my application, I might also want to make a Blazor WebAssembly version (I do use it quite often, I like creating client-side websites), and there's no wasm version of FFmpeg or OpenH264, so if there's a C# library, it could be much more beneficial for deployment against both Desktop, Mobile, and Web versions.

Next, integration. Writing a multimedia library in .NET means it integrates better with other .NET primitives like System.IO.Stream.

Finally, memory efficiency - since well, garbage collection.

[Clockwork-Muse]

Why doesn't anyone implement video and audio libraries in .NET?

The short answer, for many languages, is because they were already implemented in C, and it's way easier/faster to call into the C library then re-implement it. This is especially true if the consuming language makes it difficult or dangerous to write a similarly performant version.

we will need binaries for all architectures

This is true, but often this is accomplished by building platform-specific binaries and only distributing the relevant platform-specific dependencies. This complicates build setup somewhat, but results in smaller output binaries.

So, are there any factors that ultimately result in .NET not having any video/audio/multimedia libraries in its ecosystem (specifically NuGet, GitHub)? Developers seem to be interested, and .NET itself might actually be fast enough for writing this.

The lack of interest in writing it, probably. That's why SixLabors has a commercial (split) license on their package, for instance.

Note that this isn't something the BCL would implement, we would expect such a thing to be provided by the wider community.

Finally, memory efficiency - since well, garbage collection.

GC doesn't make things more memory efficient (it often has at least a slight cost), it just often makes sure things get cleaned up. In fact, any performant implementation likely would internally be using techniques and patterns that rely on unmanaged (not handled by the GC) memory and other resources, or at least techniques that look closer to what the "C native" implementation does.

The real benefit to GC isn't memory efficiency, it's developer efficiency due to not needing to worry about keeping track of object lifetimes. But there are ways it can be abused or misused, and for some situations you need techniques that essentially ignore or sidestep the existence of the GC.

[winscripter]

This is an amazing answer, thank you for your response.

I do have a few more questions.

I'll start by clarifying some potential misunderstandings here.

Note that this isn't something the BCL would implement, we would expect such a thing to be provided by the wider community.

Yes, I'm aware of that. The BCL is supposed to be the base of the runtime, and is supposed to be the backbone required for applications to perform the simplest yet most commonly used tasks, not something specific like imaging or multimedia processing. I didn't propose multimedia processing directly to the .NET Runtime Team, I was just asking a question: why no .NET developer(s) ever implemented it yet, considering there are more .NET developers than there are C developers and the potential ups and downs in doing so.

GC doesn't make things more memory efficient (it often has at least a slight cost), it just often makes sure things get cleaned up. In fact, any performant implementation likely would internally be using techniques and patterns that rely on unmanaged (not handled by the GC) memory and other resources, or at least techniques that look closer to what the "C native" implementation does.

By "memory efficient", I meant that it helps prevent memory leaks since, unlike C/C++, in C#, you don't need to manually free .NET objects, and you may not forget to do so (if you do in C/C++, it will cause memory leaks) - the GC does it for you.

The specific GC objects that I'm referring to are those that simply can't be allocated as unmanaged memory, and even if they can, code might not become platform-independent anymore (i.e., you'd have to switch from stackalloc to NativeMemory.Alloc or something like that, since the .NET stack is only limited to 1MB memory). Those are, for instance, frame buffers or large collections. Or for instance, abstracting objects - i.e., in most video codecs, there are so-called "Reference pictures", and in H.264 specifically, each picture can be of multiple types (Complementary field pair, frame, field), and one will have to use a reference type class to abstract them. Things like small audio blocks can be represented as a stackalloc Span, 16x16 to 64x64 frame blocks can also be represented as a stack-allocated span or perhaps lots of Vector128/Vector4 instances, and a motion vector, a small X/Y pair that represents how pixels shifted from previous frame to current frame, also used in every video codec, can be represented just like this:

public record struct MotionVector(int X, int Y);

Now, onto questions:

1. Java has multimedia libraries, so why not .NET?

Libraries in Java like mp4parser - an MP4 library with a bit of H.264, H.265, AC-3, E-AC-3, DTS, and MP3 support, JCodec - implementation of many codecs like H.264, VP8, VP9, PNG, JPEG, Motion JPEG, and container formats like AVI, MP4, DPX, WebP (just to name a few), and even JAAD - implementation of an AAC decoder - with support for the HE channel whose patents still didn't expire - in Java, all exist.

So:

	.NET	Java
Better performance? (In general)	✅ (And can be officially tuned in many ways)	❌
Better memory efficiency? (In general)	✅ (Usually)	❌
Cross-platform?	✅	✅
Supports interoperability with native libraries?	✅	✅

Considering those factors, I'm really curious to know why Java has more multimedia libraries directly implemented in that language compared to .NET.

2. The potential outcomes - why still no interest in implementing?

As mentioned earlier, .NET developers are really eager to see any multimedia implementation written in C# - even relatively simple multimedia implementations, like an RTSP client, can get many stars on GitHub. So, if developers aren't interested in writing a multimedia library in .NET, don't they also think of the outcomes? If they write a multimedia library in C#, it would very likely get lots of traction, considering that this is the case for many other multimedia implementations, including those that are doable. I think there are developers who might want to write it as they know that it's likely to get much traction - it's just that there's perhaps some other reason that's blocking them from doing so (I assume).

[Clockwork-Muse]

By "memory efficient", I meant that it helps prevent memory leaks since, unlike C/C++, in C#, you don't need to manually free .NET objects, and you may not forget to do so (if you do in C/C++, it will cause memory leaks) - the GC does it for you.

There are still ways to "leak" memory in C# and other memory-managed languages (forgetting to evict things from a cache being the traditional example). Given the likely implementation needs of a multi-media library, you're likely to run into similar problems as non-managed languages, too.

Part of this is because expecting the GC to clean up for you has performance penalties

Java has multimedia libraries, so why not .NET?

You'd have to ask the library authors.

That said, I suspect history.

• The tooling for Java has been accessible and free to use (and importantly, free to use for commercial purposes) longer than similar tooling for C#.
• Browser-based games and applications used to be commonly written in Java (this required a plugin).
• Java ended up being a common into-to-programming course series/language in colleges.

Java also ended up being used to program things like smart TVs and other multimedia devices, but I'm not sure if it's because the media libraries already existed, or the media libraries existed because of this.

[winscripter]

Thanks again for your amazing answer.

One more question: considering this, do you think it's worth it for me, myself, to try and implement these codecs and file formats in .NET, make them open-source and platform independent, and publish on NuGet? I mean, do you think it's worth the time and effort to write and use that in C# instead of using the interop approach? I do have time to write this, even if it's going to span months, but I'm just curious to know if I should do it for any benefit or I should just bind to native libraries and tools. Which is more recommended in general: binding or manual implementation, even if it means lots of time and effort?

[huoyaoyuan]

One more question: considering this, do you think it's worth it for me, myself, to try and implement these codecs and file formats in .NET, make them open-source and platform independent, and publish on NuGet?

It depends on your goal: to get reliable functionality, or to learn/contribute to .NET ecosystem.

I do have time to write this, even if it's going to span months, but I'm just curious to know if I should do it for any benefit or I should just bind to native libraries and tools.

An interesting sample is the ManagedZLib project. It was an intern project and never made for production. Another example is ManagedQuic, which is implemented by the current System.Net maintainers.

The difficulty for managed implementation is very high - it needs optimization and verification, and continuous maintenance to get latest functionalities.

Which is more recommended in general: binding or manual implementation, even if it means lots of time and effort?

I would only recommend manual implementation for parsing mature formats. For example, a decade ago I wrote MP3 header parsing manually.

[winscripter]

Thanks for your answer, @huoyaoyuan. Based on this, I think I'll at least try to implement those codecs and file formats and publish them to GitHub and NuGet. I'll give the manual implementation a shot and see how far I can get. If it proves too complex or inefficient, I may fall back on native binaries like FFmpeg - but I'm excited to explore the possibilities first.

The difficulty for managed implementation is very high - it needs optimization and verification, and continuous maintenance to get latest functionalities.

I've worked with multimedia before - implemented RTSP/SDP and studied specs like H.264 and JPEG - so I'm hopeful I can make progress, especially with .NET optimizations like SIMD and parallelism. I will make sure to rely on unmanaged resources, like stackalloc, fixed-size buffers, vectors from System.Runtime.Intrinsics, and value-type structs to ensure best memory efficiency and performance.

I would only recommend manual implementation for parsing mature formats.

Codecs like H.264, AAC and HEVC (H.265) are mature, they're used everywhere in professional video/audio today and they've been around for a while (with HEVC being the newest, 2013) so I might be able to implement those manually... if I'm skilled enough and I grasp the specification well.

---

To anyone curious looking into this post, the implementations will be under Project ContentDotNet. It's still barely in Alpha release, so if there's something weird or off, it's probably to be expected. I will make sure to reimplement many components.