ideas to improve windows header files and libc #9998
Comments
|
There are at least a couple other things mingw provides:
The first is something we could probably generate from the win32 metadata, but there may be hidden dragons in 2. Particularly, C libraries that assume that windows && !MSVC means mingw, and use mingw-specific header declarations. |
|
Yeah def files are trivial to generate with win32metadata. I didn't really consider C++, there could definitely be some gotchas there. |
|
@marler8997 Wouldn't it be easier to maintain our own fork of Mingw-w64? Unfortunately https://github.com/microsoft/win32metadata also has bugs. Currently there are some critical errors for some DX APIs. Also, looking at: https://github.com/mingw-w64/mingw-w64/commits/master it seems that development is pretty active. Have you tried reporting issues there? |
andrewrk
added
os-windows
proposal
|
This is quite an ambitious and interesting topic. I like the energy. Let's go over all the pieces here individually; there is a lot to consider.
Can you elaborate on this? Seems like you have some insight that I am not privy to.
I share your preference for more "modern" systems for patches than email. However, if I'm being honest, it can take months for Zig pull requests to get merged too, sometimes. We have a few open since May of this year and that's on me, I really need to spend the time to figure it out and get those PRs merged or reviewed or otherwise addressed. But point being, can we do better than MinGW on this one metric? Maybe not. I've also had some really pleasant interactions with Martin Storsjö when I asked for help on the MinGW-w64 on IRC. I do think that upstream is willing to work with us, and I personally see that relationship as valuable. This particular point, I do not see as compelling. However...
I'm pretty sure I agree with you, but I'm curious what in particular you are considering.
I'm also curious to hear more details about this. I've used mingw-w64 in the past for various projects, and found its binary compatibility with MSVC to be flawless. Maybe I just got "lucky" but either way, it would help to have some more information on this axis to help move this issue along.
So, I'm actually finding myself kind of disagreeing with your reasoning here, but agreeing with your conclusion regardless. Idea being that if we can fulfill the Windows target features, as you listed them (implementation of libc for windows, .def files for windows, C/C++ headers for windows), without depending on this third party project, that would be an improvement. Let's address each use case independently: implementation of libc for windowsIndeed, there is a plan to do this already: #2879 Even if we stick with mingw-w64, the plan is to reduce the number of files that we depend on. For example, at the very least, all the math functions could be common and shared between all of zig's libc targets. Even if this issue were not accepted, any progress towards replacing components of mingw-w64 libc with zig libc files would be welcome. .def files for windows
Neat! That's good to know. C/C++ headers for windows
This is a cool idea - I hadn't considered that this is possible. Do you think it has a realistic chance of working, in the sense that it could generate a viable windows.h file that would be compatible with existing software? It might be worth doing a proof-of-concept for this to explore whether this is feasible. @michal-z's point is important:
That's good to know. One of the benefits mingw-w64 provides is that it has people using it and curating fixes to stuff like this. So we would be taking on that labor burden if we tried to do it with win32metadata ourselves. It could still be done, but that's something to consider. If win32metadata has bugs, that can be worked around, by us having "patch" data on top of it, that applies before lowering win32metadata to .h file(s). This is similar to the strategy that we do with tools/update_cpu_features.zig which applies some adjustments on top of LLVM's CPU features table data. ConclusionSeems like we should start with migrating logic from mingw-w64 to zig libc and then re-evaluate after that.
Does the existing mingw-w64 integrations not already solve this problem? If not, can we list some explicit problems with status quo? Anyway, seems like we have some stuff to explore. My goals are:
I think I forgot a couple points, but hopefully this is enough to keep the issue moving forward. |
|
Andrew thanks for taking the time to respond with all your thoughts. I think we're mostly in agreement here. I think I'm going to go ahead with enhancing my zigwin32gen project to generate C/C++ headers first. This will be an easy (and fun) task for me that will give us alot of information about the feasibility of decreasing dependence on MinGW, and getting an idea of what benefits may come from it as well. In simple terms, this task is generating all the files in Below you'll find some of my thoughts and experiences. Since I think we're already in agreement, it doesn't seem necessary but I've included it anyway since you asked for it. Thoughts and Experiences (unnecessary reading)The "seems to have alot of bugs" statement comes from trying to use "zig cc" on various projects and getting header file errors from the mingw headers. For example I've come across this one a few times: I'm guessing a portion of the errors I've seen over time were because of "zig cc" and not mingw itself. It just "seems" to me that it tends to have bugs for simple things that it shouldn't have, non-complicated use cases. Note that my use of the word "seems" here is intentional and comes from my limited experience. But even if mingw isn't all that buggy, Zig's usage of it (manual inclusion of particular files) is, and its usage is brittle since it's not supported by MinGW. I don't think MinGW is going to put in effort into supporting Zig's use case (and why would they) so this "bugginess" which results from the combination of Zig is the reality of our situation. There's a lot of work left to do on the Zig windows toolchain, and when I see a problem with mingw itself, I know it's going to take along time to fix. For example, I submitted a 2-line change to fix how they were including a particular header. This change was needed in order to allow projects to have header files named I'll also point out that fixes to MinGW are dependent on the mingw release schedule. This could be mitigated with a pre-release patch system, but I'm weighing that work against what it would take to release or decrease dependence on mingw.
I should explain my motivation here. For some reason, the idea that Zig could be a replacement for Visual Studio is what excites me. My goal is to get Zig to become a drop-in replacement for msvc in as many projects as possible. I think dependence on MinGW is going to hold us back here. MinGW seems to have lower aspirations. For example, when I submitted a patch to fix an issue with some WSA inline function, rather than try to match what MSVC was doing which would have resulted in the most compatible solution, they were more concerned about modifying their headers and potentially breaking projects that depended on mingw's alternative behavior. I think this is a valid approach for MinGW to have, but I don't think it's the right approach for Zig. If we want to be a drop-in replacement, then compatibility with MSVC would be more important than supporting projects who have become dependent on our particular toolchain idiosyncrasies. |
Short term definitely. Long term, almost certainly not. Long term we should already have a common libc we can leverage, and zigwin32gen/win32metadata can also be leveraged for most everything else. I'm not sure about libc++, but we could still leverage MinGW for that if we were so inclined.
True, but they're "usually" quick to fix them and when we don't want to wait for them, I've already got systems in place to patch them.
Patching data like win32metadata is easier than patching a codebase like MinGW. |
|
That doesn't mean dropping optional compatibility with MinGW? Just because microsoft compiler is the "dominant" one doesn't mean that there isn't many great projects that depend on MinGW in some form. |
|
I am discovering mingw-w64 headers are missing key updates to the Direct3D 12 headers, and the latest D3D12 headers from the Windows SDK are incompatible with mingw-w64's COM definitions etc. so I cannot intermix them. This is preventing me from building Dawn / WebGPU on Windows. I saw your project above and I found your genc PR marlersoft/zigwin32gen#10 but I am curious how far along this work is, and if you still plan to work on it further? |
|
@slimsag can you check mingw-w64 master branch and see if those headers are present and have the desired updates? List of changed files: |
|
@andrewrk Yeah, this was the first thing I checked actually. Unfortunately, latest mingw-w64 headers do not fix this. I will see about filing issues upstream on mingw-w64 Here are some concrete examples of what I've found to be missing in the latest upstream
I suspect that there are many more definitions than just these missing |
|
Please note that latest DirectX 12 headers from Microsoft are here: https://github.com/microsoft/DirectX-Headers (those are standalone and MIT licensed). DirectX 12 headers that come with Windows SDK have problems with functions that return structure by value (when compiled as C code). mingw-w64 should use headers from Microsoft git repo if possible. |
|
@slimsag I think that your best bet is to take headers from https://github.com/microsoft/DirectX-Headers and put them into Dawn source tree. |
|
@michal-z The primary problem isn't that the d3d12.h is outdated, I can workaround that by including updated d3d headers as you noted. The big problem is those last two bullet points I mentioned:
These are direct dependencies of Dawn and the mingw-w64 headers are incompatible with them. |
|
Note that we can carry patches to our mingw-w64 headers in the zig repo as long as the patches have been accepted upstream as well. |
|
@slimsag Regarding macro definitions, I think you can do something like: in But, yeah, UWP headers are problematic. |
|
Got one piece of info from |
|
Lengthy write-up incoming, hoping this is useful to whoever else stumbles across this issue and attempts to solve it. After spending ~2 weeks on this, I don't feel I've made much progress and feel a bit ill at how large of a problem I now realize this may be for Zig w.r.t. cross compiling for Windows. Hopefully some of this information is useful for us finding a long-term solution to this. It feels a bit like I've traveled inside a broken Fukushima reactor and back again, I'm planning to find some band-aids for myself and I probably won't dive back in again for now. Here goes.. Windows header file sourcesAuthoritative sources for Windows API headersWe can find these in two places:
MinGW-w64 may have some other headers (sal.h?) which it does not acquire from Wine, I didn't investigate this, but I strongly suspect almost all headers mingw-w64 distributes are generated from Wine's IDL files. Could be wrong though. Non-authoritative sources for Windows API headers
Using IDL filesHow IDL -> .h happensThere exist two IDL compilers which parse
IDL files are actually incredibly similar to the C header files themselves, what the generator does is quite minimal translation of syntax (but more on this below.) IncompatibilitiesAny header compiled with Microsoft's MIDL compiler, i.e. the
Are fundamentally incompatible with mingw-w64 and gcc outright, and fundamentally incompatible with clang/zig when targeting a GNU ABI. Header files generated using the Wine WIDL compiler (those distributed by Zig, those distributed by mingw-w64, and those in the Wine repository) are compatible with with all compilers (more on this below.) Atrocities of COM / win32 headersThe COM calling convention differs from the C calling convention, in specific msvc has undocumented behavior in which aggregates are returned by pointer using a hidden 2nd parameter in COM C/C++ functions and methods. Clang and Zig emulate this undocumented msvc behavior when targetting msvc ABI, but not gnu ABI. the IDL files distributed by Microsoft have attributes which denote which COM methods require a hidden parameter in order to store the return value. How this is handled is where Wine and Microsoft headers differ greatly. When it comes to the C definitions, they are identical. For example Wine and Microsoft headers both define: D3D12_CPU_DESCRIPTOR_HANDLE *( STDMETHODCALLTYPE *GetCPUDescriptorHandleForHeapStart )(
ID3D12DescriptorHeap * This,
D3D12_CPU_DESCRIPTOR_HANDLE * RetVal);However, when it comes to C++ definitions Microsoft defines just: virtual D3D12_CPU_DESCRIPTOR_HANDLE STDMETHODCALLTYPE GetCPUDescriptorHandleForHeapStart( void) = 0;Despite what is written above, msvc and msvc-emulating compilers detect that the return value of this function is an aggregate and inject a second parameter (the return pointer.) Wine on the other hand defines this same C++ virtual method as: #ifdef WIDL_EXPLICIT_AGGREGATE_RETURNS
virtual D3D12_CPU_DESCRIPTOR_HANDLE* STDMETHODCALLTYPE GetCPUDescriptorHandleForHeapStart(D3D12_CPU_DESCRIPTOR_HANDLE *__ret) = 0;
D3D12_CPU_DESCRIPTOR_HANDLE STDMETHODCALLTYPE GetCPUDescriptorHandleForHeapStart()
{
D3D12_CPU_DESCRIPTOR_HANDLE __ret;
return *GetCPUDescriptorHandleForHeapStart(&__ret);
}
#else
// We crash here!
virtual D3D12_CPU_DESCRIPTOR_HANDLE STDMETHODCALLTYPE GetCPUDescriptorHandleForHeapStart() = 0;
#endifWhen Relevant links:
Today, GCC/MinGW-w64 still cannot emulate msvc's COM calling convention behavior. Only clang can: llvm/llvm-project@85a0f8f#diff-48f9c7403ada1feaf517d77cd12f97c5bd239d6247190543d74b463c7da242ccR1058 (here you can see the ARM logic they use to detect an aggregate requiring a hidden return-by-paramete, which is much more complex than just "uses the COM calling convention".) @marler32 and @michal-z ran into this for example in microsoft/win32metadata#636 and thought the Microsoft header definitions may be wrong, unfortunately not - msvc just has undocumented behavior of injecting a hidden parameter. Microsoft actually cannot alter the header definitions to match C calling convention, as msvc would still inject the secret parameter. Me and @Andoryuuta found only one single semi-official reference to this undocumented behavior in msvc, as almost a footnote of this Raymond Chen article in 2014: https://devblogs.microsoft.com/oldnewthing/20040114-00/?p=41053
The harsh reality of all of this is that any Zig project which includes Microsoft headers and uses the default GNU ABI will compile just fine, but get a segfault when they invoke a COM method returning an aggregate as the hidden return parameter will be null. Additionally, it means any C++ project compiled with Zig or MinGW that does not explicitly define Failed approaches I investigatedThe headers distributed in win32metadata are not sufficient@andrewrk suggested we might be able to drop the GNU ABI support and only support msvc ABI. On that note, I checked to see if we can just use the The problem with this approach is that win32metadata is not complete: the headers included in this repository are generated by the Microsoft MIDL compiler distributed with Visual Studio and because of this they have a dependency on
Could we run the Wine WIDL compiler on Microsoft's IDL files?Another thought I had was: what if we could merely run Wine's WIDL compiler on Microsoft's IDL files? Effectively giving us super-up-to-date headers for MinGW-w64 that could theoretically even be used by MinGW and Wine themselves. I ran quite far with this experiment, and I believe it may be possible however I ran into two hurdles:
This was the approach I got furthest with. In fact, with relatively few patches I was able to use the WIDL compiler to compile most of the DirectX headers from Microsoft. I have not tested these yet, but I suspect they'll work and be compatible with MinGW/clang/zig with GNU ABIs using the Can we stick with wine/mingw-w64 headers and contribute to them?This is the path of least effort, and what we're currently doing. I will say however that after what I have found here.. I feel quite confident the Wine headers are currently, and I suspect will remain for quite some time, very outdated. Unfortunately, merging win32metadata IDL file changes into Wine's IDL files would be extremely non-trivial, too, because Wine's are hand-crafted much in the same way Microsoft's are - they differ quite a lot in syntax, ordering of definitions, and even which files symbols are defined in sometimes. Attempting to merge the two would be a seriously tedious undertaking. |
|
Thank you for this really, really helpful writeup.
Despite your apparent emotional defeat, my take on this is that you successfully pioneered the way forward, and I think the Zig project can carry the torch from here. 🔥 |
|
This looks like handcrafted, Wine-based IDL with latest interfaces (up to ID3D12Device9), maybe we could use this one? This is basically nice and clean IDL without Microsoft specific annotations and it is also up to date. I think we should run Wine WIDL compiler on this. |
|
@michal-z Ah, yes, I forgot to mention vk3d-proton. I did look into this as well. I do believe they are compatible with Wine's WIDL. However, they are actually quite stripped down and do not include D3D11 or prior, only good if you care about D3D12 (which in my case is a problem as Dawn also supports D3D11) It also doesn't solve the larger issue of mingw-w64 headers being quite out of date, I think that is worth solving. As for a temporary workaround: it's a good solution, also note I did manage to get the WIDL compiler to build the full latest d3d headers with minor patches - buried in my message above:
|
|
@slimsag Using d3d11.idl from Wine and latest d3d12.idl from Proton should work, right? This would give us headers to build Dawn I suppose. |
|
@michal-z The workaround I quoted above (should, I think / hope) give us headers to build Dawn, I just haven't tested it yet. But no, mixing the Wine and Proton headers does not generally work. Multiple reasons for this: (1) proton headers strip out non-d3d12 things from files shared between d3d12/d3d11, e.g. in dxgibase, d3dcommon, etc. (2) proton headers use Basically if you want to do that, you need to manually port/copy over the portions of Proton's headers that you want into the Wine headers which is non-trivial (but slightly easier than copying from Windows SDK IDL -> Wine IDL, you don't have to remove annotations etc.) |
|
Note the "secret first parameter" trick is officially documented here in the "x64 calling convention return values": https://docs.microsoft.com/en-us/cpp/build/x64-calling-convention?view=msvc-170#return-values
|
|
Extending from the x64 calling convention, there is 2 new rules in returning an struct / class from a method.
Now, this is news to me. It's clearly not documented on the msdn page on calling conventions. And I don't see any reason for these changes to the abi. I made a godbolt snippet where I explored the issue: https://godbolt.org/z/1xE6YnEnx
|
|
Agner Fog explains calling conventions in detail here: https://www.agner.org/optimize/calling_conventions.pdf All possible cases are described in above document. |
|
A lot great information there, but I don't see it describing returning a simple structure, class or union from a method. If I follow Table 7, I would expect returning a 4 byte struct from a method to do so via registers. But in my godbolt link the object is returned by pointer. See the |
I'd guess that nobody has exercised those corners of Wine on aarch64 yet. But also, I think in practice, the difference between being non-POD or C++14 aggregate only is visible in more corner cases (not exactly familiar with the details of what constitutes a C++14 aggregate though). If you look at the same Godbolt testcase as above, but for aarch64 targets, you get the same difference for the
I think this one is the correct one, but I don't know exactly in which cases the distinction differs. But a much bigger issue regarding arm/aarch64, is that Do you happen to have a minimal C++ standalone testcase that sets up a COM object and calls such a method, that I could try out for arm/aarch64 targets? |
Yup! I think Zig (in GNU mode only) and MinGW-w64 should both be defining this by default, as it stands out of the box programs will compile but then segfault.
Unfortunately not, what I have currently is quite entangled in a mess of other stuff. We will need to create such a minimal testcase from scratch. |
Seems you're right! I think I got confused here. |
Indeed. Not sure if Zig is defining this manually (or whether it should), as mingw-w64 headers already do define it automatically where they think it's relevant: https://github.com/mingw-w64/mingw-w64/blob/master/mingw-w64-headers/crt/_mingw_mac.h#L356-L361 |
|
That's strange; I am pretty confident I ran into this with Zig in GNU mode, compiling C++ on an x86_64 platform - maybe that header was not getting included for me somehow. I'll see if I can debug further. |
Hmm, I think most of the mingw-w64 provided headers would end up pulling in those base headers. I guess it's plausible that it was missed if testing something specific to the IDL generated headers though - or if Zig just does something different. (Or if I'm misreading the headers somehow, and that define isn't normally defined? A minimal buildable and runnable testcase would be awesome here.) |
|
To clarify Zig's end goal regarding libc, our goal is to provide static libc for Windows, with no dependency on msvcrt.dll and no dependency on UCRT either.
Edit: ah, I see that the mingw-w64 headers are intended to define this with preprocessor defines; let's explore sending a patch to mingw-w64 before merging this workaround. |
|
I think my memory may be wrong about encountering this with Zig in a GNU target, sorry. I'd assume for now there is no issue with the |
That's a quite nice goal in itself (and presumably worthwhile for static binaries on all platforms). Are you planning to provide standalone C headers for this libc then too, or just rely on the platform's existing headers (mingw-w64 or UCRT, musl, etc)? I presume the latter. For math functions, it should be quite straightforward I think. For things like printf etc, it's less straightforward though, to provide a cross-platform libc implementation. If potential future MSVC header compatibility is a target, the UCRT headers only have |
|
Ideally:
And then yep, we would just provide I also wouldn't be against maintaining a patchset against UCRT headers if there were some improvements that could be made. The benefits of such patches would be weighed against the maintenance burden. |
|
Hello all, I stumbled across this issue from it being posted on Hacker News. Another source of the Windows SDK that I've not seen mentioned in this thread is the You can download the package directly using the link on the right, These packages are unfortunately not permissively licensed so distribution might be problematic, but maybe Zig could just automatically download them as needed? If the main goal is to avoid having developers install Visual Studio or the Visual Studio Build Tools this is probably the closest you could get to an official solution. (Apologies if I've misunderstood the goals here. I've not actually even had a chance to use Zig yet so I definitely don't have a great idea of what its needs are 😅) |
I have to emphase this. The Windows SDK is very far from being MIT or redistributable; both headers and IDLs are copyrighted heavily. The Win32metadata gives a tool to generate, but if you look at the headers generated, they are not at all MIT. |
Very true. However, as @mstorsjo and I independently discovered, I have to politely but strongly disagree with mstorsjo here: I think it's very clear the files are MIT licenced, given the licence notice on the repo, but why draw unnecessary attention, in case corporate changes it mind? It's also worth noting Microsoft plans to (but has not yet) open source most of vcruntime/vcstartup (minus some proprietary intel intrinsics). This is as part of their open source STL project. Also, the the cppwinrt files are MIT licenced, in the actual Windows SDK itself. Together these facts mean we could, in theory, put together a large subset of the Windows SDK, that's free software and MSVC-ABI compatible. |
I don’t think this is correct. The main license of the repo might be MIT but the files are not. Examples:
The main project of win32metadata is the tooling to process the files, which is what the MIT is about.
It's very clear they are not. See "all right reserved" |
Keep in mind that it's totally possible to have a mixture of licenses within one repo (e.g. BSD/MIT/LGPL/whatever). And these files very specifically say "All rights reserved", not "distributable according to the toplevel license file". Given the unclear situation, I wouldn't start relying on this until the situation is clarified. And if it's a case where this is more or less accidental and they could realize their mistake and change their mind, then we definitely shouldn't rely on that. |
Good examples. But the question is, when there's a contradiction, what takes precedence?
I'll note “All rights reserved” is also used with the BSD licences. It doesn't mean there can't be a separate permissive licence.
On the other hand, if these files have been validly permissively licenced, that can't be undone. I should probably clarify: I appreciate the points yous are making. I'm not saying it's 100% clear cut, more that it's arguable. I can also see why you'd prefer something was 100% clear, not just arguable. I also wonder what legal counsel would say. Regardless, hopefully there is some positive clarity. |
|
In case of conflict, of course the information of the file prevails. This has always been the case in all Open Source distributions compliance. All rights reserved is pretty clear, not arguable…
Copyright and authorships are valid whatever the license header say. Those rights are unalienable, by default. Else I could take any file from anyone and slap a MIT header on it… |
andrewrk
changed the title
|
@89z |
|
@89z You're definitely not understanding what this thread is about:
MinGW provides both a GNU-based toolchain and an alternative SDK for building Windows software. Zig is using MinGW-w64, which represents only the latter. Those NuGet packages are the Microsoft-official equivalent of MinGW-w64. |
|
@89z No idea, at a glance it looks like they're using |
|
windows linker is provided by llvm (and soon self hosted compiler) not mingw |
|
FWIW mingw did some major headers update in the last year or so (possibly even surpassing the mach sdk) |
|
Here are what I know: For C:
For C++:
I compared the files from "open sourced UCRT part" and UCRT in Windows SDK:
So I think it is possible to use the open sourced UCRT with the redistributable to form a build environment. But I'm not familiar to the problems about ABI. |
|
However, I found that |
and others
Here's what MinGW provides for Zig:
(Let me know if I'm missing something here)
The problems I've been having with MinGW
Every software has bugs, but MinGW's main problem here is that it's difficult, time consuming and slow to fix them. I think if this wasn't the case, I would have no issue continuing to use MinGW and submitting patches.
So what can we do? A big chunk of work is going to be getting C/C++ headers for the win32 APIs. Because Microsoft has released the https://github.com/microsoft/win32metadata project, and we are already using it to generate Zig bindings, we can enhance that project to generate C/C++ headers as well. I anticipate the initial work to support this would be just a few weeks.
That leaves us with the 2nd chunk of work, an implementation of libc for Windows. This work is already planned (#2879). With this it may be feasible to drop MinGW with work that's already planned and leveraging existing tools.
Thoughts, concerns?
P.S. Long term goal here is to completely replace the need for Visual Studio using this proposal and #8973
The text was updated successfully, but these errors were encountered: