ComInterfaceGenerator Docs should be very clear about how return values / HResult on the managed method translates to the native signature

[jtschuster]

It can be very confusing that a return value in managed becomes an out parameter in native and a new int return value is added. Our documentation and announcements should be very clear about how these translate, and when to use [PreserveSig].

Tagging subscribers to this area: @dotnet/interop-contrib
See info in area-owners.md if you want to be subscribed.

Issue Details

---

It can be very confusing that a return value in managed becomes an out parameter in native and a new int return value is added. Our documentation and announcements should be very clear about how these translate, and when to use [PreserveSig].

Author:	jtschuster
Assignees:	jtschuster
Labels:	documentation, area-System.Runtime.InteropServices
Milestone:	8.0.0

[AaronRobinsonMSFT]

@jtschuster I assume dotnet/docs#36833 will handle this issue?

[jtschuster]

@jtschuster Jackson Schuster FTE I assume dotnet/docs#36833 will handle this issue?

Yes, the preserve-sig.md page should take care of this.

[MatejKafka]

I have a related question:

In #35855, the first example (result GetValue();) illustrates that if the generated out pointer parameter for the return value is NULL, the method correctly runs, the return value is ignored and S_OK is returned. Is this documented somewhere? If not, is this behavior defined, or should I not rely on it and instead use [PreserveSig] and marshal manually?

As an example, if I have the following interop declaration in C#:

int Method();

, which is (roughly) translated to

HRESULT Method(int* output);

and an unmanaged caller calls obj->Method(NULL), it is a defined behavior that the method correctly executes, returns S_OK and the return value is ignored?

[jtschuster]

@MatejKafka #35855 is related to built-in COM and that behavior is still the same (inconsistent behavior depending on how you define the method). The COM generator should throw a null reference exception in the marshalling code and return E_POINTER for all cases.

Is this documented somewhere? If not, is this behavior defined, or should I not rely on it and instead use [PreserveSig] and marshal manually?

I don't think it's documented anywhere besides the issue linked, and it isn't defined behavior. If you're using built-in COM and want to handle a null parameter, I would recommend [PreserveSig] and convert out parameters to pointers and marshal manually. It's worth noting, though, that passing null (or any invalid pointer) for a parameter defined in COM IDL as [out] is invalid use and isn't strictly required to be handled by the called method.

Also, closing this issue as the docs page has been published at https://learn.microsoft.com/dotnet/standard/native-interop/preserve-sig.

Feel free to follow up on your question here, though.

[MatejKafka]

@jtschuster My question is, is the inconsistent behavior for the built-in COM defined, or is it just an implementation detail that may change in future version of .NET?

EDIT: ah, sorry, missed the (originally quoted) part of the answer

[jtschuster]

@MatejKafka It's an implementation detail that may change.