From ae95bde64a92594b91409747235adfd3bb1b4f85 Mon Sep 17 00:00:00 2001 From: Dylan Perks <11160611+Perksey@users.noreply.github.com> Date: Thu, 12 Oct 2023 17:45:18 -0500 Subject: [PATCH 1/9] Update Proposal - Generation of Library Sources and PInvoke Mechanisms.md --- ... Library Sources and PInvoke Mechanisms.md | 381 +++++++++--------- 1 file changed, 183 insertions(+), 198 deletions(-) diff --git a/documentation/proposals/Proposal - Generation of Library Sources and PInvoke Mechanisms.md b/documentation/proposals/Proposal - Generation of Library Sources and PInvoke Mechanisms.md index 5393371f67..75b091cd43 100644 --- a/documentation/proposals/Proposal - Generation of Library Sources and PInvoke Mechanisms.md +++ b/documentation/proposals/Proposal - Generation of Library Sources and PInvoke Mechanisms.md @@ -6,283 +6,268 @@ Proposal design for a platform invoke (P/Invoke) mechanism for Silk.NET 3.0. - Kai Jellinghaus (@HurricanKai) # Current Status -- [x] Proposed -- [x] Discussed with Working Group (WG) -- [x] Approved +- [ ] Proposed +- [ ] Discussed with Working Group (WG) +- [ ] Approved - [ ] Implemented # Design Decisions - This proposal builds on the foundations laid out by Silk.NET's move to source generators in 2.0, and the introduction of the SilkTouch source generator as a result of this. - This proposal assumes no knowledge of Silk.NET 2.0's SilkTouch. - This takes the knowledge and insight gained during development of SilkTouch, and uses it to create a new set of generators which incorporate lessons learnt. -- This proposal breaks down the generator process into three distinct stages: +- Text herein marked **INFORMATIVE** does not form a normative part of this proposal, and is for background only. +- Within this proposal, the key words **must**, **required**, **shall**, **should**, **recommended**, **may**, **could**, and **optional** are to be interpreted as described in [RFC 2119 - Key words for use in RFCs to Indicate Requirement Levels](https://www.ietf.org/rfc/rfc2119.txt). The additional key word **optionally** is an alternate form of **optional**, for use where grammatically appropriate. These key words are highlighted in the proposal for clarity. -## SilkTouch Scraper +# INFORMATIVE: Points of Contention in 2.X -The Scraper is responsible for creating partial classes from some input source. It is a drop-in replacement for what BuildTools does today. Instead of doing all the parsing and interpretation of the input source ourselves, the proposed Scraper will instead use only C headers and have a "preburner" for gathering minimal metadata to feed into the generation process. +## API Objects -The generation process of the proposed Scraper will be entirely different. Silk.NET will no longer do any parsing and interpretation of C headers or XML of C headers, instead we will delegate this to the ClangSharp P/Invoke Generator in the form of a "subagent" (a separate process spun up by the Scraper), adding appropriate modifications to ClangSharp P/Invoke Generator as necessary. This means that we will no longer be using the Khronos XML registries for generating bindings. Instead, we'll use the preburner stage to use the XML registry only to gather minimal metadata instructing the ClangSharp subagent to add metadata attributes to certain functions, parameters, or types; which will then be picked up on by the later stages of SilkTouch. An example of such metadata would be parsing the flow and len XML attributes to add appropriate C# attributes to influence overload generation. +In Silk.NET 1.0 Previews 1-3, AdvancedDLSupport was used which used System.Reflection.Emit to generate function pointer calls at runtime. It did this by implementing abstract methods, and therefore an API object was required. In 1.0 Preview 4 Ultz's SuperInvoke library was used instead, which was an MIT-licensed clean-room implementation of AdvancedDLSupport produced in response to community confusion over the license grant that made Silk.NET users exempt from AdvancedDLSupport's LGPL license. It used the same abstract method mechanism. -This also naturally makes us entirely dependent on an external dependency, but I propose we work with Microsoft as much as possible to add the functionality we need in the least breaking way possible, and in a way that satisfies both us and Microsoft. All designs for such modifications will be formalized in the ClangSharp repo. Should we fail to do this, we'll maintain a fork so we can still benefit from improvements made upstream, while giving ourselves the freedom to add the functionality we need. +In Silk.NET 2.X, a source generator was used instead, which became SilkTouch. SilkTouch was developed when source generators and function pointers were still in heavy development and not even in preview yet, and as a result they were not well understood at time of development. In this respect SilkTouch was too bleeding-edge, as its usage of source generators have been realised to be less than ideal and somewhat abusive of what source generators were meant to do. This was first acknowledged by Kai, SilkTouch's creator, in his [November 2020 blog post](https://dotnet.github.io/Silk.NET/blog/nov-2020/silktouch-invokes-marshalling.html) on SilkTouch. In addition, because its development had completed and was effectively on maintenance mode by the time 2.0 was in full release, innovations that came after this point were rarely wielded, such as the incremental source generator API. -Microsoft have already stated that they're happy to work with us to get Silk.NET using ClangSharp, one maintainer even saying they're happy to add a CI test stage into the ClangSharp repo to ensure no incoming changes break Silk.NET's generation process. +One of the mistakes in hindsight that 2.X made was stick to the 1.X model of API objects, which was already disliked by a small subset of the Silk.NET community. There wasn't really any discussion around this decision, it was just naturally made during SilkTouch's initial tasking: "replace `abstract` with `partial`, and Bob's your uncle". This decision was felt the most when the Clang backend of BuildTools came online, and bindings to traditional C/C++ libraries that objectively should be static methods were instead using the same API object mechanism designed for the Khronos bindings. -There is no required behaviour for the Scraper (due to a lot of unknowns at the moment) other than it **MUST** invoke ClangSharp to generate C# Emitter-compatible classes, and it **MUST** add appropriate attributes to invoke the Overloader stage according to any metadata available from Khronos XML if applicable. +## BuildTools Portability -## SilkTouch Emitter -The Emitter, one of the two final stages whose resources **MUST** be entirely independent of eachother, is responsible for generating the actual indirect calls for performing the P/Invoke. +Before Silk.NET started development, Dylan Perks (@Perksey) and Jarl Gullberg (@Nihlus) were maintainers of the OpenTK project focusing on developing the OpenTK 4.0 rewrite project. The majority of their work focused around rewriting the various generators that OpenTK was using to generate its OpenGL bindings (namely [Generator.Bind](https://github.com/opentk/opentk/tree/70c36adba8ccad34f15584e75cabc0e0f5aebb2f/src/Generators/Generator.Bind) and [Generator.Convert](https://github.com/opentk/opentk/tree/70c36adba8ccad34f15584e75cabc0e0f5aebb2f/src/Generators/Generator.Convert)), although significant work on the [OpenAL bindings](https://github.com/opentk/opentk/tree/70c36adba8ccad34f15584e75cabc0e0f5aebb2f/src/OpenAL) was also done by Jarl to act as a demonstration of what the generator's bindings infrastructure should look like. Upon their departure from Silk.NET, OpenTK 4.0 was unwinded back to a form that looked more familiar with OpenTK 3.0, and Dylan founded Silk.NET and contacted Jarl for permission to include the OpenAL bindings verbatim into Silk.NET - to this day these bindings remain for the most part unchanged from their original form in their version of OpenTK 4.0. In addition, Dylan brought forward the rewritten generators, and merged them into a single generator called BuildTools. -The Emitter operates on partial methods, the behaviour of the implementations of which depending on the context in which it's used. +This history is important to illustrate just how intertwined OpenGL and BuildTools were, and how easily issues arose when future tasking let to more advanced bindings generation mechanisms being retro-fitted into a bindings model that fundamentally was not built for it. BuildTools has multiple backends, but it still all boils down to the same object model and generation mechanisms, which did not prove extensible especially for the later Clang-based bindings (e.g. the COM-based Windows SDK). This, combined with the the API object concept, meant using these bindings felt completely foreign to use compared to other alternatives such as the .NET Foundation's TerraFX library. -All attributes **MUST** be name matched only, to allow the user to define these themselves and not create a hard dependency on a specific Silk.NET library such as the Silk.NET Core. +## Excessive Overloads -Candidate methods for implementation by the Emitter **MUST** be partial and not have an implementation part yet. Their containing types **MUST** also be partial. +One of the areas of BuildTools that has gone through the largest number of iterations is the overloader. OpenTK was (in)famous for the concessions it made in API accuracy for the sake of usability. For instance, `IntPtr` often was given an `int` overload, for CLS compliance unsigned numbers were replaced with signed ones, and there were a slew of other overloads which made OpenTK very usable but did not match the 1:1 ethos of Silk.NET. Because of this, the overloader was rewritten to provide some overloads of similar usability as well as some original overloads too. -The Emitter **MUST** be able to be invoked via the SilkTouch CLI and **MAY** be able to be invoked via an incremental Roslyn source generator. +1.0 Preview 1 was shipped with array (including generic arrays) and ref overloads, along with function-transforming overloads such as the overloaders that handled `glDelete`, `glGen`, etc. The later previews and releases expanded on this. However, a consistent item of feedback throughout the entirety of 1.X's lifecycle is that the overloads did not feel sufficient or were inconsistently applied (e.g. if you had one parameter using an overloaded variant, the other parameters would not be overloaded). This resulted in a lot of tradeoffs in user code in where to use `unsafe` code, and most users just ended up using the unsafe variants always. -### Call Styles +This was [attempted to be addressed](https://github.com/dotnet/Silk.NET/pull/275) in 2.0 Preview 2 (back when the only Clang-based bindings were SDL and Assimp), and for the most part our users are a lot more satisfied with the overload situation in 2.X from a code-cleanliness perspective. The new overloader would generate every single permutation of overloaded parameters to allay complaints of inconsistency from users. However, this had another unintended side-effect in that the IDE experience was now terrible due to the shear quantity of overloads produced. In the early days of Silk.NET 2.0's development cycle this was acceptable, and the Silk.NET developers (both on the Silk.NET team and in the community) clearly grew comfortable enough with it such that this mechanism was never questioned again before Silk.NET 2.0's initial release. -The Emitter's primary purpose is to load and use function pointers in a native library sourced from an operating system's kernel, though this doesn't necessarily have to be the case. This logic will be emitted by the Emitter itself, and will not require an external dependency like the Silk.NET Core. However, this logic will no longer be implicit. +Later on in 2.X's cycle, this mechanism became more and more called into question, given that as more overloaders were added the problem was compounded. The most famous examples came when we were adding new bindings with all the overloads in place, such as Direct2D which had a 16-parameter function that resulted in 65565 overloads being generated. -#### Built-in: Dynamic-Link Library Call Style +# Previous Iterations of This Proposal -Consider the following example: -```cs -[UseDynamicLibrary("glfw3.dll")] -public partial class Glfw -{ - public partial void glfwInit(); -} -``` +The general theme behind these points of contention is the generation mechanisms and infrastructure put in place for Silk.NET were very much designed for OpenGL and other similar Khronos APIs, with the later non-Khronos bindings not being specifically designed for, instead the tasking was "how can we fit these other non-Khronos bindings to fit our existing bindings infrastructure built for OpenGL?". -The `UseDynamicLibrary` attribute instructs the Emitter that it **MUST** use [`DllImport`](https://docs.microsoft.com/en-us/dotnet/api/system.runtime.interopservices.dllimportattribute?view=net-5.0) to access native functions. [`NativeLibrary`](https://docs.microsoft.com/en-us/dotnet/api/system.runtime.interopservices.nativelibrary?view=net-5.0) can be used to modify the libsuperrary loading process per other requirements defined below. For the entry point, the function name **MUST** be used unless a `NativeApi` attribute is provided, in which case the `EntryPoint` indicated by that attribute **MUST** be used. +The Working Group approved an earlier version of this proposal on 25th February 2022, whereby a hybrid source generator/AOT generator solution was used. The solution looked and felt a lot like Silk.NET 2.X, and had a similar overloading mechanism (just different mechanisms of invocation). However, given that Silk.NET 2.X was again built without these points of contentions in mind and without non-Khronos bindings in mind, we ran the risk of running into similar issues. -Consider the following example: -```cs -[UseDynamicLibrary("glfw3.dll", "libglfw3.dylib")] -public partial class Glfw { /* ... */ } -``` +However, the approved version of this proposal has a lot of great points as well. One of the most pertinent points in the approved proposal is the ability to have the overloader applied to user-defined and/or non-generated functions. Another pertinent point was the ability to have a very stable foundation for the native signatures, as Silk.NET 2.X underwent a lot of massively-breaking updates due to the signatures/types already present in the shipped version of the library were fundamentally wrong due to generator error. -`UseDynamicLibrary` **MUST** be able to be specified on either a function or type. +Since the creation of that proposal, the [SilkX](https://github.com/Perksey/SilkX) project (led by Dylan Perks/@Perksey) has been pioneering the underlying concepts of the approved proposal but using mechanisms that provide the least contention for the generator users (primarily the Silk.NET maintainers). This proposal is based on the conclusion of that project, and if this proposal is approved SilkX shall be merged into the main Silk.NET repo and become the official Silk.NET 3.0 implementation. -The Emitter **MUST** allow multiple candidate library names and cycle through each candidate until one loads successfully. +The Working Group should note that Kai did make some progress with the original implementation of Silk.NET 3.0 as approved by the Working Group, however the SilkX project was spawned from the realisation that the proposed solution was too large in scope given the time constraints of the primary Silk.NET developers. This work still lives in the develop/3.0 branch until this proposal is approved. -#### Built-in: Static-Link Library Call Style +# ClangSharp Generation -Consider the following example: -```cs -[UseStaticLibrary] -public partial class Glfw -{ - public partial void glfwInit(); -} -``` +As in the previously approved version of this proposal, the generation process of the proposed generator will be entirely different to BuildTools. Silk.NET will no longer do any parsing and interpretation of C headers or XML of C headers, instead we will delegate this to the ClangSharp P/Invoke Generator library, adding appropriate modifications to inputs/outputs as necessary. This means that we will no longer be using the Khronos XML registries for generating bindings directly. -The `UseStaticLibrary` attribute instructs the Emitter that it **MUST** use [`DllImport`](https://docs.microsoft.com/en-us/dotnet/api/system.runtime.interopservices.dllimportattribute?view=net-5.0) to access native functions. `__Internal` **MUST** be used for the library name. For the entry point, the function name **MUST** be used unless a `NativeApi` attribute is provided, in which case the `EntryPoint` indicated by that attribute **MUST** be used. +Instead, we'll use a "mod" system to use the XML registry and other metadata sources to influence the various stages of the generator cycle. Each mod is initialised (which in turn may do HTTP requests, XML parsing, etc) and **shall** be given the ClangSharp input response files. Once the modifications (if any) have been applied to those inputs, ClangSharp **shall** be run with each of those inputs, and the C# syntax trees saved. Those syntax trees **shall** then be fed back into the mod, and the syntax modifications (if any) will be applied. Finally, the syntax trees **shall** be added to the MSBuild workspace where the mods **shall** have one final opportunity to enact workspace-wide modifications. The details of this system is implementation-defined. -`UseStaticLibrary` **MUST** be able to be specified on either a function or type. +This naturally makes us entirely dependent on an external dependency, but I propose we work with Tanner Gooding (the ClangSharp project lead) as much as possible to add the functionality we need in the least breaking way possible, and in a way that satisfies both us and ClangSharp. All designs for such modifications will be formalized in the ClangSharp repo. Should we fail to do this, we'll maintain a fork so we can still benefit from improvements made upstream, while giving ourselves the freedom to add the functionality we need. -Consider the following example: -```cs -#if __IOS__ -[UseStaticLibrary] -#endif -[UseDynamicLibrary("glfw3.dll")] -public partial class Glfw -{ - public partial void glfwInit(); -} -``` +Tanner has already stated that they're happy to work with us to get Silk.NET using ClangSharp, even saying they're happy to add a CI test stage into the ClangSharp repo to ensure no incoming changes break Silk.NET's generation process. -The Emitter **MUST** only generate code if all preprocessor directives guarding the `UseStaticLibrary` attribute evaluate to true. If the attribute is defined on both the function and the containing type, the preprocessor directives surrounding the function's attribute **MUST** be used instead of the preprocessor directives surrounding the type. +# Implicit Cast-Based Overloading -Note to the reader: Given preprocessor directives are processed at parse time in Roslyn, both of those last requirements are basically benign. +The pivotal proposal around which the SilkX experiment revolves is solving the excessive overload problem by having no overloads at all. Namely, Silk.NET.Core will have a set of "pointer-like" types that can be implicitly casted to by a multitude of different types. This gives the user the same flexibility as having all different permutations of overloaded parameters generated without having to do so, and thereby alleviates the requirement of having overloads at all. -#### Custom Call Style: Procedure Address Expressions +`Ptr` and `Mut` `ref struct`s **shall** be defined wrapping a `ref` and `ref readonly` respectively. These types **shall** have generic and non-generic variants. This shall be repeated up to 3 pointer dimensions, taking into account C's mutability rules in each instantiation. This means that there will be 14 top-level generic structs and 14 top-level non-generic structs. These **should** be auto-generated to ensure they remain maintainable. IL weaving **may** be required for the higher dimensions. -Custom code may be used as a call style by providing a pointer to SilkTouch using a Procedure Address Expression. This is useful in scenarios such as COM interop. +Unless deemed inappropriate, inapplicable, and/or infeasible by the Silk.NET team (the team reserved the right to do so without Working Group approval), the types **should** have the following characteristics: +- An instance **must** be constructable from a `ref T` or `ref readonly T` (depending on the outer dimension mutability) where `T` is either the generic type (or `byte` for non-generic variants), a "pointer-like" type of the inner dimension of type being constructed. +- An indexer **must** be present accepting a `nuint` for the index, returning a `ref T` or `ref readonly T` (depending on the outer dimension mutability) where `T` is either the generic type (or `byte` for non-generic variants), a "pointer-like" type of the inner dimension of type being indexed. +- A `GetPinnableReference` **must** be present taking no parameters and returning a `ref T` or `ref readonly T` (depending on the outer dimension mutability) where `T` is the native raw pointer representation of the inner dimension. +- For single-dimension pointers, an `AsSpan` method **must** be present taking an `int` length argument (due to historical reasons in the .NET BCL) returning a `Span` or `ReadOnlySpan` (depending on the outer dimension mutability) where `T` the pointee type. For non-generic pointer-like types, a generic argument **may** be used to specify the type of the span. +- An array of the same jagged dimension **must** be implicitly convertible to the type. +- For single-dimension pointers, a multi-dimensional array (up to 3 dimensions) **must** be implicitly convertible to the type. +- A raw pointer of the same dimensions **must** be implicitly convertible to the type. +- `NullPtr` **must** be implicitly convertible to the type, and the returned pointer must represent a null reference. +- An explicit operator **must** be present to unsafely convert the reference represented by the pointer to a raw pointer without the user using `fixed`/`GetPinnableReference`. +- An explicit operator **must** be present to unsafely convert the reference represented by the pointer to a void pointer without the user using `fixed`/`GetPinnableReference`. +- For generic pointer types, an implicit operator **must** be present to convert the pointer type to a `string` (or an array of strings of the inner dimension's jaggedness i.e. `PtrPtrPtr` becomes `string[][]`). Much like `ReadOnlySpan`, `ToString` **must** call this implicitly if the type is a single-dimension pointer to a string pointee type. The implicit cast **may** throw an exception if the pointee type is not a string pointee type. This is because we can't constrain the type used on implicit operators on generic types. +- An `==` and `!=` operator **must** be present to check equality with another pointer of the same type. +- An `==` and `!=` operator **must** be present to check equality with `NullPtr` i.e. check whether the pointer is null. +- For generic pointer types to a string pointee type, where the inner-most dimension is immutable, a `string` (or an array of strings of the inner dimension's jaggedness i.e. `PtrPtrPtr` becomes `string[][]`) **must** be implicitly convertible to the pointer type. The implicit cast **may** throw an exception if the pointee type is not a string pointee type. This is because we can't constrain the type used on implicit operators on generic types. +- For single dimension generic pointer types, `Span` **must** be implicitly convertible to the pointer type. +- For single dimension generic immutable pointer types, `ReadOnlySpan` **must** be implicitly convertible to the pointer type. -Procedure Address Expressions are C# expressions that **MUST** evaluate to a `void*`, `nint`, or `IntPtr`. This is the actual address in memory of the function being invoked. +Our goal with this implicitness it to make the experience of using pointers as similar to C as possible without feeling completely alien in a high-level language. This is the logic behind the string casting as well. The public API surface for our bindings **shall** use these wrapper types. -Consider the following example: -```cs -public partial struct IUnknown -{ - public void** LpVtbl; - [UseExpression("LpVtbl[1]")] - public partial uint AddRef(); -} -``` +For the avoidance of doubt, a string pointee type **shall** be defined as one of: `byte`, `sbyte`, `char`, `short`, `ushort`, `int`, `uint`. For single-byte string pointee types, UTF-8 **shall** be used. For 2-byte string pointee types, UTF-16 **shall** be used. For 4-byte string pointee types, UTF-32 **shall** be used. -`GetProcAddress` indicates that the C# code given **MUST** be used as the Procedure Address Expression to retrieve the function address to call. Any arbritrary code can be inserted into this attribute, so long as the result of the code once evaluated meets the Procedure Address Expression definition. For example, this is valid: -```cs -public partial struct IUnknownNullableContainer -{ - public IUnknownPtr? Value; - [UseExpression("Value.GetValueOrDefault().InnerValue->LpVtbl[1]")] - public partial uint AddRef(); -} +**INFORMATIVE:** This was discussed informally by the Working Group in a Coffee & Code Catchup. The recording for this catchup can be found on the .NET Foundation YouTube channel [here](https://www.youtube.com/watch?v=N7qcETE4X_I). The most pertinent points were regarding ease of use and discoverability - unless the user reads the documentation (which will exist per the Working Group approved Software Development Plan), they won't know what all of these `Ptr` types mean and how to use them. It is our goal to make this as easy to understand as possible, with the goal being the ability to read one short page and instantly know how to use the majority of the library, but this was nonetheless a concern from the Working Group. There was also some desire to revisit the 1.X style of overloading, but the Silk.NET team were hesitant to do so to avoid history repeating itself. This is not an invalid suggestion however, given that TerraFX exists and can be treated as an equal now due to .NET Foundation membership, and their focus is exclusively unsafe so users that want unsafe can just use TerraFX. However, Silk.NET still wants to be as flexible as possible thus we have persisted in this model for this proposal. -public struct IUnknownPtr -{ - public IUnknown* InnerValue; -} +**INFORMATIVE:** The Working Group previously expressed concerns for implicit casting for the trivial case of `string` to `ReadOnlySpan` in the 2021 meeting regarding the previous version of this proposal (see Meeting Notes). + +# API Objects & Static Methods + +One of the most common complaints throughout the entirety of Silk.NET's lifetime was the lack of static functions. We have argued that this is required for the sake of validity, and this is completely true *for the Khronos bindings*. This should have never been the case for the non-Khronos bindings. However, there are some users that do indeed prefer the API object mechanism and it does provide undeniable flexibility and control versus the alternative of static state that often does not lend itself to multi-context and/or multi-backend solutions. We still want to keep that flexibility, however we recognise that not all users need it. -public struct IUnknown +This is why the solution proposed includes both static functions and API objects, with one being a wrapper over the other where most appropriate. + +Each binding **shall** have a "V-Table" interface generated like so: + +```cs +public interface IMyStringLibrary { - public void** LpVtbl; + public interface Static + { + static abstract byte* ToLower(byte* str); + [MethodImpl(MethodImplOptions.AggressiveInlining)] + static virtual Ptr ToLower(Ptr str) + { + fixed (byte* nStr = str) + { + return ToLower(nStr); + } + } + static abstract void FreeResult(byte* str); + [MethodImpl(MethodImplOptions.AggressiveInlining)] + static virtual FreeResult(Ptr str) + { + fixed (byte* nStr = str) + { + FreeResult(nStr); + } + } + } + byte* ToLower(byte* str); + Ptr ToLower(Ptr str) + { + fixed (byte* nStr = str) + { + return ToLower(nStr); + } + } + void FreeResult(byte* str); + void FreeResult(Ptr str) + { + fixed (byte* nStr = str) + { + FreeResult(nStr); + } + } } ``` -The Emitter **SHOULD** implicitly parenthesise the expression given in the attribute. +The Silk.NET team reserves the right to define the behaviour for when the `static abstract` and `static virtual` or DIM and non-DIM conflict by return types only. This will be worked out during development, but **may** require removing the native representations and duplicating the marshalling logic for each of the implementations. + +Each library being bound to **shall** have an interface generated, named using the class name provided by the generator user prefixed with `I`. + +This interface **shall** contain a subinterface named `Static`. -Unless another call style is applicable, the Emitter **MUST** mandate that every function has a `UseExpression` (Procedure Address Expression) specified. +The `Static` subinterface **shall** contain a `static abstract` or `static virtual` (depending on the conflict outcome above) function representing the native signature using the wrapper types for each function exposed by the library. -The Emitter **MUST** call the function pointer returned by the Procedure Address Expression as part of this call style. +The `Static` subinterface **should** contain a `static abstract` function representing the native, raw, and blittable signature for each function exposed by the library. -#### Custom Call Style: Procedure Address Methods +The top-level interface **shall** contain a function representing the native signature using the wrapper types for each function exposed by the library. This **should** be implemented using a Default Interface Method. -A level more abstracted than Procedure Address Expressions, which allows any custom code to retrieve a function pointer; Procedure Address Methods work similarly to the native library call style from an API perspective, but function similarly to the Procedure Address Expressions call style. +The top-level interface **should** contain a function representing the native, raw, and blittable signature for each function exposed by the library. -The aim of this call style is to provide flexibility without comprimising code readability. Consider the following example: +This contains both a static and a non-static variant. An implementation **shall** be created as follows: ```cs -[UseMethod(nameof(GetProcAddressShim))] -public partial class Glfw +public class MyStringLibrary : IMyStringLibrary { - [UseDynamicLibrary("glfw3.dll")] - public partial nint glfwGetProcAddress(byte* str); + public static class DllImport : IMyStringLibrary.Static + { + [DllImportAttribute("mystringlib")] + public static extern byte* ToLower(byte* str); + [DllImportAttribute("mystringlib")] + public static extern void FreeResult(byte* str); + } + public class StaticWrapper : IMyStringLibrary where T: IMyStringLibrary.Static + { + public StaticWrapper(); + public byte* ToLower(byte* str) => T.ToLower(str); + public void FreeResult(byte* str) => T.FreeResult(str); + } + public class ThreadLocal : IMyStringLibrary.Static + { + private static ThreadLocal _current = new(); + public static void MakeCurrent(IMyStringLibrary current) => _current.Value = current; + public static byte* ToLower(byte* str) => _current.Value.ToLower(str); + public static void FreeResult(byte* str) => _current.Value.FreeResult(str); + public static Ptr ToLower(Ptr str) => _current.Value.ToLower(str); + public static void FreeResult(Ptr str) => _current.Value.FreeResult(str); + } + + // Non-Static Interface + private Func _getProcAddress; + byte* IMyStringLibrary.ToLower(byte* str) + { + var ptr = _getProcAddress("ToLower"); + if (ptr is 0) throw new("some symbol loading exception..."); + return ((delegate* unmanaged)ptr)(str); + } + + void IMyStringLibrary.FreeResult(byte* str) + { + var ptr = _getProcAddress("FreeResult"); + if (ptr is 0) throw new("some symbol loading exception..."); + return ((delegate* unmanaged)ptr)(str); + } - // shim to convert the string, which SilkTouch needs to use, to a byte pointer - THIS IS NOT A MODEL EXAMPLE! - private nint GetProcAddressShim(string str) => glfwGetProcAddress((byte*) Marshal.StringToHGlobalAnsi(str)); + public static IMyStringLibrary Create() => new StaticWrapper(); + public static IMyStringLibrary Create(Func getProcAddress) => new MyStringLibrary { _getProcAddress = getProcAddress }; - public partial void glBegin(uint mode); + // Static Interface + public static byte* ToLower(byte* str) => DllImport.ToLower(str); + public static Ptr ToLower(Ptr str) => DllImport.ToLower(str); + public static void FreeResult(byte* str) => DllImport.FreeResult(str); + public static void FreeResult(Ptr str) => DllImport.FreeResult(str); } ``` -Procedure Address Methods are method groups (or an otherwise callable expression) within the scope of the method that **MUST** return a `void*`, `nint`, or `IntPtr` when invoked. This is the actual address in memory of the function being invoked. +**INFORMATIVE:** `getProcAddress` delegates replace "native contexts". -Procedure Address Methods **MUST** take one parameter of type `string`. +There exist requirements for all of the following: +- A native function retrieved using a thread-specific "native context" can be called using a static function (for OpenGL) +- A native function retrieved using a custom "native context" delegate can be called using a static function (for Vulkan, OpenCL, OpenXR). +- A native function retrieved using the platform-default mechanism can be called using a static function (for literally everything else). -For the parameter passed into the callable specified in the attribute, the function name **MUST** be used unless the `EntryPoint` property in the `NativeApi` attribute is provided, in which case the `EntryPoint` indicated by the attribute **MUST** be used. +**INFORMATIVE:** `Create` replaces `GetApi`. -The Emitter **MUST** call the function pointer returned by the Procedure Address Method as part of this call style. +While this somewhat balloons the API surface, this provides the most flexibility and most entry points into the bindings without adding lots of API-specific code into SilkTouch. -#### Call Style Priority +Each bindings **shall** have a class generated to match the generated interface. The name of this class **shall** be the same as the interface without the leading `I` prefix. -Consider the following example: +Within each binding class a `static class` **shall** be generated for each `DllImport` look-up name provided by the generator user. If there are multiple, this class **shall** carry the pascal case version of the look-up name provided. If there is only one, this class **shall** be named `DllImport`. -```cs -[UseDynamicLibrary("glfw3.dll")] -public partial class Glfw -{ - public partial nint glfwGetProcAddress(byte* str); - [UseExpression("glfwGetProcAddress((byte*)Unsafe.AsPointer(ref Unsafe.AsRef(0x006e696765426c67)))")] - public partial void glBegin(uint mode); -} -``` +**INFORMATIVE:** This does not mean platform-specific name, we still have the NativeLibrary callbacks after all. This is referring to OpenAL Soft vs OpenAL for example, though even that could be implemented using NativeLibrary. -Here, a class using the Dynamic Library call style has a method which does not follow the call style defined at the class level, and is overridden using a `UseExpression` attribute. +Within each binding class a `class` **shall** be generated implementing the top-level interface over a generic type parameter implementing the static subinterface. This allows users to construct API objects over `DllImport`ed libraries. This **shall** be named `StaticWrapper`. -If multiple call styles are applicable, the following order of preference **MUST** be respected: -- Procedure Address Expressions -- Procedure Address Methods -- Static-Link Library -- Dynamic-Link Library +Within each binding class a `static class` **shall** be generated implementing the static subinterface using a `ThreadLocal` containing an instance of the top-level interface. This allows users to call stateful native libraries (like OpenGL) using a static function. -Function-level attributes **MUST** be preferred over type-level ones, and follow the same order of preference. +`ThreadLocal` implementations **shall** implement a `MakeCurrent` method taking an instance of the top-level interface as the parameter. On the thread on which this method is called, all subsequent static method calls on `ThreadLocal` will use the given API object. -### Native Calls -For the most part, the resultant native signature used by the Emitter is matched 1:1 with the method signature. However there are certain modifications you can apply. Namely, the `NativeApi` attribute will allow specification of specific calling conventions. For example: +The binding class **shall** expose a `static` `Create` method with no parameters returning an instance of the top-level interface. This **should** use `StaticWrapper` where `T` is the generator user's configured *static default*. -```cs -[NativeApi(Conventions = new[]{typeof(CallConvMemberFunction), typeof(CallConvSuppressGCTransition)}] -public partial D3D12_HEAP_PROPERTIES GetCustomHeapProperties(uint nodeMask, D3D12_HEAP_TYPE heapType); -``` +**INFORMATIVE:** The Silk.NET team looks back very fondly on the API-as-interfaces scheme present in only the earliest Silk.NET 1.0 previews. The backstory is that AdvancedDLSupport originally required interfaces in order to implement the abstract class, so we added interfaces just for that, but they were removed once they became unnecessary. However, these interfaces are obviously very useful in the advent of C# having advanced mainstream dependency injection. While this is never an explicit target for a low-level, high-speed interoperability library such as Silk.NET, we take pleasure in being able to cater for this use case. +Unless `ThreadLocal` is itself the *static default* configured by the generator user, the thread-local value within `ThreadLocal` should be instantiated with a value factory calling the `Create` method. If `ThreadLocal` is the *static default*, then no value factory is provided and the user must set the value using `MakeCurrent`. -`Conventions` will be used as the primary mechanism for customizing the behaviour of generation, just as `NativeApi` will be used as the primary attribute for this as well. The behaviour of each bit will be described in documentation comments in the Proposed API section. +The binding class **shall** expose a `static` `Create` method with a `Func` parameter returning an instance of the top-level interface. This **shall** use an instance of the binding class itself. -The Emitter does not do any marshalling. As such, the Emitter **MUST** mandate that every parameter and return type of every function fits the `unmanaged` constraint. For the readers benefit, this can be done using a property on `ITypeSymbol` in Roslyn. +**INFORMATIVE:** It is undecided whether we want SilkTouch to output the `Create` methods itself or whether we want it in a non-generated partial. -## SilkTouch Overloader -The Overloader, one of the two final stages whose resources **MUST** be entirely independent of eachother, creates overloads of functions that expose a more user-friendly interface than the function it overloads, and do appropriate marshalling to lower the parameter types used down to the original function's types. +The binding class **shall** itself implement the top-level interface, using the `getProcAddress` delegate stored from the `Create` method to implement the native calls using a direct function pointer call. -The Overloader **MUST** be able to be used on any function and not be tied to any of the Emitter's constraints. +The binding class **shall** also contain shorthand functions for calling the static functions contained within the static subinterface using the *static default*. -The Overloader **MUST** be able to be invoked via the SilkTouch CLI and **MAY** be able to be invoked via an incremental Roslyn source generator. +**INFORMATIVE:** It is undecided whether we want to implement the static subinterface on the binding class itself as well. -The Overloader does not care about existing methods. If the Overloader generates an overload that also happens to exist manually, it is the user's repsonsibility to disable the relevant overloads for these cases. +**INFORMATIVE:** Do we want to make a MakeCurrent shorthand as well? This may cause confusion for OpenGL users because it doesn't make the underlying context (WGL/EGL/GLX) current, it just makes it the current source of function pointers. -However, if the Overloader thinks that the overload it's generating may conflict with another overload or the original function, it **SHOULD** output the overload as an extension method rather than a method within the containing type, unless the original method is static in which case it **MUST** discard the overload and generate a warning. It **SHOULD** also do this if the containing type is not partial. +**INFORMATIVE:** The `DllImport` and `StaticWrapper` names need bikeshedding if performance-aware users are expected to use them directly. Perhaps `DllImport` could be called `Exports`? -The Silk.NET team does not wish to specify the functionality of the overloader at this time, and wishes to instead define this by experimenting with the overloader's functionality during development; with the understanding that the Silk.NET team must formalize a proposal with the working group before a "go live" release ships. +# Khronos Extension Handling -# Proposed API -- Here you do some code blocks, this is the heart and soul of the proposal. DON'T DO ANY IMPLEMENTATIONS! Just declarations. +**TODO:** Once the preceding document contents have been implemented in SilkX, describe how we're going to handle extensions. I personally think we should just do a "hint" rather than anything hard, like `SupportedOSPlatform`. -## `UseDynamicLibrary` -```cs -namespace Silk.NET.Core -{ - [AttributeUsage(AttributeTargets.Function | AttributeTargets.Class)] - public class UseDynamicLibraryAttribute : Attribute - { - public UseDynamicLibrary(string libraryName, params string[] alternativeNames); - public string LibraryName { get; } - public string[] AlternativeNames { get; } - } -} -``` +# Safety in Structs -## `UseStaticLibrary` -```cs -namespace Silk.NET.Core -{ - [AttributeUsage(AttributeTargets.Function | AttributeTargets.Class)] - public class UseStaticLibraryAttribute : Attribute - { - } -} -``` - -## `UseExpression` -```cs -namespace Silk.NET.Core -{ - [AttributeUsage(AttributeTargets.Function | AttributeTargets.Class)] - public class UseExpressionAttribute : Attribute - { - public UseExpressionAttribute(string expr); - public string Expression { get; } - } -} -``` - -## `UseMethod` -```cs -namespace Silk.NET.Core -{ - [AttributeUsage(AttributeTargets.Function | AttributeTargets.Class)] - public class UseMethodAttribute : Attribute - { - public UseMethodAttribute(string expr); - public string Expression { get; } - } -} -``` - -## `NativeApi` -```cs -namespace Silk.NET.Core -{ - public class NativeApiAttribute : Attribute - { - public string EntryPoint { get; set; } - public Type[] Conventions { get; set; } - } -} -``` +**TODO:** Once the preceding document contents have been implemented in SilkX, describe the impact the wrapper types have on structs, which are currently uncatered for and always lacking overloads. @Perksey already has ideas here though. # Meeting Notes From 642c5e33d634f3b3973feb61fb56022f744cf674 Mon Sep 17 00:00:00 2001 From: Dylan Perks <11160611+Perksey@users.noreply.github.com> Date: Sun, 15 Oct 2023 18:32:03 -0500 Subject: [PATCH 2/9] Update documentation/proposals/Proposal - Generation of Library Sources and PInvoke Mechanisms.md --- ...al - Generation of Library Sources and PInvoke Mechanisms.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/proposals/Proposal - Generation of Library Sources and PInvoke Mechanisms.md b/documentation/proposals/Proposal - Generation of Library Sources and PInvoke Mechanisms.md index 75b091cd43..75703e53c6 100644 --- a/documentation/proposals/Proposal - Generation of Library Sources and PInvoke Mechanisms.md +++ b/documentation/proposals/Proposal - Generation of Library Sources and PInvoke Mechanisms.md @@ -70,7 +70,7 @@ Tanner has already stated that they're happy to work with us to get Silk.NET usi The pivotal proposal around which the SilkX experiment revolves is solving the excessive overload problem by having no overloads at all. Namely, Silk.NET.Core will have a set of "pointer-like" types that can be implicitly casted to by a multitude of different types. This gives the user the same flexibility as having all different permutations of overloaded parameters generated without having to do so, and thereby alleviates the requirement of having overloads at all. -`Ptr` and `Mut` `ref struct`s **shall** be defined wrapping a `ref` and `ref readonly` respectively. These types **shall** have generic and non-generic variants. This shall be repeated up to 3 pointer dimensions, taking into account C's mutability rules in each instantiation. This means that there will be 14 top-level generic structs and 14 top-level non-generic structs. These **should** be auto-generated to ensure they remain maintainable. IL weaving **may** be required for the higher dimensions. +`Ptr` and `Mut` `ref struct`s **shall** be defined wrapping a `ref readonly` and `ref` respectively. These types **shall** have generic and non-generic variants. This shall be repeated up to 3 pointer dimensions, taking into account C's mutability rules in each instantiation. This means that there will be 14 top-level generic structs and 14 top-level non-generic structs. These **should** be auto-generated to ensure they remain maintainable. IL weaving **may** be required for the higher dimensions. Unless deemed inappropriate, inapplicable, and/or infeasible by the Silk.NET team (the team reserved the right to do so without Working Group approval), the types **should** have the following characteristics: - An instance **must** be constructable from a `ref T` or `ref readonly T` (depending on the outer dimension mutability) where `T` is either the generic type (or `byte` for non-generic variants), a "pointer-like" type of the inner dimension of type being constructed. From b68a9afc4051c1e9c8ea15f0987bd0a7c6c16ddc Mon Sep 17 00:00:00 2001 From: Dylan Perks <11160611+Perksey@users.noreply.github.com> Date: Sun, 15 Oct 2023 18:33:55 -0500 Subject: [PATCH 3/9] Update documentation/proposals/Proposal - Generation of Library Sources and PInvoke Mechanisms.md --- ...al - Generation of Library Sources and PInvoke Mechanisms.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/proposals/Proposal - Generation of Library Sources and PInvoke Mechanisms.md b/documentation/proposals/Proposal - Generation of Library Sources and PInvoke Mechanisms.md index 75703e53c6..7e5beea05e 100644 --- a/documentation/proposals/Proposal - Generation of Library Sources and PInvoke Mechanisms.md +++ b/documentation/proposals/Proposal - Generation of Library Sources and PInvoke Mechanisms.md @@ -74,7 +74,7 @@ The pivotal proposal around which the SilkX experiment revolves is solving the e Unless deemed inappropriate, inapplicable, and/or infeasible by the Silk.NET team (the team reserved the right to do so without Working Group approval), the types **should** have the following characteristics: - An instance **must** be constructable from a `ref T` or `ref readonly T` (depending on the outer dimension mutability) where `T` is either the generic type (or `byte` for non-generic variants), a "pointer-like" type of the inner dimension of type being constructed. -- An indexer **must** be present accepting a `nuint` for the index, returning a `ref T` or `ref readonly T` (depending on the outer dimension mutability) where `T` is either the generic type (or `byte` for non-generic variants), a "pointer-like" type of the inner dimension of type being indexed. +- An indexer **must** be present accepting a `nuint` for the index, returning a `ref T` or `ref readonly T` (depending on the outer dimension mutability) where `T` is either the generic type (or `byte` for non-generic variants), a "pointer-like" type of the inner dimension (i.e. a `PtrPtr`'s indexer will return a `Ptr`) of type being indexed. - A `GetPinnableReference` **must** be present taking no parameters and returning a `ref T` or `ref readonly T` (depending on the outer dimension mutability) where `T` is the native raw pointer representation of the inner dimension. - For single-dimension pointers, an `AsSpan` method **must** be present taking an `int` length argument (due to historical reasons in the .NET BCL) returning a `Span` or `ReadOnlySpan` (depending on the outer dimension mutability) where `T` the pointee type. For non-generic pointer-like types, a generic argument **may** be used to specify the type of the span. - An array of the same jagged dimension **must** be implicitly convertible to the type. From c7ffb7c617f039488a9b1b612119af275d006beb Mon Sep 17 00:00:00 2001 From: Dylan Perks <11160611+Perksey@users.noreply.github.com> Date: Sun, 15 Oct 2023 18:34:59 -0500 Subject: [PATCH 4/9] Update documentation/proposals/Proposal - Generation of Library Sources and PInvoke Mechanisms.md --- ...al - Generation of Library Sources and PInvoke Mechanisms.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/proposals/Proposal - Generation of Library Sources and PInvoke Mechanisms.md b/documentation/proposals/Proposal - Generation of Library Sources and PInvoke Mechanisms.md index 7e5beea05e..71529e3043 100644 --- a/documentation/proposals/Proposal - Generation of Library Sources and PInvoke Mechanisms.md +++ b/documentation/proposals/Proposal - Generation of Library Sources and PInvoke Mechanisms.md @@ -255,7 +255,7 @@ The binding class **shall** itself implement the top-level interface, using the The binding class **shall** also contain shorthand functions for calling the static functions contained within the static subinterface using the *static default*. -**INFORMATIVE:** It is undecided whether we want to implement the static subinterface on the binding class itself as well. +The binding class **shall** implement the static subinterface (i.e. to proxy calls to the *static default* for ease of use). **INFORMATIVE:** Do we want to make a MakeCurrent shorthand as well? This may cause confusion for OpenGL users because it doesn't make the underlying context (WGL/EGL/GLX) current, it just makes it the current source of function pointers. From da870ce8e5b79c7d721bb0fd0fc6b927fb02dc43 Mon Sep 17 00:00:00 2001 From: Andrew Davis Date: Thu, 16 Nov 2023 11:55:54 -0500 Subject: [PATCH 5/9] Update SilkX Proposals (#1792) * Update SilkX Proposals - Updated Pointer Proposal to match current implementation - Add Generic Math Proposal based on Current work * Added missing function proposals and fixing some language * Implementing Review Feedback * Major rewrite of Generic Math Proposal -Slimmed it down though * Remove Vector 5 note * Addressing Review Comments * More Review Comments * Vector Transform Adjustments -and fix CreateFromPointNormal nomenclature * Remove the Endians that escaped the first time * Update for Rect/Box changes * Added LH/RH Matrix Functions * Box/Rect changes Making this a set of master types with obfuscated implmentation. Go For Perf --- ... Library Sources and PInvoke Mechanisms.md | 39 +- .../proposals/Proposal - Generic Math.md | 351 ++++++++++++++++++ 2 files changed, 379 insertions(+), 11 deletions(-) create mode 100644 documentation/proposals/Proposal - Generic Math.md diff --git a/documentation/proposals/Proposal - Generation of Library Sources and PInvoke Mechanisms.md b/documentation/proposals/Proposal - Generation of Library Sources and PInvoke Mechanisms.md index 71529e3043..ce377d66c0 100644 --- a/documentation/proposals/Proposal - Generation of Library Sources and PInvoke Mechanisms.md +++ b/documentation/proposals/Proposal - Generation of Library Sources and PInvoke Mechanisms.md @@ -4,6 +4,7 @@ Proposal design for a platform invoke (P/Invoke) mechanism for Silk.NET 3.0. # Contributors - Dylan Perks (@Perksey) - Kai Jellinghaus (@HurricanKai) +- Andrew Davis (@Curin) # Current Status - [ ] Proposed @@ -70,31 +71,47 @@ Tanner has already stated that they're happy to work with us to get Silk.NET usi The pivotal proposal around which the SilkX experiment revolves is solving the excessive overload problem by having no overloads at all. Namely, Silk.NET.Core will have a set of "pointer-like" types that can be implicitly casted to by a multitude of different types. This gives the user the same flexibility as having all different permutations of overloaded parameters generated without having to do so, and thereby alleviates the requirement of having overloads at all. -`Ptr` and `Mut` `ref struct`s **shall** be defined wrapping a `ref readonly` and `ref` respectively. These types **shall** have generic and non-generic variants. This shall be repeated up to 3 pointer dimensions, taking into account C's mutability rules in each instantiation. This means that there will be 14 top-level generic structs and 14 top-level non-generic structs. These **should** be auto-generated to ensure they remain maintainable. IL weaving **may** be required for the higher dimensions. +A `Ref` `ref struct` **shall** be defined wrapping a `ref`. A `Ptr` `struct` **shall** be defined wrapping a native pointer. These types **shall** have generic and non-generic variants. This shall be repeated up to 3 pointer dimensions. This means that there will be 6 top-level generic structs and 6 top-level non-generic structs. IL weaving **may** be required for `Ref2D` and `Ref3D`. -Unless deemed inappropriate, inapplicable, and/or infeasible by the Silk.NET team (the team reserved the right to do so without Working Group approval), the types **should** have the following characteristics: -- An instance **must** be constructable from a `ref T` or `ref readonly T` (depending on the outer dimension mutability) where `T` is either the generic type (or `byte` for non-generic variants), a "pointer-like" type of the inner dimension of type being constructed. -- An indexer **must** be present accepting a `nuint` for the index, returning a `ref T` or `ref readonly T` (depending on the outer dimension mutability) where `T` is either the generic type (or `byte` for non-generic variants), a "pointer-like" type of the inner dimension (i.e. a `PtrPtr`'s indexer will return a `Ptr`) of type being indexed. -- A `GetPinnableReference` **must** be present taking no parameters and returning a `ref T` or `ref readonly T` (depending on the outer dimension mutability) where `T` is the native raw pointer representation of the inner dimension. -- For single-dimension pointers, an `AsSpan` method **must** be present taking an `int` length argument (due to historical reasons in the .NET BCL) returning a `Span` or `ReadOnlySpan` (depending on the outer dimension mutability) where `T` the pointee type. For non-generic pointer-like types, a generic argument **may** be used to specify the type of the span. +Unless deemed inappropriate, inapplicable, and/or infeasible by the Silk.NET team (the team reserved the right to do so without Working Group approval), the `Ref` types **should** have the following characteristics: +- An instance **must** be constructable from a `ref T` where `T` is either the generic type (or `byte` for non-generic variants), a "pointer-like" type of the lower dimension of type being constructed. +- An indexer **must** be present accepting a `nuint` for the index, returning a `ref T` where `T` is either the generic type (or `byte` for non-generic variants), a "pointer-like" type of the lower dimension (i.e. a `Ref2D`'s indexer will return a `Ref`) of type being indexed. +- A `GetPinnableReference` **must** be present taking no parameters and returning a `ref T` where `T` is the native raw pointer representation of the lower dimension. +- For single-dimension pointers, an `AsSpan` method **must** be present taking an `int` length argument (due to historical reasons in the .NET BCL) returning a `Span` where `T` the pointee type. For non-generic pointer-like types, a generic argument **may** be used to specify the type of the span. - An array of the same jagged dimension **must** be implicitly convertible to the type. - For single-dimension pointers, a multi-dimensional array (up to 3 dimensions) **must** be implicitly convertible to the type. - A raw pointer of the same dimensions **must** be implicitly convertible to the type. - `NullPtr` **must** be implicitly convertible to the type, and the returned pointer must represent a null reference. - An explicit operator **must** be present to unsafely convert the reference represented by the pointer to a raw pointer without the user using `fixed`/`GetPinnableReference`. - An explicit operator **must** be present to unsafely convert the reference represented by the pointer to a void pointer without the user using `fixed`/`GetPinnableReference`. -- For generic pointer types, an implicit operator **must** be present to convert the pointer type to a `string` (or an array of strings of the inner dimension's jaggedness i.e. `PtrPtrPtr` becomes `string[][]`). Much like `ReadOnlySpan`, `ToString` **must** call this implicitly if the type is a single-dimension pointer to a string pointee type. The implicit cast **may** throw an exception if the pointee type is not a string pointee type. This is because we can't constrain the type used on implicit operators on generic types. +- For generic pointer types, an explicit operator **must** be present to convert the pointer type to a `string` (or an array of strings of the inner dimension's jaggedness i.e. `Ref3D` becomes `string[][]`). The explicit cast **may** throw an exception if the pointee type is not a string pointee type. This is because we can't constrain the type used on explicit operators on generic types. +- An `==` and `!=` operator **must** be present to check equality with another pointer of the same type. +- An `==` and `!=` operator **must** be present to check equality with `NullPtr` i.e. check whether the pointer is null. +- For generic pointer types to a string pointee type, a `string` (or an array of strings of the inner dimension's jaggedness i.e. `Ref3D` becomes `string[][]`) **must** be implicitly convertible to the pointer type. The implicit cast **may** throw an exception if the pointee type is not a string pointee type. This is because we can't constrain the type used on implicit operators on generic types. +- For single dimension generic pointer types, `Span` and `ReadOnlySpan` **must** be implicitly convertible to the pointer type. + +Unless deemed inappropriate, inapplicable, and/or infeasible by the Silk.NET team (the team reserved the right to do so without Working Group approval), the `Ptr` types **should** have the following characteristics: +- An instance **must** be constructable from a `T*` where `T` is either the generic type (or `void*` for non-generic variants), a "pointer-like" type of the lower dimension of type being constructed. +- An indexer **must** be present accepting a `nuint` for the index, returning a `ref T` where `T` is either the generic type (or `byte` for non-generic variants), a "pointer-like" type of the lower dimension (i.e. a `Ptr2D`'s indexer will return a `Ptr`) of type being indexed. +- A `GetPinnableReference` **must** be present taking no parameters and returning a `T` where `T` is the native raw pointer representation of the lower dimension. +- For single-dimension pointers, an `AsSpan` method **must** be present taking an `int` length argument (due to historical reasons in the .NET BCL) returning a `Span` where `T` is the pointee type. For non-generic pointer-like types, a generic argument **may** be used to specify the type of the span. +- For single-dimension pointers, a `ToArray` method **must** be present taking an `int` length argument (due to historical reasons in the .NET BCL) returning a `T[]` where `T` is the pointee type. For non-generic pointer-like types, a generic argument **may** be used to specify the type of the span. +- A raw pointer of the same dimensions **must** be implicitly convertible to the type. +- For multi-dimensional pointers, a `ToArray` method **must** be present taking an `int` length arguments (due to historical reasons in the .NET BCL) returning a jagged array of `T` of the same dimensionality where `T` is the pointee type. For non-generic pointer-like types, a generic argument **may** be used to specify the type of the span. +- `NullPtr` **must** be implicitly convertible to the type, and the returned pointer must represent a null reference. +- An implicit operator **must** be present to convert to a raw pointer. +- An implicit operator **must** be present to convert to a void pointer. +- For generic pointer types, an explicit operator **must** be present to convert the pointer type to a `string` (or an array of strings of the inner dimension's jaggedness i.e. `Ptr3D` becomes `string[][]`). The explicit cast **may** throw an exception if the pointee type is not a string pointee type. This is because we can't constrain the type used on explicit operators on generic types. - An `==` and `!=` operator **must** be present to check equality with another pointer of the same type. - An `==` and `!=` operator **must** be present to check equality with `NullPtr` i.e. check whether the pointer is null. -- For generic pointer types to a string pointee type, where the inner-most dimension is immutable, a `string` (or an array of strings of the inner dimension's jaggedness i.e. `PtrPtrPtr` becomes `string[][]`) **must** be implicitly convertible to the pointer type. The implicit cast **may** throw an exception if the pointee type is not a string pointee type. This is because we can't constrain the type used on implicit operators on generic types. -- For single dimension generic pointer types, `Span` **must** be implicitly convertible to the pointer type. -- For single dimension generic immutable pointer types, `ReadOnlySpan` **must** be implicitly convertible to the pointer type. Our goal with this implicitness it to make the experience of using pointers as similar to C as possible without feeling completely alien in a high-level language. This is the logic behind the string casting as well. The public API surface for our bindings **shall** use these wrapper types. For the avoidance of doubt, a string pointee type **shall** be defined as one of: `byte`, `sbyte`, `char`, `short`, `ushort`, `int`, `uint`. For single-byte string pointee types, UTF-8 **shall** be used. For 2-byte string pointee types, UTF-16 **shall** be used. For 4-byte string pointee types, UTF-32 **shall** be used. -**INFORMATIVE:** This was discussed informally by the Working Group in a Coffee & Code Catchup. The recording for this catchup can be found on the .NET Foundation YouTube channel [here](https://www.youtube.com/watch?v=N7qcETE4X_I). The most pertinent points were regarding ease of use and discoverability - unless the user reads the documentation (which will exist per the Working Group approved Software Development Plan), they won't know what all of these `Ptr` types mean and how to use them. It is our goal to make this as easy to understand as possible, with the goal being the ability to read one short page and instantly know how to use the majority of the library, but this was nonetheless a concern from the Working Group. There was also some desire to revisit the 1.X style of overloading, but the Silk.NET team were hesitant to do so to avoid history repeating itself. This is not an invalid suggestion however, given that TerraFX exists and can be treated as an equal now due to .NET Foundation membership, and their focus is exclusively unsafe so users that want unsafe can just use TerraFX. However, Silk.NET still wants to be as flexible as possible thus we have persisted in this model for this proposal. +**INFORMATIVE:** This implementation was discussed informally in the discord #team-chat [here](https://discord.com/channels/521092042781229087/587346162802229298/1166427977060450415) and [here](https://discord.com/channels/521092042781229087/587346162802229298/1167076268316045352). The purpose of this implementation was to simplify the naming scheme so that it was easily understandable. Const correctness was discarded in favor of simplicity. + +**INFORMATIVE:** This is in contrast to the previous proposal which was discussed informally by the Working Group in a Coffee & Code Catchup. The recording for this catchup can be found on the .NET Foundation YouTube channel [here](https://www.youtube.com/watch?v=N7qcETE4X_I). The most pertinent points were regarding ease of use and discoverability - unless the user reads the documentation (which will exist per the Working Group approved Software Development Plan), they won't know what all of these `Ptr` types mean and how to use them. Which while a short document could summarize, it was fekt this implementation was lacking in clarity. There was also some desire to revisit the 1.X style of overloading, but the Silk.NET team were hesitant to do so to avoid history repeating itself. This is not an invalid suggestion however, given that TerraFX exists and can be treated as an equal now due to .NET Foundation membership, and their focus is exclusively unsafe so users that want unsafe can just use TerraFX. However, Silk.NET still wants to be as flexible as possible thus we have persisted in this model for this proposal. **INFORMATIVE:** The Working Group previously expressed concerns for implicit casting for the trivial case of `string` to `ReadOnlySpan` in the 2021 meeting regarding the previous version of this proposal (see Meeting Notes). diff --git a/documentation/proposals/Proposal - Generic Math.md b/documentation/proposals/Proposal - Generic Math.md new file mode 100644 index 0000000000..899abcd3af --- /dev/null +++ b/documentation/proposals/Proposal - Generic Math.md @@ -0,0 +1,351 @@ +# Summary +A new API for generic maths in Silk.NET.Maths 3.X, leveraging modern .NET features such as `INumber` and vectorization. + +This API aims to replace the existing implementation of Silk.NET.Maths. + +# Contributors +- Maxine H (@uwx) +- Andrew Davis (@Curin) + +# Current Status +- [x] Proposed +- [ ] Discussed with API Review Board (ARB) +- [ ] Approved +- [ ] Implemented + +# Design Decisions +- This proposal should replace the 2.X implementation of `Silk.Net.Maths`, matching `System.Numerics` where possible, with concessions for design oversights in that api. +- This proposal assumes no knowledge of the 2.x Math library. +- Text herein marked **INFORMATIVE** does not form a normative part of this proposal, and is for background only. +- Within this proposal, the key words **must**, **required**, **shall**, **should**, **recommended**, **may**, **could**, and **optional** are to be interpreted as described in [RFC 2119 - Key words for use in RFCs to Indicate Requirement Levels](https://www.ietf.org/rfc/rfc2119.txt). The additional key word **optionally** is an alternate form of **optional**, for use where grammatically appropriate. These key words are highlighted in the proposal for clarity. + +# **INFORMATIVE** Integer and Floating Point Types +While investigating the use of generic math we came to the conclusion that making types which supports both integer and floating point types would not be optimal. This was discussed at length on the discord [here](https://discord.com/channels/521092042781229087/587346162802229298/1167705816812498974). Ultimately it was decided to provide both an integer and floating point variant for each vector type and every type built from them. These types are generic where `Vector2I` will be a 2D vector which takes any binary integer type for `T`. Similarly `Vector2F` will be a 2D vector which takes any floating point type for `T`. By extension we get types like `BoxI` and `RectangleF`. The integer types are granted the bitwise operators `&`, `~`, `|`, and `^`. Floating point types will include some operations that require certain functions unavailable to integer types like `Length` which requires `Sqrt`. + +# I types versus F Types +Each type in this proposal, aside from `Quaternion`, ends in I or F, defining whether it is an integer type or floating point type. Integer types **must** use a generic type argument `T` with the constraint of `IBinaryInteger`. On the other hand, floating point types **must** use a generic type argument `T` with the constraint of `IFloatingPointIeee754`. + +# Vector Types + +The main types defined for this proposal are two sets of vector types, `VectorNI` and `VectorNF` where `N` defines the dimensionality of the vector between 2 and 4. + +For each vector struct, the following requirements **must** fulfill the following requirements: +- Implements IEquatable with itself as the generic parameter +- Implements IReadonlyList with the components as the list elements +- Implements ISpanFormattable +- Implements ISpanParsable +- Implements IUtf8SpanFormattable +- Implements IUtf8SpanParsable +- Implements IParsable +- Implemetns IFormattable +- The relevant number of properties to represent the mathematical vector's components (X and Y for Vector2) and relevant unit vectors +- Constructors which take either a single parameter and uses it for every component, a parameter for each component, or a ReadOnlySpan of values which has the same number of elements as our vector has components. +- Constructors for 3 dimensions and up **must** include lower dimension variants that use the lower dimensions for their specific components (vector2 -> X,Y). +- A ref indexer that takes a int index and returns the corresponding component value (0 -> x, 1 -> y, etc.) +- An AsSpan function which returns this vector as a Span of the generic type +- A LengthSquared property which returns the dot product of the vector with itself. +- A Dot function which takes another vector and returns the dot product with our original vector. + - A static implementation of this function **must** be available as well. +- For 3D Vectors, a Cross function which takes another vector and returns the cross product with our original vector. + - A static implementation of this function **must** be available as well. +- `+`, `-`, `*`, `/`, and `%` operators defined between two vectors of the same type which returns a vector which has had each operation applied component-wise. +- `+`, `-`, `*`, `/`, and `%` operators defined between a vector and a scalar value that matches the generic type which returns a vector which has had each operation applied component-wise with the scalar value. Both vector first and scalar first should be implemented. +- A `-` unary operator which returns the negated vector. +- A `+` unary operator which returns the vector. +- Overrides ToString to show component values. +- Max and Min functions, which takes another vector and returns a new vector which component-wise has the Max or Min value, respectively. + - A Static implementation of this function **must** be available as well. +- Max and Min functions, which takes a scalar value which matches the generic type and returns a new vector which component-wise has the Max or Min value with the scalar value, respectively. + - A Static implementation of this function **must** be available as well. +- A Clamp function which takes a Max vector and a Min vector and returns a vector which has its components bounded between the Min and Max vectors. + - A Static implementation of this function **must** be available as well. +- A Clamp function which takes a Max scalar and a Min scalar, both which match the generic type, and returns a vector which has its components bounded between the Min and Max scalars. + - A Static implementation of this function **must** be available as well. +- An Abs function which returns a vector where each component is the absolute value of the original + - A Static implementation of this function **must** be available as well. +- CopyTo functions which copy to an array or span, with or without a starting index +- Explicit cast and checked cast operators to all standard variants of the F and I vector types of the same dimensionality +- Explicit cast and checked cast to and from matching System.Numerics vector type +- Explicit cast cast to lower dimensional matching vector with matching generic type +- A CopySign function which takes a vector and copies the signs component-wise to our vector + - A Static implementation of this function **must** be available but it should return a new vector without affecting the original +- A Copy sign function which takes a scalar which matches the generic type, and copies the scalars sign onto each component of the vector + - A Static implementation of this function **must** be available but it should return a new vector without affecting the original +- A Sign function which returns a vector where each component is only the sign segment of the original vector + - A Static implementation of this function **must** be available +- Static Unit Vectors for each component +- A Static Zero Vector with zero for all components +- A Static One Vector with one for all components +- A static AllBitsSet Vector with all bits set for all components +- Define static CreateChecked, CreateSaturating, and CreateTruncating which converts other vector types to this type + - Try variants of these methods should also be defined which out the resulting vector and return a bool representing success or failure of the operation. +- Define Transform functions which take a Matrix of higher dimensionality assuming 1 in for the final missing component and 0 for the rest (Vector 2 can use Matrix2xn, Matrix3xn, and matrix4xn) and return a vector containing the output (type should match the outer type e.g. Vector2.Transform(Matrix4x4) returns Vector2) + - A Static implementation of these functions **must** be available +- Define VectorN `*` MatrixNxM operators where N is the same for both Vector and Matrix, but M is any number + - These operators should function like Transform, but without needed assumptions +- Define TransformNormal functions which take a Matrix of higher dimensionality assuming 0 in for all missing components (Vector 2 can use Matrix2xn, Matrix3xn, and matrix4xn) and return a vector containing the output (type should match the outer type e.g. Vector2.Transform(Matrix4x4) returns Vector2) + - A Static implementation of these functions **must** be available + +For I types, the following additional requirements **must** be fulfilled: +- the bitwise `&`, `|`, and `^` operators defined between two vectors which returns a vector which has had these operators applied on a component-wise basis. +- the bitwise `&`, `|`, and `^` operators defined between a vectors and a scalar value that matches the generic type which returns a vector which has had these operators applied on a component-wise basis with the scalar. +- the unary bitwise `~` operator defined which negates the bits of the vector components. +- Define the following static functions for these types to match IBinaryInteger (Vector replaced with type, e.g. `Vector2I`) which returns a new vector with these operations applied component-wise, unless otherwise specified: + - Log2(Vector x) + - DivRem(Vector left, Vector right) + - Returns tuple of 2 Vectors (Vector Quotient, Vector Remainder) + - PopCount(Vector x) + - returns The number of set bits in the vector + +For F types, the following additional requirements **must** be fulfilled: +- A Length property which returns the square root of LengthSquared. +- A Normalize function which divides all components by the length of the vector + - A static implementation of this function **must** be available but it should return a normalized vector without affecting the original vector +- A static Lerp function which takes Two vectors to interpolate between and a vector representing the t value for each component, and returns a vector which components are linearly interpolated between the original two vectors based on the respective t values. + - A clamped version of this function **must** also be available which clamps the t-values between 0 and 1 +- A static Lerp function which takes Two vectors and a scalar value which matches the generic type, and returns a vector which is linearly interpolated between the two vectors using the scalar as the t value. + - A clamped version of this function **must** also be available which clamps the t-values between 0 and 1 +- A Reflect Function which takes a normal vector and reflects the vector over the normal + - A Static implemenation of this function **must** be available as well, but should return the reflected vector without affecting the original vector. +- The following static Vector properties which have the given value for all components + - PositiveInfinity + - NegativeInfinity + - NaN + - Epsilon + - NegativeZero + - Pi + - Tau + - E +- Define the following static functions for these types to match IBinaryFloatingPointIeee754 (Vector replaced with type, e.g. `Vector2F`) which returns a new vector with these operations applied component-wise, unless otherwise specified: + - Sqrt(Vector x) + - Acosh(Vector x) + - Asinh(Vector x) + - Atanh(Vector x) + - Cosh(Vector x) + - Sinh(Vector x) + - Tanh(Vector x) + - Acos(Vector x) + - AcosPi(Vector x) + - Asin(Vector x) + - AsinPi(Vector x) + - Atan(Vector x) + - AtanPi(Vector x) + - Cos(Vector x) + - CosPi(Vector x) + - Sin(Vector x) + - SinPi(Vector x) + - Tan(Vector x) + - TanPi(Vector x) + - DegreesToRadians(Vector degrees) + - RadiansToDegrees(Vector radians) + - SinCos(Vector x) + - Returns a tuple of 2 Vectors (Sin, Cos) + - SinCosPi(Vector x) + - Returns a tuple of 2 Vectors (SinPi, CosPi) + - Log(Vector x) + - Log(Vector x, Vector newBase) + - Log(Vector x, TScalar newBase) + - LogP1(Vector x) + - Log2(Vector x) + - Log2P1(Vector x) + - Log10(Vector x) + - Log10P1(Vector x) + - Exp(Vector x) + - ExpM1(Vector x) + - Exp2(Vector x) + - Exp2M1(Vector x) + - Exp10(Vector x) + - Exp10M1(Vector x) + - Pow(Vector x, Vector y) + - Pow(Vector x, TScalar y) + - Cbrt(Vector x) + - Hypot(Vector x, Vector y) + - Hypot(Vector x, TScalar y) + - RootN(Vector x, int n) + - Round(Vector x) + - Round(Vector x, int digits) + - Round(Vector x, MidpointRounding mode) + - Round(Vector x, int digits, MidpointRounding mode) + - Truncate(Vector x) + - Atan2(Vector x, Vector y) + - Atan2Pi(Vector x, Vector y) + - Atan2(Vector x, TScalar y) + - Atan2Pi(Vector x, TScalar y) + - BitDecrement(Vector x) + - BitIncrement(Vector x) + - FusedMultiplyAdd(Vector left, Vector right, Vector addend) + - FusedMultiplyAdd(Vector left, Vector right, TScalar addend) + - FusedMultiplyAdd(Vector left, TScalar right, Vector addend) + - FusedMultiplyAdd(Vector left, TScalar right, TScalar addend) + - ReciprocalEstimate(Vector x) + - ReciprocalSqrtEstimate(Vector x) + - ILogB(Vector x) + - Returns VectorNI, where N matches the dimensionality of the vector + - **INFORMATIVE** This may require multiple methods depending on implementation + - ScaleB(Vector x, VectorNI n) + - ScaleB(Vector x, int n) + - RoundToInt(Vector x) + - Returns `VectorNI`, where N matches the dimensionality of the vector + - **INFORMATIVE** This may require multiple methods depending on implementation + - FloorToInt(Vector x) + - Returns `VectorNI`, where N matches the dimensionality of the vector + - **INFORMATIVE** This may require multiple methods depending on implementation + - CeilingToInt(Vector x) + - Returns `VectorNI`, where N matches the dimensionality of the vector + - **INFORMATIVE** This may require multiple methods depending on implementation + - ToVector64(Vector x) + - Returns `System.Runtime.Intrinsics.Vector64` + - ToVector128(Vector x) + - Returns `System.Runtime.Intrinsics.Vector128` + - ToVector256(Vector x) + - Returns `System.Runtime.Intrinsics.Vector256` + - ToVector512(Vector x) + - Returns `System.Runtime.Intrinsics.Vector512` + +# Matrix Types + +This proposal includes the following matrix types: +- Matrix2x2F +- Matrix2x2I +- Matrix2x3F +- Matrix2x3I +- Matrix2x4F +- Matrix2x4I +- Matrix3x2F +- Matrix3x2I +- Matrix3x3F +- Matrix3x3I +- Matrix3x4F +- Matrix3x4I +- Matrix4x2F +- Matrix4x2I +- Matrix4x3F +- Matrix4x3I +- Matrix4x4F +- Matrix4x4I +- Matrix5x4F +- Matrix5x4I + +Integer Variants do not require any functions which interact with Quaternions + +Matrix structs **must** fulfill the following requirements: +- Fulfills `IEquatable` where `T` is the same matrix class +- Stored in row major format +- F matricies work with F vectors, and I Matricies work with I vectors +- Both row vectors and individual values (M11, etc.) accessible via properties +- A ref indexer that takes row and column indicies and outputs the value +- Add, subtract, and multiply operators defined with Matricies of the same size +- Multiply operators defined with compatible matricies, if the output matrix type already exists (AxB * BxC = AxC) +- Negate Operator defined +- Implicit conversion to and from the System.Numerics matrix type, if available +- Invert function for square matricies +- GetDeterminant function for square matricies and Matrix3x2, Matrix4x3, and Matrix 5x4 +- Transpose function +- static lerp function +- static identity property +- For Matrix3x2, Matrix3x3, Matrix4x3, and Matrix4x4 include the following static functions + - CreateBillboard + - CreateRotation + - 3x3, 4x3, and 4x4 Matricies get X, Y, and Z variants for this function instead + - CreateTranslation + - CreateScale + - Decompose (separate out any transformations) +- For Matrix3x2 include a CreateSkew static function +- For Matrix3x3, Matrix4x3, and Matrix4x4 include the following static functions + - CreateFromAxisAngle + - CreateFromQuaternion + - Transform + - from a Quaternion + - CreateFromYawPitchRoll +- For Matrix4x3 and Matrix4x4 include the following static functions + - CreateConstrainedBillboard + - CreateLookAtLH + - CreateOrthographicLH + - CreateOrthographicOffCenterLH + - CreatePerspectiveLH + - CreatePerspectiveFieldOfViewLH + - CreatePerspectiveOffCenterLH + - CreateLookAtRH + - CreateOrthographicRH + - CreateOrthographicOffCenterRH + - CreatePerspectiveRH + - CreatePerspectiveFieldOfViewRH + - CreatePerspectiveOffCenterRH + - CreateReflection + - CreateWorld + +# Quaternion + +A Quaternion struct **must** be defined and match the following requirements: +- A generic struct with a type parameter `T` which is constrained by `IBinaryFloatingPointIeee754` representing the scalar type +- Implements IEquatable with itself +- Contain 4 scalar properties (X, Y, Z, W) +- Define a Constructor taking 4 scalar values matching the properties +- Define a Constructor taking a Vector3F and a Scalar, with the vector 3 mapping to X, Y, Z and the Scalar to the W +- Define a Constructor taking a Vector4F +- A Vector3F Axis property mapping to (X, Y, Z) +- A T Angle property mapping to 2 * Acos(W) +- A ref Indexer which takes an int and returns the components in order +- An AsSpan function which returns this quaternion as a Span of the generic type +- An IsIdentity property which returns if this Quaternion matches an Identity Quaternion +- Define `+`, `-`, `*`, and `/` between two Quaternions +- Define `*` with `T` multiplying each component by the scalar value returning a new quaternion +- Define unary `~` +- A Dot function which takes another Quaternion and returns its the dotproduct between them + - A static implementation of this function **must** be available +- A LengthSquared property which returns the dot product of the quaternion with itself +- A Length property which returns the Square Root of LengthSquared +- An Invert function inverts the Quaternion + - a static Inverse function **must** be available but it returns the inverse rather than affecting the original +- A Normalize function which normalizes the Quaternion + - A static implemenation of this function must be available but returns the normalized Quaternion rather than affecting the original +- A Concatenate function which takes another Quaternion and concatenates it with this quaternion + - A static implementation of this function **must** be available but it returns a new Quaternion rather than affecting the originals +- A Conjugate function which returns the conjugate of this quaternion + - A static implementation of this function **must** be available +- A static CreateFromAxisAngle function which takes in a Vector3F and an angle and returns a Quaternion representing that rotation +- A static CreateFromRotationMatrix function which takes either a Matrix3x3 or Matrix4x4 and returns a Quaternion representing that rotation +- A static CreateFromYawPitchRoll which takes either each components separately or in a Vector3F and outputs a Quaternion representing that rotation +- A static Lerp function which takes 2 Quaternions and a Scalar matching the generic type which linearly interpolates between the 2 Quaternions with scalar used as the amount to lerp +- A static SLerp function which takes 2 Quaternions and a Scalar matching the generic type which Spherical linearly interpolates between the 2 Quaternions with scalar used as the amount to lerp +- A static Zero Quaternion Property +- A static Identity Quaternion Property + +# Geometric Types + +The following Geometric Types are defined: +- BoxF +- BoxI +- CircleF +- CircleI +- PlaneF +- PlaneI +- Ray2F +- Ray2I +- Ray3F +- Ray3I +- RectangleF +- RectangleI +- SphereF +- SphereI + +Each type **must** include the following: +- Intersect functions with both another instance of the type and a point +- GetDistanceToNearest(Point,Edge,etc) functions **must** be available for a given point +- For all but the rays and planes, GetInflated function that takes a point and returns the scaled object that is closest to the original and contains the given point +- Include Scale and Translation transformation functions +- For Box and Rectangle the following Vector properties **must** be defined + - Min + - Max + - Center + - Size +- For Planes and Rays, Normalize functions +- For Planes include the following static functions + - CreateFromVerticies + - CreateFromPointNormal + - Dot + - with a Vector4 + - DotCoordinate and DotNormal + - with a Vector3 + - Transform + - With a Matrix4x4 or Quaternion, if relevant \ No newline at end of file From cd767e5251b5e365a297a781909343c5297b2e08 Mon Sep 17 00:00:00 2001 From: Dylan Perks <11160611+Perksey@users.noreply.github.com> Date: Mon, 20 Nov 2023 23:51:55 -0600 Subject: [PATCH 6/9] Update Proposal - Generation of Library Sources and PInvoke Mechanisms.md --- ... Library Sources and PInvoke Mechanisms.md | 23 +++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/documentation/proposals/Proposal - Generation of Library Sources and PInvoke Mechanisms.md b/documentation/proposals/Proposal - Generation of Library Sources and PInvoke Mechanisms.md index ce377d66c0..55cf1c7a1c 100644 --- a/documentation/proposals/Proposal - Generation of Library Sources and PInvoke Mechanisms.md +++ b/documentation/proposals/Proposal - Generation of Library Sources and PInvoke Mechanisms.md @@ -7,9 +7,9 @@ Proposal design for a platform invoke (P/Invoke) mechanism for Silk.NET 3.0. - Andrew Davis (@Curin) # Current Status -- [ ] Proposed -- [ ] Discussed with Working Group (WG) -- [ ] Approved +- [x] Proposed +- [x] Discussed with Working Group (WG) +- [x] Approved - [ ] Implemented # Design Decisions @@ -89,6 +89,7 @@ Unless deemed inappropriate, inapplicable, and/or infeasible by the Silk.NET tea - An `==` and `!=` operator **must** be present to check equality with `NullPtr` i.e. check whether the pointer is null. - For generic pointer types to a string pointee type, a `string` (or an array of strings of the inner dimension's jaggedness i.e. `Ref3D` becomes `string[][]`) **must** be implicitly convertible to the pointer type. The implicit cast **may** throw an exception if the pointee type is not a string pointee type. This is because we can't constrain the type used on implicit operators on generic types. - For single dimension generic pointer types, `Span` and `ReadOnlySpan` **must** be implicitly convertible to the pointer type. + - For `ReadOnlySpan`, it is conceivable that the span we're casting represents a string slice. Therefore, the operator **must** implicitly copy the span to a new array that is suffixed with a trailing zero char to ensure it is still usable as such, and `ref` that array instead. Unless deemed inappropriate, inapplicable, and/or infeasible by the Silk.NET team (the team reserved the right to do so without Working Group approval), the `Ptr` types **should** have the following characteristics: - An instance **must** be constructable from a `T*` where `T` is either the generic type (or `void*` for non-generic variants), a "pointer-like" type of the lower dimension of type being constructed. @@ -341,4 +342,18 @@ The binding class **shall** implement the static subinterface (i.e. to proxy cal - [x] Change `Modifiers` to a CallConv\* `Type` array **FUTURE** -- [ ] Report back to the Community our findings in experimenting with overloads +- [x] Report back to the Community our findings in experimenting with overloads + +## 19/11/2023 + +[Video](https://www.youtube.com/live/yXNDZDE3AHE?feature=shared&t=3326) + +- We discussed a particular problematic case where RegisterClassEx returns an atom which is later reinterpreted to be a pointer - a debugger will explode when inspecting this pointer as it is not necessarily +- Where ReadOnlySpan represents a string and we don't just want to pass the ref as-is (i.e. we want to add the null terminator like we do for string). +- Generally we think that providing a tool that works 90% of the time is fine, the unsafe overloads are always there, but we'd worry about users making incorrect assumptions and we can probably do implicit behaviour for that final 10%. +- Require that users manually encoding strings add that null terminator and document this. Our implicit ones do the right thing. +- Don't allow ref types to throw an error when being handed something that isn't a valid pointer. +- Approved provided that we: + - make unsafe available + - special case ROSpan as above +- Future discussions need to be had on Vulkan implementation intricacies (getProcAddr) and also the addition of "complex" overloads. From 09f17b5ff5588f471810e03da41771cce1d3eacc Mon Sep 17 00:00:00 2001 From: Dylan Perks <11160611+Perksey@users.noreply.github.com> Date: Mon, 20 Nov 2023 23:55:01 -0600 Subject: [PATCH 7/9] Bring back contexts - this was noted in WG review --- ...tion of Library Sources and PInvoke Mechanisms.md | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/documentation/proposals/Proposal - Generation of Library Sources and PInvoke Mechanisms.md b/documentation/proposals/Proposal - Generation of Library Sources and PInvoke Mechanisms.md index 55cf1c7a1c..d084a74271 100644 --- a/documentation/proposals/Proposal - Generation of Library Sources and PInvoke Mechanisms.md +++ b/documentation/proposals/Proposal - Generation of Library Sources and PInvoke Mechanisms.md @@ -210,23 +210,23 @@ public class MyStringLibrary : IMyStringLibrary } // Non-Static Interface - private Func _getProcAddress; + private INativeContext _ctx; byte* IMyStringLibrary.ToLower(byte* str) { - var ptr = _getProcAddress("ToLower"); + var ptr = _ctx.LoadFunction("ToLower"); if (ptr is 0) throw new("some symbol loading exception..."); return ((delegate* unmanaged)ptr)(str); } void IMyStringLibrary.FreeResult(byte* str) { - var ptr = _getProcAddress("FreeResult"); + var ptr = _ctx.LoadFunction("FreeResult"); if (ptr is 0) throw new("some symbol loading exception..."); return ((delegate* unmanaged)ptr)(str); } public static IMyStringLibrary Create() => new StaticWrapper(); - public static IMyStringLibrary Create(Func getProcAddress) => new MyStringLibrary { _getProcAddress = getProcAddress }; + public static IMyStringLibrary Create(INativeContext ctx) => new MyStringLibrary { _ctx = ctx }; // Static Interface public static byte* ToLower(byte* str) => DllImport.ToLower(str); @@ -236,8 +236,6 @@ public class MyStringLibrary : IMyStringLibrary } ``` -**INFORMATIVE:** `getProcAddress` delegates replace "native contexts". - There exist requirements for all of the following: - A native function retrieved using a thread-specific "native context" can be called using a static function (for OpenGL) - A native function retrieved using a custom "native context" delegate can be called using a static function (for Vulkan, OpenCL, OpenXR). @@ -269,7 +267,7 @@ The binding class **shall** expose a `static` `Create` method with a `Func Date: Mon, 20 Nov 2023 23:56:39 -0600 Subject: [PATCH 8/9] Add meeting notes to the generic math proposal --- .../proposals/Proposal - Generic Math.md | 36 +++++++++++++++++-- 1 file changed, 33 insertions(+), 3 deletions(-) diff --git a/documentation/proposals/Proposal - Generic Math.md b/documentation/proposals/Proposal - Generic Math.md index 899abcd3af..35ad4a3702 100644 --- a/documentation/proposals/Proposal - Generic Math.md +++ b/documentation/proposals/Proposal - Generic Math.md @@ -9,8 +9,8 @@ This API aims to replace the existing implementation of Silk.NET.Maths. # Current Status - [x] Proposed -- [ ] Discussed with API Review Board (ARB) -- [ ] Approved +- [x] Discussed with API Review Board (ARB) +- [x] Approved - [ ] Implemented # Design Decisions @@ -348,4 +348,34 @@ Each type **must** include the following: - DotCoordinate and DotNormal - with a Vector3 - Transform - - With a Matrix4x4 or Quaternion, if relevant \ No newline at end of file + - With a Matrix4x4 or Quaternion, if relevant + +# Meeting Notes + +## 19/11/2023 + +[Video](https://www.youtube.com/live/yXNDZDE3AHE?feature=shared&t=9444) + +- We agree with the addition to add scalar operations over vectors (i.e. Vector4 * T does X*T, Y*T, ...) +- Add an analyser for encouraging the most correct and most efficient type instead of using sub-optimal types. +- Ensure we've documented that. +- Vector * Matrix? + - assuming row major, not allowing matrix * vector (only vector * matrix) + - does it make sense to just do it the way game engines do it and the way 2.0 does it? + - possibly technically invalid mathematically, even if common programmatically. + - easy to hit corner case that not all users may understand. +- Ensuring we have no upcasting and downcasting operators that could have unexpected behaviour. +- System.Numerics is strictly right hand, which is the same as XNA and somewhat default for OpenGL users. This differs with DirectX that is typically left hand. + - Happy with having explicit LH and RH functions. +- Missing matrix APIs that were added in .NET 8 to System.Numerics + - CreateLookTo (additional to CreateLookAt) + - CreateViewport + - CreateBillboardLH + - CreateConstrainedBillboardLH + - CreateBillboardRH + - CreateConstrainedBillboardRH +- Investigate newer less-hated alternatives to quaternions for future (see Freya Holmer talks) +- Possibly have a general Transform struct in the future? +- Rectangles and boxes are obscured with regards to internal layout, bindings to have interchange types that implicitly cast to the mathematical constructs that are represented by the Box/Rectangle types. +- Investigate in the future if we can either add an Origin property for compatibility or an analyser for users to know that Min is the new Origin. +- Approved notwithstanding the missing APIs. From 576d84f14c8a2eedc0ec877a7b4438bf85429415 Mon Sep 17 00:00:00 2001 From: Dylan Perks <11160611+Perksey@users.noreply.github.com> Date: Thu, 21 Dec 2023 13:22:04 -0600 Subject: [PATCH 9/9] Update Proposal - Generic Math.md --- documentation/proposals/Proposal - Generic Math.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/documentation/proposals/Proposal - Generic Math.md b/documentation/proposals/Proposal - Generic Math.md index 35ad4a3702..8c2d96c306 100644 --- a/documentation/proposals/Proposal - Generic Math.md +++ b/documentation/proposals/Proposal - Generic Math.md @@ -18,6 +18,7 @@ This API aims to replace the existing implementation of Silk.NET.Maths. - This proposal assumes no knowledge of the 2.x Math library. - Text herein marked **INFORMATIVE** does not form a normative part of this proposal, and is for background only. - Within this proposal, the key words **must**, **required**, **shall**, **should**, **recommended**, **may**, **could**, and **optional** are to be interpreted as described in [RFC 2119 - Key words for use in RFCs to Indicate Requirement Levels](https://www.ietf.org/rfc/rfc2119.txt). The additional key word **optionally** is an alternate form of **optional**, for use where grammatically appropriate. These key words are highlighted in the proposal for clarity. +- If any of the APIs contained herein are later deemed mathematically invalid in the context of their exposing primitive (e.g. a specific operation being inappropriate for a specific sized matrix), the Silk.NET team reserves the right to remove them at their own accord. # **INFORMATIVE** Integer and Floating Point Types While investigating the use of generic math we came to the conclusion that making types which supports both integer and floating point types would not be optimal. This was discussed at length on the discord [here](https://discord.com/channels/521092042781229087/587346162802229298/1167705816812498974). Ultimately it was decided to provide both an integer and floating point variant for each vector type and every type built from them. These types are generic where `Vector2I` will be a 2D vector which takes any binary integer type for `T`. Similarly `Vector2F` will be a 2D vector which takes any floating point type for `T`. By extension we get types like `BoxI` and `RectangleF`. The integer types are granted the bitwise operators `&`, `~`, `|`, and `^`. Floating point types will include some operations that require certain functions unavailable to integer types like `Length` which requires `Sqrt`. @@ -244,7 +245,8 @@ Matrix structs **must** fulfill the following requirements: - static lerp function - static identity property - For Matrix3x2, Matrix3x3, Matrix4x3, and Matrix4x4 include the following static functions - - CreateBillboard + - CreateBillboardRH + - CreateBillboardLH - CreateRotation - 3x3, 4x3, and 4x4 Matricies get X, Y, and Z variants for this function instead - CreateTranslation @@ -258,14 +260,17 @@ Matrix structs **must** fulfill the following requirements: - from a Quaternion - CreateFromYawPitchRoll - For Matrix4x3 and Matrix4x4 include the following static functions - - CreateConstrainedBillboard + - CreateConstrainedBillboardLH + - CreateConstrainedBillboardRH - CreateLookAtLH + - CreateLookToLH - CreateOrthographicLH - CreateOrthographicOffCenterLH - CreatePerspectiveLH - CreatePerspectiveFieldOfViewLH - CreatePerspectiveOffCenterLH - CreateLookAtRH + - CreateLookToRH - CreateOrthographicRH - CreateOrthographicOffCenterRH - CreatePerspectiveRH @@ -273,6 +278,7 @@ Matrix structs **must** fulfill the following requirements: - CreatePerspectiveOffCenterRH - CreateReflection - CreateWorld + - CreateViewport # Quaternion