From 543e4155251ac5501a5bf8f590b783ede97dfc20 Mon Sep 17 00:00:00 2001 From: Dylan Perks Date: Thu, 12 Dec 2024 17:27:42 +0000 Subject: [PATCH] Modify Mutli-Backend Input --- .../Proposal - Multi-Backend Input.md | 920 ++++++++++++------ 1 file changed, 619 insertions(+), 301 deletions(-) diff --git a/documentation/proposals/Proposal - Multi-Backend Input.md b/documentation/proposals/Proposal - Multi-Backend Input.md index c6ef52c9ff..12330e2925 100644 --- a/documentation/proposals/Proposal - Multi-Backend Input.md +++ b/documentation/proposals/Proposal - Multi-Backend Input.md @@ -90,7 +90,7 @@ Please see the Windowing 3.0 proposal for `INativeWindow`. Input devices all inherit from a root interface. ```cs -public interface IInputDevice +public interface IInputDevice : IEquatable { nint Id { get; } string Name { get; } @@ -103,6 +103,20 @@ public interface IInputDevice All devices originate from a backend. +An `IInputDevice` object shall be equatable to any such object retrieved from the same backend where `Id` is equal. + +`IInputDevice` objects must not store any managed state, and if there is a requirement for this in a future extension of +this API then this **must** be defined in such a way that the state storage and lifetime is user-controlled. While +`IInputDevice` objects are equatable based on `Id`s, if a physical device disconnects and reconnects the `IInputBackend` +does not provide a guarantee that the same object will be returned (primarily because doing so would require the +`IInputBackend` to keep track of every object it's ever created), rather a "compatible" one that acts identically to +the original object. This is completely benign if the object is nothing but a wrapper to the backend anyway. If there is +unmanaged state (e.g. a handle to a device that must be explicitly closed upon disconnection), then it is expected that +even in the event of reconnection, old objects (e.g. created with a now-disposed handle) **shall** still work for the +newly-reconnected device. A common way this could be implemented is storing the handles in the `IInputBackend` +implementation instead in the form of a mapping of physical device IDs (`Id`) to those handles. This solves the object +lifetime problem while also not adding undue complications to user code. + # Backends ```cs @@ -119,7 +133,15 @@ public interface IInputBackend `Id` is a globally-unique integral identifier for this backend. -`Devices` enumerates all of the **connected** devices available from this input backend. When a device is disconnected, its `IInputDevice` object should be discarded by all that consumed it, as it can not be relied upon for being reused by the input backend. An implementation is welcome to reuse old objects, but this is strictly implementation-defined. A device not being present in the `Devices` list is sufficient evidence that a device has been disconnected. +`Devices` enumerates all of the **connected** devices available from this input backend. When a device is disconnected, +its `IInputDevice` **shall** no longer function and will not be enumerated by this list. When a device is connected, an +`IInputDevice` with that physical device ID **shall** be added to this list. In addition, upon connection any past +`IInputDevice` objects previously enumerated by this list on this `IInputBackend` instance **shall** also regain +function the device being added to this list shares the same physical device ID as those previous instances. All such +previous instances **shall** be equatable to one another and to the `IInputDevice` instance added to this list. An +implementation is welcome to reuse old objects, but this is strictly implementation-defined. A device not being present +in the `Devices` (checked using `IInputDevice`'s `IEquatable` implementation) list is sufficient evidence +that a device has been disconnected. `Update` will update the state of all devices contained within this input backend. The value of the `State` properties on each device must not change until this method is called. This is a departure from 1.0's and 2.0's model of updating state as soon as new information is available, which has resulted in lots of inconsistencies in the past. @@ -159,10 +181,10 @@ The `IInputHandler` passed into `Update` may implement multiple other handler in Note that during the `Update` call, a backend must only update the device's state in the order that the events are delivered. For example when `IInputBackend.Update` is called: 1. The backend has a queued "mouse down" event. 2. The backend updates the `State` of the relevant `IMouse` for that button press. -3. The backend calls `HandleButtonDown` on the `IMouseInputHandler` (if applicable). +3. The backend calls `HandleButtonChanged` with `IsDown` set to `true` on the `IMouseInputHandler` (if applicable). 4. The backend has a queued "mouse up" event. 5. The backend updates the `State` of the relevant `IMouse` for that button release. -6. The backend calls `HandleButtonUp` on the `IMouseInputHandler` (if applicable). +6. The backend calls `HandleButtonChanged` with `IsDown` set to `true` on the `IMouseInputHandler` (if applicable). This allows the actor to work with the whole device state with the device state being representative of the time that the original event occurred. @@ -175,7 +197,7 @@ All of the `Devices` and `Update`s are aggregated and coordinated by a central i ```cs public partial class InputContext { - public Mice Mice { get; } + public Pointers Pointers { get; } public Keyboards Keyboards { get; } public Gamepads Gamepads { get; } public Joysticks Joysticks { get; } @@ -188,11 +210,11 @@ public partial class InputContext The central input object acts as the main entry point into the Input API, and is responsible for comparing the state reported by the devices for differences between `Update` calls (raising events as necessary). -`Mice`, `Keyboards`, `Gamepads`, and `Joysticks` are all custom `IReadOnlyList` types for enumerating the devices. However, these custom types also contain the events. This is so we can "scope" the events, rather than putting them at the top-level and having to call the events `MouseButtonDown`, `JoystickButtonDown`, etc. +`Pointers`, `Keyboards`, `Gamepads`, and `Joysticks` are all custom `IReadOnlyList` types for enumerating the devices. However, these custom types also contain the events. This is so we can "scope" the events, rather than putting them at the top-level and having to call the events `PointerButtonChanged`, `JoystickButtonChanged`, etc. By virtue of the `State` properties not updating until `IInputBackend.Update` is called, the states of the devices enumerated by the lists will not change until `Update` is called. -`Update` will call `IInputBackend.Update` on each of the `Backends`, passing in a handler which implements `IInputHandler`, `IMouseInputHandler`, `IKeyboardInputHandler`, `IGamepadInputHandler`, and `IJoystickInputHandler` with each of the methods invoking a matching event defined in "Custom List Types" or on the input context itself (such as `ConnectionChanged`). +`Update` will call `IInputBackend.Update` on each of the `Backends`, passing in a handler which implements `IInputHandler`, `IPointerInputHandler`, `IKeyboardInputHandler`, `IGamepadInputHandler`, and `IJoystickInputHandler` with each of the methods invoking a matching event defined in "Custom List Types" or on the input context itself (such as `ConnectionChanged`). `Backends` is a mutable list of input backends. Until `Update` is called again, no device lists, state, etc on the context will be updated. The `ConnectionChanged` rules above will still be respected e.g. when you remove a backend, all of its devices will have a disconnected event raised for them. @@ -203,44 +225,40 @@ By virtue of the `State` properties not updating until `IInputBackend.Update` is These are relatively simple list wrappers with the events fired when state changes. ```cs -public partial class Mice : IReadOnlyList +public partial class Pointers : IReadOnlyList { - public MouseClickConfiguration ClickConfiguration { get; set; } - public event Action? ButtonDown; - public event Action? ButtonUp; - public event Action? Click; - public event Action? DoubleClick; - public event Action? CursorMove; - public event Action? Scroll; + public PointerClickConfiguration ClickConfiguration { get; set; } + public event Action>? ButtonChanged; + public event Action? Click; + public event Action? DoubleClick; + public event Action? PointChanged; + public event Action? MouseScroll; } public partial class Keyboards : IReadOnlyList { - public event Action? KeyDown; - public event Action? KeyUp; + public event Action? KeyChanged; public event Action? KeyChar; } public partial class Gamepads : IReadOnlyList { - public event Action? ButtonDown; - public event Action? ButtonUp; + public event Action>? ButtonChanged; public event Action? ThumbstickMove; public event Action? TriggerMove; } public partial class Joysticks : IReadOnlyList { - public event Action? ButtonDown; - public event Action? ButtonUp; + public event Action>? ButtonChanged; public event Action? AxisMove; public event Action? HatMove; } ``` -All events will be raised when their matching handler methods are called, with the exception of `Click` and `DoubleClick` which are implemented on top of `ButtonDown` and `ButtonUp` respectively (as in 2.X). +All events will be raised when their matching handler methods are called, with the exception of `Click` and `DoubleClick` which are implemented on top of `ButtonChanged` (as in 2.X). -`DoubleClick` will be raised if `Mice.ButtonDown` is raised two consecutive times within `MouseClickConfiguration.DoubleClickTime` milliseconds, and the `MouseState.Position`'s `X` or `Y` did not change more than `MouseClickConfiguration.DoubleClickRange` between the two events. If these conditions are not met, `Click` is raised instead. For the avoidance of doubt, the behaviour of the click implementation here is exactly as it is in 2.X. +`DoubleClick` will be raised if `Pointers.ButtonChanged` is raised two consecutive times with `IsDown` set to true within `MouseClickConfiguration.DoubleClickTime` milliseconds, and the `MouseState.Position`'s `X` or `Y` did not change more than `MouseClickConfiguration.DoubleClickRange` between the two events. If these conditions are not met, `Click` is raised instead. For the avoidance of doubt, the behaviour of the click implementation here is exactly as it is in 2.X. **INFORMATIVE TEXT:** The click implementation may also even be exactly the same implementation as it is 2.X copied and pasted into 3.0, given a lot of research and effort went into this by the community contributor that implemented it. @@ -255,34 +273,39 @@ This will be configurable on `Mice` (i.e. via `InputContext.Mice.ClickConfigurat Unlike 1.0 and 2.0, this proposal uses `readonly record struct`s as their only argument for the event action. This allows us to provide more information to the event handlers without breaking in the future. These types are farily simple: ```cs -public readonly record struct ConnectionEvent(IInputDevice Device, bool IsConnected); -public readonly record struct KeyDownEvent(IKeyboard Keyboard, Key Key, bool IsRepeat); -public readonly record struct KeyUpEvent(IKeyboard Keyboard, Key Key); -public readonly record struct KeyCharEvent(IKeyboard Keyboard, char Character); -public readonly record struct MouseDownEvent(IMouse Mouse, Vector2 Position, MouseButton Button); -public readonly record struct MouseUpEvent(IMouse Mouse, Vector2 Position, MouseButton Button); -public readonly record struct MouseMoveEvent(IMouse Mouse, Vector2 Position, Vector2 Delta); -public readonly record struct MouseScrollEvent(IMouse Mouse, Vector2 Position, Vector2 WheelPosition, Vector2 Delta); -public readonly record struct MouseClickEvent(IMouse Mouse, Vector2 Position, MouseButton Button); -public readonly record struct JoystickDownEvent(IJoystick Joystick, JoystickButton Button); -public readonly record struct JoystickUpEvent(IJoystick Joystick, JoystickButton Button); -public readonly record struct JoystickHatMoveEvent(IJoystick, Vector2 Value, Vector2 Delta); -public readonly record struct JoystickAxisMoveEvent(IJoystick Joystick, int Axis, float Value, float Delta); -public readonly record struct GamepadDownEvent(IGamepad Gamepad, JoystickButton Button); -public readonly record struct GamepadUpEvent(IGamepad Gamepad, JoystickButton Button); -public readonly record struct GamepadThumbstickMoveEvent(IJoystick, Vector2 Value, Vector2 Delta); -public readonly record struct GamepadTriggerMoveEvent(IJoystick Joystick, int Axis, float Value, float Delta); +public readonly record struct ConnectionEvent(IInputDevice Device, long Timestamp, bool IsConnected); +public readonly record struct KeyChangedEvent(IKeyboard Keyboard, long Timestamp, Button Key, Button Previous, bool IsRepeat, KeyModifiers Modifiers); +public readonly record struct KeyCharEvent(IKeyboard Keyboard, long Timestamp, char? Character); +public readonly record struct ButtonChangedEvent(IButtonDevice Device, long Timestamp, Button Button, Button Previous) where T : struct, Enum; +public readonly record struct PointChangedEvent(IPointer Pointer, long Timestamp, TargetPoint? OldPoint, TargetPoint Point); +public readonly record struct PointerGripChangedEvent(IPointer Pointer, long Timestamp, float GripPressure, float Delta); +public readonly record struct MouseScrollEvent(IMouse Mouse, long Timestamp, TargetPoint Point, Vector2 WheelPosition, Vector2 Delta); +public readonly record struct PointerClickEvent(IPointer Pointer, long Timestamp, TargetPoint Point, MouseButton Button); +public readonly record struct JoystickHatMoveEvent(IJoystick Joystick, long Timestamp, Vector2 Value, Vector2 Delta); +public readonly record struct JoystickAxisMoveEvent(IJoystick Joystick, long Timestamp, int Axis, float Value, float Delta); +public readonly record struct GamepadThumbstickMoveEvent(IGamepad Gamepad, long Timestamp, Vector2 Value, Vector2 Delta); +public readonly record struct GamepadTriggerMoveEvent(IGamepad Gamepad, long Timestamp, int Axis, float Value, float Delta); ``` +`Timestamp` shall be the `Stopwatch.GetTimestamp()` at which the event was raised. This allows the user to get the +precise time of the event's occurrence, which is not otherwise possible given the requirement for state changes to be +enacted only upon a call to `Update`. + This is the part of this proposal that incorporates the ideas in Enhanced Input Events, and is why this proposal supersedes that one. -One final point to note is that throughout the rest of the proposal the following type will be used: +One final point to note is that throughout the rest of the proposal the following types will be used: ```cs public struct InputReadOnlyList : IReadOnlyList { public InputReadOnlyList(IReadOnlyList other); } + +public struct ButtonReadOnlyList : IReadOnlyList> where T : struct, Enum +{ + public ButtonReadOnlyList(IReadOnlyList> other); + public Button this[T name] { get; } +} ``` The Silk.NET team wishes to reserve the right to add more constructors to this type as it sees fit. @@ -291,84 +314,210 @@ This exists so that, should the Silk.NET choose to, we can optimize the lookup o **INFORMATIVE TEXT:** For example, for joystick and mouse buttons we could use a fixed-sized bit buffer where each bit represents an individual button: 1 for pressed, 0 for unpressed. But for something like keyboard input where there are a large amount of keys, we can't do that and will likely use `Memory` instead. -# Mouse Input +# Devices with Buttons -As discussed earlier, the interface will be very simple. +We have decided to converge functionality relating to button presses given that these are common to most device types. ```cs -public interface IMouse : IInputDevice +public readonly record struct Button(T Name, bool IsDown, float Pressure) where T : struct, Enum { - ref readonly MouseState State { get; } - ICursorConfiguration Cursor { get; } - void SetPosition(Vector2 pos); + public static implicit operator bool(Button state) => state.IsDown; +} + +public interface IButtonDevice : IInputDevice +{ + ButtonReadOnlyList State { get; } } ``` -`State` is the device state as defined earlier. +A common input handler will be exposed for these types: +```cs +public interface IButtonInputHandler where T : struct, Enum +{ + void HandleButtonChanged(ButtonChangedEvent @event); +} +``` -`Cursor` contains the cursor configuration. This isn't actually state that the end user can change, and has been made an interface rather than a state struct accordingly. +`HandleButtonChanged` must be called when any of the `Button` properties are changed. -`SetPosition` allows moving the mouse cursor without the end user physically moving their mouse. Please note that this does not immediately update `State` with the new value - the changes will be reflected next time `IInputBackend.Update` is called. +# Pointer Input -The device state returned by `State` fills out the following structure: +As discussed earlier, the interface will be very simple. ```cs -public readonly record struct MouseState -( - MouseButtonState Buttons, - Vector2 Position, - Vector2 WheelPosition -); +public interface IPointer : IButtonDevice +{ + PointerState State { get; } + ButtonReadOnlyList IButtonDevice.State => State.Buttons; + IReadOnlyList Targets { get; } +} ``` -`MouseButtonState` is defined as: +`State` is the device state as defined earlier. + +`Targets` defines the targets this pointer device can naturally point to. For a touch screen, this could be a list of +displays. For a mouse, this will contain a target for the windowed cursor mode and a target for the "raw mouse input" +mode. + +A "pointer" is an abstraction over any mechanism by which a user can point to specific coordinates on a "target". +For instance, a user could use a mouse cursor to point to a target representing the window bounds. Alternatively, that +mouse could be used to point to an arbitrary place in an infinitely large ("unbounded" herein) target (this is "raw +mouse input"). Other examples of this abstraction's application are fingers being used to point to a specific place on a +touch surface, or pens pointing to a specific place on a surface. The base abstraction is deliberately extremely vague +to meet the need of "getting a position within certain bounds," which is all people really want in most cases for a +"cursor" anyway. + +A target is defined as follows: ```cs -public readonly record struct MouseButtonState -( - InputReadOnlyList Down -) +public interface IPointerTarget { - public bool this[MouseButton btn] { get; } + /// + /// The minimum position of points on this target, where represents the lack + /// of a lower bound on a particular axis and 0 represents an unused axis if is + /// also 0 for that axis. + /// + Vector3 MinPosition { get; } + + /// + /// The maximum position of points on this target, where represents the lack + /// of an upper bound on a particular axis and 0 represents an unused axis if is + /// also 0 for that axis. + /// + Vector3 MaxPosition { get; } + + /// + /// An optional name describing the target. + /// + string? Name { get; } + + /// + /// Gets the number of points with which the given pointer is pointing at this target. + /// + /// The number of points. + /// + /// A single "logical" pointer device may have many points, and can optionally represent multiple physical pointers + /// as a single logical device - this is the case where a backend supports multiple mice to control an + /// on its "raw mouse input" target, but combines these all to a single point on its + /// "windowed" target. This is also true for touch input - a touch screen is represented as a single touch device, + /// where each finger is its own point. + /// + int GetPointCount(IPointer pointer); + + /// + /// Gets a point with which the given pointer is pointing at this target. + /// + /// The pointer device. + /// + /// The index of the point, between 0 and the number sourced from . + /// + /// The point at the given index with which the given pointer device is pointing at the target. + TargetPoint GetPoint(IPointer pointer, int point); } ``` -The indexer returns `true` if a particular button is pressed, false otherwise. If the developer wishes to enumerate the button state, they must explicitly enumerate through the `Down` buttons. +**INFORMATIVE TEXT**: Furthermore, it is our eventual goal to be able to support considering VR hands as pointer devices +through raycasting. Such a future proposal will involve a way to "fork" a `IPointerTarget` from that which represents +the 3D world (i.e. the entire VR world is a target, and the point representing the hand is _within_ that target - with +the `TargetPoint` being populated using `XrPosef` values), where calculation of points on forked targets are calculated +using raycasting from that position. -**INFORMATIVE TEXT:** This struct only exists so we can implement an indexer that accepts a `MouseButton`, given that `Down` is effectively just a list and only takes an `int` index as a result. +The functionality of these APIs are described in the XML documentation inline. -The indexer will be implemented in terms of `Down`, which is the only property that a backend will need to set. +A point shall be defined as follows: +```cs +/// +/// Flags describing a state. +/// +[Flags] +public enum TargetPointFlags +{ + /// + /// No flags are set, indicating that the point is not being pointed at and therefore may not be valid. + /// + NotPointingAtTarget = 0, + + /// + /// Indicates that the point has been resolved as a valid point at which the pointer is pointing. + /// + PointingAtTarget = 1 << 0 +} -Changes to `MouseState` also have matching handler methods which are subject to the handler method rules i.e. the backend should call them in the order in which the backend received the events where possible etc (read the Input Handlers section). +/// +/// Represents a point on a target at which a pointer is pointing. +/// +/// Flags describing the state of the point. +/// The absolute position on the target at which the pointer is pointing. +/// +/// The normalized position on the target at which the pointer is pointing, if applicable. If this is not available +/// (e.g. due to the target being infinitely large a.k.a. "unbounded"), then this property shall have a value of +/// default. +/// +/// +/// The angle at which the pointer is pointing at the point on the target. An identity quaternion shall be interpreted +/// as the point directly perpendicular to and facing towards the target. This shall carry an identity quaternion if +/// there is no orientation available. +/// +/// The distance of the pointer from the point the pointer is pointing at. +/// +/// The pressure applied to the point on the target by the pointer, between 0.0 representing the minimum amount +/// of pressure and 1.0 representing the maximum amount of pressure. This shall be 1.0 if such data is +/// unavailable but the point is otherwise valid. +/// +public readonly record struct TargetPoint( + TargetPointFlags Flags, + Vector3 Position, + Vector3 NormalizedPosition, + Quaternion Orientation, + Vector3 Distance, + float Pressure +) { + public bool IsValid => (Flags & Flags.PointingAtTarget) != Flags.NotPointingAtTarget; +} +``` +The `PointerState` shall be defined as follows: ```cs -public interface IMouseInputHandler : IInputHandler +public class PointerState { - void HandleButtonDown(MouseDownEvent @event); - void HandleButtonUp(MouseUpEvent @event); - void HandleCursorMove(MouseCursorEvent @event); - void HandleScroll(MouseScrollEvent @event); + public ButtonReadOnlyList Buttons { get; } + public InputReadOnlyList Points { get; } + public float GripPressure { get; } } + +public readonly record struct PointerStatePoint(IPointerTarget Target, TargetPoint Point); ``` -`HandleButtonDown` must be called when a button is added to `MouseState.Buttons.Down`. +`Points` represents the `TargetPoint`s this pointer is pointing at on its "native targets" i.e. that which is enumerated +by `IPointer.Targets`. -`HandleButtonUp` must be called when a button is removed from `MouseState.Buttons.Down`. +`GripPressure` represents the amount of pressure the user is applying to the device itself (e.g. the pen barrel) between +`0.0` and `1.0`. This shall be `1.0` if unavailable. -`HandleCursorMove` must be called when `MouseState.Position` changes. +Additional APIs to construct `PointerState` will be added as appropriate. -`HandleScroll` must be called when `MouseState.WheelPosition` changes. +Changes to `PointerState` also have matching handler methods which are subject to the handler method rules i.e. the backend should call them in the order in which the backend received the events where possible etc (read the Input Handlers section). -Note that the click events, just as in 2.X, are not implemented by the backend and instead implemented by the input context because it is not a requirement that backends can record clicks. **INFORMATIVE TEXT:** The original reason for this requirement in 2.X is because GLFW doesn't actually send click and double click events. +The handler for pointer inputs shall be defined as follows: +```cs +public interface IPointerInputHandler : IButtonInputHandler +{ + void HandlePointChanged(PointChangedEvent @event); + void HandleGripChanged(PointerGripChangedEvent @event); +} +``` -## Enums +`HandlePointChanged` must be called when a point within `PointerState.Points` changes. + +`HandleGripChanged` must be called when `PointerState.GripPressure` changes. +`PointerButton` shall be defined as follows: ```cs -public enum MouseButton +public enum PointerButton { - Unknown, - LeftButton, - RightButton, - MiddleButton, + Primary, + Secondary, + Button3, + MiddleButton = Button3, Button4, Button5, Button6, @@ -396,24 +545,73 @@ public enum MouseButton Button28, Button29, Button30, - Button31 + EraserTip = Button30, + Button31, + Button32 +} +``` + +There will be derived types for different types of pointers. + +## Mouse Input + +```cs +public interface IMouse : IPointer +{ + MouseState State { get; } + PointerState IPointer.State => State; + ICursorConfiguration Cursor { get; } + bool TrySetPosition(Vector2 position); +} +``` + +`Cursor` contains the cursor configuration. This isn't actually state that the end user can change, and has been made an interface rather than a state struct accordingly. + +`TrySetPosition` allows moving the mouse cursor without the end user physically moving their mouse. Please note that this does not immediately update `State` with the new value - the changes will be reflected next time `IInputBackend.Update` is called. + +The device state returned by `State` fills out the following structure: + +```cs +public class MouseState : PointerState +{ + public Vector2 WheelPosition { get; } } ``` +Additional APIs to construct `MouseState` will be added as appropriate. + +Changes to `MouseState` also have matching handler methods which are subject to the handler method rules i.e. the backend should call them in the order in which the backend received the events where possible etc (read the Input Handlers section). + +```cs +public interface IMouseInputHandler : IButtonInputHandler +{ + void HandleScroll(MouseScrollEvent @event); +} +``` + +`HandleScroll` must be called when `MouseState.WheelPosition` changes. + +Note that the click events, just as in 2.X, are not implemented by the backend and instead implemented by the input context because it is not a requirement that backends can record clicks. **INFORMATIVE TEXT:** The original reason for this requirement in 2.X is because GLFW doesn't actually send click and double click events. + ## Cursor Configuration `ICursorConfiguration` is defined as: ```cs +public readonly ref struct CustomCursor +{ + public int Width { get; init; } + public int Height { get; init; } + public ReadOnlySpan Data { get; init; } // Rgba32 +} + public interface ICursorConfiguration { CursorModes SupportedModes { get; } CursorModes Mode { get; set; } CursorStyles SupportedStyles { get; } CursorStyles Style { get; set; } - CursorFlags SupportedFlags { get; } - CursorFlags Flags { get; set; } - RawImage? Image { get; set; } + CustomCursor Image { get; set; } } ``` @@ -423,7 +621,7 @@ Please note that the `Hotspot` properties present in 1.X and 2.0 have been remov `SupportedStyles` is a bitmask containing all of the cursor styles that are supported by this backend. This must be queried before setting `Style` - the currently active cursor style. An exception should be thrown if an attempt is made to set `Style` to an unsupported style or multiple styles (i.e. multiple bits set). -`Image` uses `RawImage` as-is from Silk.NET.Core, and when set to a non-null value implicitly sets `Style` to custom. As such, you must query `SupportedStyles` before using this property as well. Setting `Image` to `null` will set `Style` back to a standard cursor style, defined by the implementation. It is therefore recommended you set `Style` explicitly when disabling a custom cursor. Note that setting `Style` to a non-`Custom` value will also implicitly set this property to `null`. Setting `Mode` **to** `Custom` explicitly is undefined behaviour, as `Image` won't be set at the time of setting `Mode`. +`Image` when set to a non-`default` value implicitly sets `Style` to custom. As such, you must query `SupportedStyles` before using this property as well. Setting `Image` to `default` will set `Style` back to a standard cursor style, defined by the implementation. It is therefore recommended you set `Style` explicitly when disabling a custom cursor. Note that setting `Style` to a non-`Custom` value will also implicitly set this property to `default`. Setting `Mode` **to** `Custom` explicitly is undefined behaviour, as `Image` won't be set at the time of setting `Mode`. `SupportedFlags` is a bitmask containing other supported options for the cursor which can be mixed and matched if supported. This must be queried before setting `Flags` - the currently active options. An exception should be thrown if an attempt is made to set an unsupported flag on `Flags`. Unlike the other properties, `Flags` can have multiple bits set. @@ -435,10 +633,9 @@ Please note that the `Hotspot` properties present in 1.X and 2.0 have been remov [Flags] public enum CursorModes { - Normal, - Hidden = 1 << 0, - Disabled = 1 << 1, - Raw = 1 << 2 + Normal = 1 << 0, + Confined = 1 << 1, + Unbounded = 1 << 2, } ``` ```cs @@ -452,7 +649,8 @@ public enum CursorStyles Hand = 1 << 3, HResize = 1 << 4, VResize = 1 << 5, - Custom = 1 << 6, + Hidden = 1 << 6, + Custom = 1 << 7, } ``` ```cs @@ -469,10 +667,11 @@ public enum CursorFlags Once again, the interface is very simple. ```cs -public interface IKeyboard : IInputDevice +public interface IKeyboard : IButtonDevice { - ref readonly KeyboardState State { get; } + KeyboardState State { get; } string? ClipboardText { get; set; } + bool TryGetKeyName(KeyName key, [NotNullWhen(true)] out string? name); void BeginInput(); void EndInput(); } @@ -487,197 +686,344 @@ public interface IKeyboard : IInputDevice `KeyboardState` is defined as follows: ```cs -public readonly record struct KeyboardState -( - InputReadOnlyList? Text, - KeyState Keys -); -``` - -`Text` contains the characters typed on the keyboard since `IKeyboard.BeginInput`, and accounts for backspaces. This is cleared (set to `null`) when `IKeyboard.EndInput` is called, and will not be non-`null` again until another `IKeyboard.BeginInput` call. Given that `KeyChar` events are raised one character at a time, this property will update one character at a time to keep the state consistent with the event. - -**INFORMATIVE TEXT:** This is something we can optimize in `InputList` to not be allocatey, rest assured it is not acceptable to the Silk.NET team to allocate a new list for every character. - -```cs -public readonly record struct KeyState -( - InputReadOnlyList Down -) +public class KeyboardState { - public bool this[KeyName btn] { get; } - public bool this[int scancode] { get; } + public InputReadOnlyList? Text { get; } + public ButtonReadOnlyList Keys { get; } + public KeyModifiers Modifiers { get; } } ``` -The indexer returns `true` if a particular key is pressed, false otherwise. If the developer wishes to enumerate the key state, they must explicitly enumerate through the `Down` buttons. - -**INFORMATIVE TEXT:** This struct only exists so we can implement an indexer that accepts a `KeyName` or scancode, given that `Down` is effectively just a list and only takes an `int` index as a result. - -The indexer will be implemented in terms of `Down`, which is the only property that a backend will need to set. - -Note because not all keys are named, and because some developers may prefer to use scancodes instead, a `Key` struct is used instead of just having the list be a list of key names. - -```cs -public readonly record struct Key(KeyName Name, int Scancode); -``` +`Text` contains the characters typed on the keyboard since `IKeyboard.BeginInput`, and accounts for backspaces. This is cleared (set to `null`) when `IKeyboard.EndInput` is called, and will not be non-`null` again until another `IKeyboard.BeginInput` call. Given that `KeyChar` events are raised one character at a time, this property will update one character at a time to keep the state consistent with the event. -`KeyName` will be `Unknown` for scancode-only, unnamed keys. +**INFORMATIVE TEXT:** This is something we can optimize in `InputList` to not be allocatey, rest assured it is not acceptable to the Silk.NET team to allocate a new list for every character. Changes to `KeyboardState` also have matching handler methods which are subject to the handler method rules i.e. the backend should call them in the order in which the backend received the events where possible etc (read the Input Handlers section). ```cs -public interface IKeyboardInputHandler : IInputHandler +public interface IKeyboardInputHandler : IButtonInputHandler { - void HandleKeyDown(KeyDownEvent @event); - void HandleKeyUp(KeyUpEvent @event); + void HandleKeyChanged(KeyChangedEvent @event); void HandleKeyChar(KeyCharEvent @event); } ``` -`HandleKeyDown` must be called when a `Key` is added to the `KeyState.Down` list. - -`HandleKeyUp` must be called when a `Key` is removed from the `KeyState.Down` list. +`HandleKeyChanged` must be called in the same circumstances as `IButtonInputHandler.HandleButtonChanged`. The +purpose of this event duplication is to provide more keyboard-specific information if the handler has a use for it. `HandleKeyChar` must be called when a character is added to `KeyboardState.Text`. - ## Enums ```cs public enum KeyName { + // These values are from usage page 0x07 (USB keyboard page). Unknown = 0, - Space, - Apostrophe /* ' */, - Comma /* , */, - Minus /* - */, - Period /* . */, - Slash /* / */, - Number0, - Number1, - Number2, - Number3, - Number4, - Number5, - Number6, - Number7, - Number8, - Number9, - Semicolon /* ; */, - Equal /* = */, - A, - B, - C, - D, - E, - F, - G, - H, - I, - J, - K, - L, - M, - N, - O, - P, - Q, - R, - S, - T, - U, - V, - W, - X, - Y, - Z, - LeftBracket /* [ */, - BackSlash /* \ */, - RightBracket /* ] */, - GraveAccent /* ` */, - Escape, - Enter, - Tab, - Backspace, - Insert, - Delete, - Right, - Left, - Down, - Up, - PageUp, - PageDown, - Home, - End, - CapsLock, - ScrollLock, - NumLock, - PrintScreen, - Pause, - F1, - F2, - F3, - F4, - F5, - F6, - F7, - F8, - F9, - F10, - F11, - F12, - F13, - F14, - F15, - F16, - F17, - F18, - F19, - F20, - F21, - F22, - F23, - F24, - F25, - Keypad0, - Keypad1, - Keypad2, - Keypad3, - Keypad4, - Keypad5, - Keypad6, - Keypad7, - Keypad8, - Keypad9, - KeypadDecimal, - KeypadDivide, - KeypadMultiply, - KeypadSubtract, - KeypadAdd, - KeypadEnter, - KeypadEqual, - ShiftLeft, - ControlLeft, - AltLeft, - SuperLeft, - ShiftRight, - ControlRight, - AltRight, - SuperRight, - Menu -} -``` - -The `KeyName` enum is exactly the same as the `Key` enum in 2.X. The integral values of each enumerant, not included here, must match the en-US scancode for that key. A backend must match a scancode to a `KeyName` as if it were an en-US scancode, as this is the keyboard layout from which these key names were derived. - -The Silk.NET team wishes to reserve the right to remove any key names which do not have a matching en-US scancode. This is because the above enum is just copied and pasted from 2.X, and has not been cross-referenced with the keyboard layout at this time. + A = 4, + B = 5, + C = 6, + D = 7, + E = 8, + F = 9, + G = 10, + H = 11, + I = 12, + J = 13, + K = 14, + L = 15, + M = 16, + N = 17, + O = 18, + P = 19, + Q = 20, + R = 21, + S = 22, + T = 23, + U = 24, + V = 25, + W = 26, + X = 27, + Y = 28, + Z = 29, + Number1 = 30, + Number2 = 31, + Number3 = 32, + Number4 = 33, + Number5 = 34, + Number6 = 35, + Number7 = 36, + Number8 = 37, + Number9 = 38, + Number0 = 39, + Return = 40, + Escape = 41, + Backspace = 42, + Tab = 43, + Space = 44, + Minus = 45, + Equals = 46, + LeftBracket = 47, + RightBracket = 48, + Backslash = 49, + NonUs1 = 50, // US: \| Belg: µ`£ FrCa: <}> Dan:’* Dutch: <> Fren:*µ Ger: #’ Ital: ù§ LatAm: }`] Nor:,* Span: }Ç Swed: , * Swiss: $£ UK: #~. + Semicolon = 51, + Apostrophe = 52, + Grave = 53, + Comma = 54, + Period = 55, + Slash = 56, + CapsLock = 57, + F1 = 58, + F2 = 59, + F3 = 60, + F4 = 61, + F5 = 62, + F6 = 63, + F7 = 64, + F8 = 65, + F9 = 66, + F10 = 67, + F11 = 68, + F12 = 69, + PrintScreen = 70, + ScrollLock = 71, + Pause = 72, + Insert = 73, + Home = 74, + PageUp = 75, + Delete = 76, + End = 77, + PageDown = 78, + Right = 79, + Left = 80, + Down = 81, + Up = 82, + NumLockClear = 83, + KeypadDivide = 84, + KeypadMultiply = 85, + KeypadMinus = 86, + KeypadPlus = 87, + KeypadEnter = 88, + Keypad1 = 89, + Keypad2 = 90, + Keypad3 = 91, + Keypad4 = 92, + Keypad5 = 93, + Keypad6 = 94, + Keypad7 = 95, + Keypad8 = 96, + Keypad9 = 97, + Keypad0 = 98, + KeypadPeriod = 99, + NonUs2 = 100, // Belg:<\> FrCa:«°» Dan:<\> Dutch:]|[ Fren:<> Ger:<|> Ital:<> LatAm:<> Nor:<> Span:<> Swed:<|> Swiss:<\> UK:\| Brazil: \|. Typically near the Left-Shift key in AT-102 implementations. + Application = 101, + Power = 102, + KeypadEquals = 103, + F13 = 104, + F14 = 105, + F15 = 106, + F16 = 107, + F17 = 108, + F18 = 109, + F19 = 110, + F20 = 111, + F21 = 112, + F22 = 113, + F23 = 114, + F24 = 115, + Execute = 116, + Help = 117, + Menu = 118, + Select = 119, + Stop = 120, + Again = 121, + Undo = 122, + Cut = 123, + Copy = 124, + Paste = 125, + Find = 126, + Mute = 127, + VolumeUp = 128, + VolumeDown = 129, + KeypadComma = 133, + OtherKeypadEquals = 134, // Equals sign typically used on AS-400 keyboards. + International1 = 135, + International2 = 136, + International3 = 137, + International4 = 138, + International5 = 139, + International6 = 140, + International7 = 141, + International8 = 142, + International9 = 143, + Lang1 = 144, + Lang2 = 145, + Lang3 = 146, + Lang4 = 147, + Lang5 = 148, + Lang6 = 149, + Lang7 = 150, + Lang8 = 151, + Lang9 = 152, + AlternativeErase = 153, // Example, Erase-Eaze™ key. + SystemRequest = 154, + Cancel = 155, + Clear = 156, + Prior = 157, + Return2 = 158, + Separator = 159, + Out = 160, + Oper = 161, + ClearAgain = 162, + // For more information on these two consult IBM's "3174 Establishment Controller - Terminal User's Reference for + // Expanded Functions" (GA23-03320-02, May 1989) + CursorSelect = 163, + ExtendSelect = 164, + Keypad00 = 176, + Keypad000 = 177, + ThousandsSeparator = 178, + DecimalSeparator = 179, + CurrencyUnit = 180, + CurrencySubunit = 181, + KeypadLeftParenthesis = 182, + KeypadRightParenthesis = 183, + KeypadLeftBrace = 184, + KeypadRightBrace = 185, + KeypadTab = 186, + KeypadBackspace = 187, + KeypadA = 188, + KeypadB = 189, + KeypadC = 190, + KeypadD = 191, + KeypadE = 192, + KeypadF = 193, + KeypadXor = 194, + KeypadPower = 195, + KeypadPercent = 196, + KeypadLess = 197, + KeypadGreater = 198, + KeypadAmpersand = 199, + KeypadDoubleAmpersand = 200, + KeypadVerticalBar = 201, + KeypadDoubleVerticalBar = 202, + KeypadColon = 203, + KeypadHash = 204, + KeypadSpace = 205, + KeypadAt = 206, + KeypadExclamation = 207, + KeypadMemoryStore = 208, + KeypadMemoryRecall = 209, + KeypadMemoryClear = 210, + KeypadMemoryAdd = 211, + KeypadMemorySubtract = 212, + KeypadMemoryMultiply = 213, + KeypadMemoryDivide = 214, + KeypadPlusMinus = 215, + KeypadClear = 216, + KeypadClearEntry = 217, + KeypadBinary = 218, + KeypadOctal = 219, + KeypadDecimal = 220, + KeypadHexadecimal = 221, + ControlLeft = 224, + ShiftLeft = 225, + AltLeft = 226, + SuperLeft = 227, + ControlRight = 228, + ShiftRight = 229, + AltRight = 230, + SuperRight = 231, + Mode = 257, + // These values are mapped from usage page 0x0C (USB consumer page). + Sleep = 258, + Wake = 259, + ChannelIncrement = 260, + ChannelDecrement = 261, + MediaPlay = 262, + MediaPause = 263, + MediaRecord = 264, + MediaFastForward = 265, + MediaRewind = 266, + MediaNextTrack = 267, + MediaPreviousTrack = 268, + MediaStop = 269, + MediaEject = 270, + MediaPlayPause = 271, + MediaSelect = 272, + ApplicationNew = 273, + ApplicationOpen = 274, + ApplicationClose = 275, + ApplicationExit = 276, + ApplicationSave = 277, + ApplicationPrint = 278, + ApplicationProperties = 279, + ApplicationSearch = 280, + ApplicationHome = 281, + ApplicationBack = 282, + ApplicationForward = 283, + ApplicationStop = 284, + ApplicationRefresh = 285, + ApplicationBookmarks = 286, + // 501-512 is reserved for non-standard (i.e. not from an industry-standard HID page) keys. + SoftLeft = 501, // Left button on mobile phones + SoftRight = 502, // Right button on mobile phones + Call = 503, + EndCall = 504, +} +``` + +```cs +[Flags] +public enum KeyModifiers +{ + None = 0, + ShiftLeft = 1 << 0, + ShiftRight = 1 << 1, + ControlLeft = 1 << 2, + ControlRight = 1 << 3, + AltLeft = 1 << 4, + AltRight = 1 << 5, + SuperLeft = 1 << 6, + SuperRight = 1 << 7, + NumLock = 1 << 8, + CapsLock = 1 << 9 +} +``` + +The `KeyName` enumerates standard USB HID usage IDs where possible - mappings to PS/2 are not included and should be +done manually based on a translation table. A `KeyName` value **must** always map to a _physical_ scancode and not be +layout-specific. In theory, no key will be missing from this enum. However, the backend **may** cast an integer value +between 1-500 to this enum in the case that there is a standard USB HID page/usage ID that hasn't been accounted for by +`KeyName` yet. The Silk.NET team reserves the right to update `KeyName` to reflect latest specifications as it sees fit, +and to add non-standard keys as deemed applicable for the userbase. + +**INFORMATIVE TEXT**: There has been lots of discussion about our localization approach. We remain of the opinion that +any such approaches should be scancode-oriented, as these are effectively immutable and standardised across every +keyboard. The intention with the `KeyName` enum is to reflect a set of scancodes that would be recognizable to an +English developer, such that they could map to what is WASD using this enum, and have that automatically translated to +the keys in the same physical location on the end user's keyboard regardless of keyboard layout in use. This is also why +`IKeyboard` exposes `GetKeyName`, as this will localize this QWERTY-biased enum to whatever the equivalent key is in the +same physical position for the end user (e.g. for configuration UIs). We believe this presents a natural approach for +both the developer and the end user. + +**FUTURE IMPROVEMENT**: We obviously acknowledge that this does not account for non-English developers, for which we +could investigate adding more `KeyName` enums for different keyboard layouts in the future i.e. so those developers can +develop their application in terms of their native layout. + +**INFORMATIVE TEXT**: There has been some questions on whether we should expose options to generalise keyboard input +even further to common key bindings or common use cases. Such ideas have included providing a way to consider a +`IKeyboard`/`IPointer` combination as an `IGamepad` implicitly, or having a `KeyName`-like enum that has more generic +names that are more closely aligned with the user's use case e.g. instead of `W` we have `Forward`. Both ideas are +shelved for now, but we believe we will explore the former in future proposals (namely Axis-Based Input and/or Input Actions). # Gamepad Input ```cs -public interface IGamepad : IInputDevice +public interface IGamepad : IButtonDevice { - ref readonly GamepadState State { get; } + GamepadState State { get; } + ButtonReadOnlyList IButtonDevice.State => State.Buttons; IReadOnlyList VibrationMotors { get; } } ``` @@ -700,12 +1046,12 @@ This is exactly as in 2.X. `GamepadState` is defined as follows: ```cs -public readonly record struct GamepadState -( - JoystickButtonState Buttons, - DualReadOnlyList Thumbsticks, - DualReadOnlyList Triggers, -); +public class GamepadState +{ + public ButtonReadOnlyList Buttons { get; } + public DualReadOnlyList Thumbsticks { get; } + public DualReadOnlyList Triggers { get; } +} ``` `GamepadState` reuses a lot of the joystick API types, which are defined later in this proposal. @@ -728,19 +1074,13 @@ This is used where the list will only ever have exactly two elements, mainly bec Changes to `GamepadState` also have matching handler methods which are subject to the handler method rules i.e. the backend should call them in the order in which the backend received the events where possible etc (read the Input Handlers section). ```cs -public interface IGamepadInputHandler : IInputHandler +public interface IGamepadInputHandler : IButtonInputHandler { - void HandleButtonDown(GamepadDownEvent @event); - void HandleButtonUp(GamepadUpEvent @event); void HandleThumbstickMove(GamepadThumbstickMoveEvent @event); void HandleTriggerMove(GamepadTriggerMoveEvent @event); } ``` -`HandleButtonDown` must be called when a button is added to `GamepadState.Buttons.Down`. - -`HandleButtonUp` must be called when a button is removed from `GamepadState.Buttons.Down`. - `HandleThumbstickMove` must be called when any value of `GamepadState.Thumbsticks` changes. `HandleTriggerMove` must be called when any value of `GamepadState.Triggers` changes. @@ -750,17 +1090,18 @@ public interface IGamepadInputHandler : IInputHandler This is the polyglot interface for any other human input device that roughly meets the description of being "joystick". ```cs -public interface IJoystick : IInputDevice +public interface IJoystick : IButtonDevice { - ref readonly JoystickState State { get; } + JoystickState State { get; } + ButtonReadOnlyList IButtonDevice.State => State.Buttons; } ``` ```cs -public readonly record struct JoystickState +public class JoystickState { - InputReadOnlyList Axes, - JoystickButtonState Buttons, - InputReadOnlyList Hats + public InputReadOnlyList Axes { get; } + public ButtonReadOnlyList Buttons { get; } + public InputReadOnlyList Hats { get; } } ``` @@ -768,23 +1109,6 @@ This is pretty closely modeled as in 2.X: `Axes` containing the individual axes **INFORMATIVE TEXT:** The only difference is `Hats` is now a `Vector2` instead of a `Position2D`. It is still intended that the X and Y values are only ever `0` or `1`, but this is not a requirement for more exotic backends. -`JoystickButtonState` is defined as follows: -```cs -public readonly record struct JoystickButtonState -( - InputReadOnlyList Down -) -{ - public bool this[JoystickButton btn] { get; } -} -``` - -The indexer returns `true` if a particular button is pressed, false otherwise. If the developer wishes to enumerate the button state, they must explicitly enumerate through the `Down` buttons. - -**INFORMATIVE TEXT:** This struct only exists so we can implement an indexer that accepts a `JoystickButton`, given that `Down` is effectively just a list and only takes an `int` index as a result. - -The indexer will be implemented in terms of `Down`, which is the only property that a backend will need to set. - `JoystickButton` is defined as follows: ```cs public enum JoystickButton @@ -815,19 +1139,13 @@ public enum JoystickButton Changes to `JoystickState` also have matching handler methods which are subject to the handler method rules i.e. the backend should call them in the order in which the backend received the events where possible etc (read the Input Handlers section). ```cs -public interface IJoystickInputHandler : IInputHandler +public interface IJoystickInputHandler : IButtonInputHandler { - void HandleButtonDown(JoystickDownEvent @event); - void HandleButtonUp(JoystickUpEvent @event); void HandleAxisMove(JoystickAxisMoveEvent @event); void HandleHatMove(JoystickHatMoveEvent @event); } ``` -`HandleButtonDown` must be called when a button is added to `JoystickState.Buttons.Down`. - -`HandleButtonUp` must be called when a button is removed from `JoystickState.Buttons.Down`. - `HandleAxisMove` must be called when any value of `JoystickState.Axes` changes. `HandleHatMove` must be called when any value of `JoystickState.Hats` changes.