Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge the Windowing proposal #5

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 0 additions & 23 deletions wit/animation-frame.wit

This file was deleted.

33 changes: 0 additions & 33 deletions wit/key-events.wit

This file was deleted.

35 changes: 0 additions & 35 deletions wit/mini-canvas.wit

This file was deleted.

36 changes: 0 additions & 36 deletions wit/pointer-events.wit

This file was deleted.

37 changes: 37 additions & 0 deletions wit/raw-display.wit
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
package wasi:webgpu;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How's this different from frame-buffer.wit?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a display API, frame-buffer is a graphics API. But you're right, the intended use is to connect a display to a frame-buffer to draw into, primarily for embedded devices. The difference is that frame-buffer is an arbitrary framebuffer, but this maps to hardware displays and their framebuffers. This is just a proof that multiple display APIs are supported, we don't have to actually include it.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason we can't use frame-buffer for both?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

frame-buffer has no way to specify a screen, or which screen.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see.
But I'd much rather have one interface for both.
Wanna open a separate issue or PR to discuss?



/// The raw-display interface is designed for when an application has raw access to the full display framebuffer, such as on an embedded device.
interface raw-display {

enum framebuffer-format {
rgb888,
}

/// Gets information on the connected displays.
displays: func() -> list<display-info>;


record display-info {
/// Internal id for the display, used to connect a grapics-context.
id: u32,
/// An implementation defined identifier of the display. Should be unique.
/// This is so that an application can differentiate e.g. between front and back screens on a device.
name: string,
width: u32,
height: u32,
/// The framebuffer format used.
format: framebuffer-format,
/// Whether the drawing is done in the main framebuffer or in a back buffer.
/// When drawing in a back buffer, blitting is neccessary.
back-buffer: bool,
}

/// Connects a graphics context to a fisplay.
connect-graphics-context: func(context: borrow<graphics-context>, display-id: u32);

/// Swaps the front and back buffer of a display, if there is one.
/// TODO: should this invalidate the grapics-context connection? Since the connected buffer changes.
blit: func(display-id: u32);

}
228 changes: 228 additions & 0 deletions wit/window.wit
Original file line number Diff line number Diff line change
@@ -0,0 +1,228 @@
package wasi:webgpu;

interface window {
use graphics-context.{graphics-context};
use wasi:io/[email protected].{pollable};


/// https://developer.mozilla.org/en-US/docs/Web/API/UI_Events/Keyboard_event_code_values
enum key-code {
// TODO is this needed? I don't think the MVP needs key codes.
unidentified,
}

/// https://developer.mozilla.org/en-US/docs/Web/API/UI_Events/Keyboard_event_key_values#document_keys
/// Excerpt of the most useful keys.
enum special-key {
alt,
alt-graph,
caps-lock,
ctrl,
meta,
num-lock,
scroll-lock,
shift,

arrow-down,
arrow-left,
arrow-right,
arrow-up,
end,
home,
page-down,
page-up,

backspace,
clear,
delete,
insert,
paste,

context-menu,
escape,
pause,

brightness-down,
brightness-up,
print,

f1,
f2,
f3,
f4,
f5,
f6,
f7,
f8,
f9,
f10,
f11,
f12,
f13,
f14,
f15,
f16,
f17,
f18,
f19,
f20,

app-switch,

channel-down,
channel-up,
fast-forward,
media-pause,
media-play,
media-play-pause,
media-record,
media-rewind,
media-stop,
media-track-next,
media-track-previous,

audio-volume-down,
audio-volume-up,
audio-volume-mute,
}

flags modifiers {
ctrl,
alt,
alt-graph,
shift,
caps-lock,
meta,
}

/// Describes the key location on the keyboard, for keys that appear more than once on a keyboard.
enum key-location {
standard,
left,
right,
numpad,
}

variant key {
/// The key has an associated Unicode scalar value.
character(u32),
/// The key is a key without a Unicode scalar value.
special(special-key),
/// A dead key was pressed. The next key event will contain the combined character.
dead,
/// The key could not be identified.
unidentified,
}


record key-data {
modifiers: modifiers,
key: key,
code: key-code,
location: key-location,
repeat: bool,
}

flags mouse-buttons {
left,
middle,
right,
}

record mouse {
position: position,
buttons: mouse-buttons,
}

record scroll {
mouse: mouse,
MendyBerger marked this conversation as resolved.
Show resolved Hide resolved
x: f32,
y: f32,
}


/// Window system capabilities that are supported.
/// Methods that use unsupported capabilities will trap in WASM.
flags capabilities {
tareksander marked this conversation as resolved.
Show resolved Hide resolved
/// The windows have a title that can be set.
title,
/// The windows have position on a display which can be moved by the program. Not given e.g. on the web, where position is handled by CSS.
position,
// The windows have icons which can be set by the program.
icon,
/// Windows can request to go into fullscreen mode. This capability not being present can mean fullscreen windows
/// are not supported, or the window is forced fullscreen.
fullscreen,
}


record position {
x: u16,
y: u16,
}

record size {
width: u16,
height: u16,
}

record create-desc {
title: string,
position: position,
size: size,
visible: bool,
}

/// Checks whether window functionality is available at all in the runtime.
available: func() -> bool;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does this mean? Why would you even be able to get here, when there's no windowing capabilities?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. You talked about the possibility of having multiple presentation APIs, and I also added an API intended for raw framebuffer access on connected screens, for embedded devices. If this were Rust, I'd just make the window constructor return an option, but I don't think that's possible in WIT?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wit does support options: e.g. option<bool> instead of bool.

But I'm not sure I'm following why this is necessary. If another presentation api is used, how would you reach this point at all?
You would use wasi-windowing-other instead of wasi-windowing, and wasi-windowing-other would have a completely different set of functions.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently the unit of implementation is a world. So if something is included in the world, you have to implement it, even if it's stubbed. Since I can't make the constructor fail gracefully, there needs to be a function to query runtime support. Of course each display API could get its own world, too.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this were Rust, I'd just make the window constructor return an option, but I don't think that's possible in WIT?

The way to do that in WIT is to use a static factory function:

ctor: static func([...]) -> option<the-type>;

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Of course each display API could get its own world, too.

That's what I'd prefer.


resource window {
constructor(desc: create-desc);

connect-graphics-context: func(context: borrow<graphics-context>);

set-title: func(title: string);
get-title: func() -> string;

// TODO add a display interface for querying the displays, so programs can see the resolution and find out where on which screen the window is.
set-position: func(position: position);
get-position: func() -> position;

set-size: func(size: size);
get-size: func() -> size;

/// Sets the visibility of the window.
set-visible: func(visible: bool);

subscribe-resize: func() -> pollable;
get-resize: func() -> option<size>;

subscribe-reposition: func() -> pollable;
get-reposition: func() -> option<position>;


subscribe-key-down: func() -> pollable;
get-key-down: func() -> option<key>;

subscribe-key-up: func() -> pollable;
get-key-up: func() -> option<key>;


subscribe-mouse-down: func() -> pollable;
get-mouse-down: func() -> option<mouse>;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason you're calling these mouse instead of pointer? I would opt for pointer, like the web, because we wanna cover all kinds of inputs, like touch and pencil.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Touch input is a bit more complicated, with multi touch and all, I haven't looked at touch yet. There will be separate events for pointers, but implementations are free to convert e.g. a tap into mouse-down + mouse-up.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd start with pointer since it covers all the basics (mouse, pen, touch). We can later add in things like multi touch, but I'd prefer if we can start with something broad rather than something narrow.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

我会从指针开始,因为它涵盖了所有基本功能(鼠标、笔、触摸)。我们稍后可以添加多点触摸等功能,但我更希望我们能从广泛的内容开始,而不是从狭窄的内容开始。

strongly agree like https://developer.mozilla.org/zh-CN/docs/Web/API/Pointer_events


subscribe-mouse-up: func() -> pollable;
get-mouse-up: func() -> option<mouse>;

subscribe-mouse-move: func() -> pollable;
get-mouse-move: func() -> option<mouse>;

subscribe-scroll: func() -> pollable;
get-scroll: func() -> option<scroll>;
tareksander marked this conversation as resolved.
Show resolved Hide resolved


// If the pollable is ready, that means a close event has occurred. Since there is no data, ther e is no need
// for a get function, and the ready status should be cleared on query.
subscribe-close: func() -> pollable;
}
}