Cross platform support

[drmfinlay]

There have been discussions on making dragonfly more cross platform by using packages that support cross platform user input and window management on Windows and Linux (X11). Aenea already does a lot of this in its server implementations, so it may be a good starting point.

TODO

• Implement cross-platform clipboard class using pyperclip.
  The implementation is located in dragonfly.util.clipboard.

• Implement the Key and Text action for X11 and Mac OS using pynput or PyUserInput.

• Implement Mouse actions using pynput.
  The implementation should allow using the same key names and specs for Mouse as on Windows. Other platforms should be supported by moving code into other modules:
  - MouseBase action and related classes: actions/mouse/_base.py
  - Windows Mouse action sub-class: actions/mouse/_win32.py
  - X11/MacOS Mouse action sub-class: actions/mouse/_pynput.py

• Implement any utility functions for input actions such as the get_cursor_position() and set_cursor_position() functions in action_mouse.py.

• Implement Monitor classes for X11 and Mac OS.

• Implement an X11 equivalent for dragonfly's Window class for window management using libxdo.
  python-xlib could also be used, but libxdo handles checking various things like what information the window manager can give us about windows (i.e. window manager hints).

• Implement a Mac OS equivalent for the Window class.
  As I mentioned in Window class implementations for other platforms #35, there are some Swift projects for doing this. It would probably be better to use pyobjc though, as it is a project requirement on Mac OS now (for Monitor).

• Implement what is possible for Wayland.
  See notes and comments below.

Notes

• The Mouse action requires dragonfly's Window class to be at least partially implemented in order for relative mouse movements to work.
• Wayland doesn't currently allow emulating keyboard/mouse input or window management. python-evdev could be used to emulate input via the uinput Linux kernel module, although this requires elevated privileges. This is what Aenea's Wayland server implementation does.

Edit: I've re-written this post to mostly be a list of things to do rather than using the GitHub project for that.

[BazookaMusic]

I can try making an adapter for keyboard keys to keep the same convention on their spec.also,care should be given to avoid the sudo requirement of some of these midules

[drmfinlay]

Thanks 👍

At least some of these projects already do the adaption for platform specific key tables, e.g. PyUserInput's implementations, so that should make things easier. There is also work in Aenea for translating from dragonfly keys to what xdotool uses and what Mac OS uses.

For requirements like python-xlib, something like "python-xlib;platform_system!='Windows'" could be used in the setup.py install_requires list to make it only a requirement for non-Windows systems. The docs on that setuptools feature are here. I think dragonfly's required version for setuptools might need to be updated though, I'm not sure.

[calmofthestorm]

You may want to consider libxdo instead of shelling out to xdotool.

[drmfinlay]

Yep, that would be better. I was hoping to just rely on a Python library instead. I'm currently leaning towards pynput. PyUserInput is also promising. Both of them work for me on a mac and on Linux with xfce4/xfwm4. Both do the input via python-xlib, which is kind of neat. I'll update the top post.

Any sort of X11 window management should probably be done with libxdo as it already handles X11 extensions properly.

[drmfinlay]

I'd like to note here that it is not possible to fully support Wayland in its current state. This is because emulating keyboard and mouse input, getting window info, etc was left out of the Wayland protocol specification for security reasons. I don't know why these things can't be done securely via policy-kit somehow, similar to MacOS handles this, but there's not much we can do about the limitation except recommend using X11.

Aenea's server implementation for Wayland uses python-evdev, emulating keyboard and mouse input via the uinput Linux kernel module. Only the superuser can access the /dev/uinput device by default. Running dragonfly as root is not ideal though.

[multimeric]

I just had a look at python-evdev, and I'm happy with how it works, and the fact that it works under Wayland, and doesn't depend on X11 like a lot of the other libraries. Would a PR be accepted that adds this capability? I realise the permissions on uinput are annoying, but they can be modified by a superuser to be more permissive where necessary, so it's not unreasonable to want to use it.

Or, should I try getting this upstream into pynput, which it looks like we already use somewhat?

[drmfinlay]

It would be nice to have the keyboard and mouse input parts of Dragonfly work with uinput like this. ydotool is a similar project which looks promising. It's also nice that this would allow using Dragonfly in TTYs.

I suppose requiring root / changing the permissions of uinput would be better than just not supporting Wayland at all, so feel free to make a Dragonfly PR on this! 👍

Dragonfly uses xdotool on X11 rather than pynput. I guess you noticed the X11-specific pynput keyboard code. That's not used any more, I just forgot to remove it. pynput is only used on Mac OS.

Some pointers on implementing this:

• Add evdev as a Linux-only requirement in setup.py.

• Modify the dragonfly/actions/keyboard/__init__.py file to import your Typeable, KeySymbols and Keyboard classes from a new module. Do this after the X11 block:

  elif os.environ.get("XDG_SESSION_TYPE") == "x11" and not doc_build:
      ...
  
  elif sys.startswith("linux") and not doc_build:
      # evdev is Linux-specific (I think?)
      from ._evdev import EvDevKeyboard as Keyboard, Typeable, EvDevKeySymbols as KeySymbols

• I'm pretty sure setting up uinput in EvDevKeyboard.__init__() will cause permission issues with Dragonfly's CI build. Maybe it could be lazily initialised in Keyboard.send_keyboard_events() instead?

[multimeric]

I've had a think about some of these options. The main issue is that we have to deal with keymapping, since we can't assume QWERTY.

• ydotool sounds promising, but I can't get its dependencies to compile, and even people in the AUR repo seem to complain about it. In addition, it seems to have trouble with non-QWERTY layouts: Use active keymap instead of hard-coding English qwerty YukiWorkshop/libevdevPlus#2. So I'm ruling out this option for now.
• Use the low-level kernel keymap. I explain this slightly more here: uinput-based backend for Linux moses-palmer/pynput#184 (comment). This should work, but it requires sudo to access the keymap, which isn't ideal
• Become a Wayland client in order to access the keymap: uinput-based backend for Linux moses-palmer/pynput#184 (comment)

As you can see from those links, I started discussing the issue in pynput, because I still think it would be nicer to implement this upstream, and get it for free here in dragonfly. I see now that you don't use pynput for Linux at the moment, but it is already a dependency so we don't lose anything by using it for Wayland too.

[drmfinlay]

Thanks for looking into this so thoroughly!

Aenea also has an evdev keyboard implementation, but has the same keymap issues. A few keymaps are hardcoded in at the moment, which works for most people, but isn't really ideal.

I agree that it would be nicer to implement it upstream. If done that way, then we should just be able to use Dragonfly's Keyboard class for pynput (this one), assuming the Controller class behaves the same.

Regarding the 3rd option, would becoming a Wayland client also allow us to get window information and manipulate windows in a Wayland session? Or is that just not possible for Wayland? Just thinking of things like AppContext.

If the keyboard implementation ends up relying on Wayland client queries, then the code above should just check XDG_SESSION_TYPE instead:

elif os.environ.get("XDG_SESSION_TYPE") == "wayland" and not doc_build:
    from ._evdev import EvDevKeyboard as Keyboard, Typeable, EvDevKeySymbols as KeySymbols

[multimeric]

Hmm, I think the Wayland protocol would let us get the window size, but I don't think you can manipulate windows. Wayland tries hard to stop applications interfering with each other.

[drmfinlay]

Ah, okay then. I figured as much. I don't keep up with Wayland development, but I hope they plan to implement some sort of authorised way to "interfer" with other applications for accessibility purposes. It would make this sort of thing easier.

The current window dimensions would actually be useful. That would allow relative mouse movements to work whenever we get the Mouse action working on Linux, assuming you can also get the position of windows on the screen.

[multimeric]

I think relative movement is a built-in capability to evdev: https://python-evdev.readthedocs.io/en/latest/apidoc.html#evdev.device.InputDevice.capabilities. You just provide inputs as either relative or absolute movements. So we don't even need the window for that.

[drmfinlay]

Oh sorry, I meant mouse movements relative to the current window's dimensions, not desktop-relative movements: https://dragonfly2.readthedocs.io/en/latest/actions.html#example-mouse-actions
Still, it's good that we get that type of relative movement for free. It is the same with xdotool.

[drmfinlay]

I might do some work on porting the Mouse action today now that I'm thinking about it. It's the only major thing left for X11.

[comodoro]

I have a question about Wayland: there should be a compatibility layer with X, is it necessary to implement separately?

[multimeric]

Yes, it is necessary to implement separately. XWayland is just another Wayland client that itself exposes an X server (see diagram below). Which means, it's still limited in functionality by what Wayland clients can do. And Wayland clients can't do things like moving the mouse, injecting keys, identifying the current window etc. Thus, we still have to resort to uinput as the primary input simulation method.

[comodoro]

Thanks. I see you had made an issue in pynput, moses-palmer/pynput#184, so maybe few changes will be necessary on this end, apart from perhaps needing permissions on install

[multimeric]

That's what I'm hoping, yes. We will have to use pynput if linux and wayland but otherwise it should be pretty easy from the dragonfly end.

[multimeric]

Just updating here to share my conclusion from the other thread: There are some Wayland protocols in the works that would solve this in a much cleaner way, so I think it would be best to hold off on implementation until they're accepted, rather than using the uglier uinput method.

[drmfinlay]

Yep, there would only be a few changes on this end.

Thanks for sharing that. Good to know there is some work on Wayland protocol extensions for this, although I can also see some uses for a uinput-pynput implementation.

This Python package supports keyboard and mouse input through uinput on Linux: https://github.com/boppreh/keyboard
It does explicitly require sudo though.

[multimeric]

Oh, I missed that library entirely. But it seems like our best bet for non-X11 Linux at the moment. And it uses the dumpkeys + uinput strategy I was envisaging. I might try to hook up keyboard (and mouse) to Dragonfly if I get the chance.

[drmfinlay]

Yep, it seems like it. It would be good to have that in the meantime, even if it is replaced later on.

I should have #140 merged soon enough. That will make the job easier for mouse if you get to it.

[drmfinlay]

With #140 merged, Mouse and Monitor should now work on X11 and Mac OS! 🎉

I've edited the TODO items in the top post with the things that are left.

[drmfinlay]

I will be closing this issue once PR #218 is merged and the changes are released in a new version (0.22.0). That PR adds macOS support for the Kaldi engine, which gives us a very usable speech recognition engine backend on each major desktop platform, making Dragonfly truly cross-platform at last! 🎉

Of course, that doesn't mean Dragonfly's cross-platform support won't be improved upon, but it will be sufficient to close this issue.

Regarding Wayland, it looks like Dragonfly-Wayland support will always be limited to, at best, keyboard/mouse input simulation without window context information. Unfortunately, the designers of Wayland seem to view desktop applications sharing information, even simple information like window titles, as a security risk rather than as an accessibility feature. An accessibility feature which, might I add, every other major desktop system has had for decades. The consequence for Dragonfly is that Window, AppContext and friends will never work under Wayland, at least not generally.

Given these things, I will be encouraging Dragonfly-Wayland users to swap to X11 instead, regardless of the obscure security issues. Wayland is simply not ready for people who depend on accessibility software like Dragonfly. This isn't to say I'm not open to Wayland support, I would be happy to review pull requests on it, but I will not be committing development time to implementing it myself.

[drmfinlay]

I'm closing this issue now that version 0.22.0 has been released.

[droundy]

Is there a new issue for Wayland support specifically? X11 has issues with my high DPI monitor setup that make it close to unusable for me, so I'd love to see even limited Wayland support.

[drmfinlay]

Hi @droundy. I'll open a new issue for Wayland support.