AltTab doesn't list windows from other Spaces opened before it was started

[lwouis]

Describe the bug

As discussed in this other ticket, fullscreen windows open before AltTab is opened, will not be listed in AltTab.

Steps to reproduce the bug

• Bring a window to another Space (or fullscreen a window)
• Switch to another space (not the one with the window)
• Start AltTab
• Notice that the window is missing from AltTab

[lwouis]

Update: actually it seems to affect any window on another space, not just fullscreen window. Seems like a regression

[lwouis]

Ok I found the root cause. It's quite a complex story, and I'm not sure which way to go forwards.

Recently, I've improved the multi-threading, and handling of some OS calls. Namely, AX calls (made to get the windows title, type, position, etc) were put on a separate thread, so they can be processed in the background without affecting the UI responsiveness. There are 2 reasons here:

• These AX calls can flat-out fail (if the app in question is unresponsive for example), so we want a retry-loop to try again. This is critical for slow-starting apps like Photoshop, Gimp, etc.
• These AX calls are blocking, and can be slow to finish blocking (if the app in question is unresponsive for example), so we want to move them to a background thread so the UI stays responsive while we wait for the OS to come back with the response.

We added this background and retry-loop logic everywhere, including AltTab startup code, which plays some tricks. At startup, AltTab teleports all windows from other Spaces into the current Space, then attempts to get their AX info like titles, then teleports them back to their original Space. This is ugly, uses private APIs, makes the screen flash for a few milliseconds, but is the only way to be able to list these windows in AltTab at startup. Indeed the OS doesn't list windows from other Spaces otherwise, so it's the only way I know of to achieve that.

Here is the actual code from startup:

// on initial launch, we use private APIs to bring windows from other spaces into the current space, observe them, then remove them from the current space
CGSAddWindowsToSpaces(cgsMainConnectionId, windowsOnlyOnOtherSpaces as NSArray, [Spaces.currentSpaceId])
Applications.observeNewWindows()
CGSRemoveWindowsFromSpaces(cgsMainConnectionId, windowsOnlyOnOtherSpaces as NSArray, [Spaces.currentSpaceId])

The issue is that in observeNewWindows, we now make the AX calls on a background thread, so that method returns instantly, before the AX calls are actually made. Then the windows are teleported back, and when the AX calls are finally made, they return nothing as the windows are no longer in the current Space.

2 ways to go forwards I can imagine:

• We accept the current situation. I think it's very reasonable given that you just need to visit the other spaces, and windows from that space will be listed in AltTab. The issue fixes itself as the user gets confused by it basically. It's also uncommon since users are expected to start AltTab at login. Actual scenario happens because of updates with restart the app.

• We make the AX calls on the main thread as we used to. This means that if an app is slow to respond, the user desk will stay for potentially minutes with all the buggy teleported windows from other Spaces, thinking that a catastrophe happened. I think this is an unacceptable potential situation.

Writing this out, I think it's clear that solution 2 is crazy, so I want to go with solution 1. I actually want to go further, and completely remove the hacky code above from AltTab. It was a useful hack, and is a valiant effort to list windows from other Spaces at startup, but in the end it can't be done in a clean way since any app may be unresponsive, and we don't want to wait seconds with the app magically showing in the current Space, while we try to get it's AX info.

I tried the same scenario with HyperSwitch, and somehow it's able to list the windows. There must be another trick than teleporting them. Here is a dump of the symbols they use as a reference. I'll update this ticket to reflect that goal.

[sk-gara]

I was just about to report this. When I switch on my Mac every day, I'll have to switch to the other 2 Desktops (Spaces) to fully populate AltTabs List.

[mparry]

Re. crazy solution 2, i.e.

We make the AX calls on the main thread as we used to. This means that if an app is slow to respond, the user desk will stay for potentially minutes with all the buggy teleported windows from other Spaces, thinking that a catastrophe happened. I think this is an unacceptable potential situation.

I realise it might amount to another rewrite of this logic, but -

Couldn't you do this with a (manual) timeout per app/window query instead? For example, if one doesn't respond within 100ms then it is omitted, until the user switches to the Space manually. That would still give better results than option 1, and presumably well-behaved apps generally ought to respond pretty quickly.

I know you said they're blocking but perhaps you could also paralellise these queries, so that it's still fast?

(Equally, this isn't a very big issue, for me at least, so it seems entirely reasonable not to want to spend that much time on it.)

[lwouis]

That approach would still flash the screen. It is also pretty tricky to do. I think I would rather keep current status quo, and when looking to improve (i.e. fix this ticket), try to find a novel way. If HyperSwitch is doing it, there is a way.

I need to revisit the private APIs I haven't tried, and find a new trick to get these windows on other spaces at startup. Please feel free to join the investigation:

• APIs I documented previously
• All the private API symbols
• Some symbols already retro-engineered and documented

[koekeishiya]

You can get both the window title and position (and screenshot) from a window using private APIs that only require the target window id. You can also retrieve a list of window_ids for all windows regardless of the space they are at. You will however not be able to retrieve an AXUIElementRef unless the window is on the currently active space.

I believe you already have used some of these before. Fairly certain I mentioned a few of them in various issues/PRs here as well, but they are easy to spot if you look at the exported symbols in the SkyLight (or CoreGraphics) framework.

[lwouis]

@koekeishiya i'm sorry, my memory failed me I think. I'll find these private APIs then! However, let's say i have title, screenshot, and app icon, i still need to be able to focus the window with the wid only. I tried that 2 days ago and while i can actually have the window be active, the code is not able to switch Space. I then looked for private APIs to switch space, but then remembered why i moved the windows previously: all the private APIs for Space manipulation are not actually moving to different Spaces. Instead they do logical assignement of windows to Spaces which result in graphical glitches.

Do you think it's possible to focus a window on another Space only with its wid? I remember spending a few weeks on this last time, playing with the methods from yabai, but in the end i still had to add an AX call to make it work

[koekeishiya]

The scripting addition that yabai injects into the Dock is capapble of focusing a window using only its window id, by calling a non-exported function. IIRC this function is sending a message to some apple.dock.Server port to implement that. I’m not sure if this is simply a mach call that any process could send, to implement the same feature, or if it is a protected xpc service. You can combine CGSShowSpaces, CGSHideSpaces and CGSSetCurrentSpaceForManagedDisplay to do a proper space switch (but mission control is out of sync, unless you update the internal datastructures in the Dock like yabai does.)

…

Sent from my iPhone On 19 Jul 2020, at 06:52, lwouis <[email protected]> wrote:  @koekeishiya<https://github.com/koekeishiya> i'm sorry, my memory failed me I think. I'll find these private APIs then! However, let's say i have title, screenshot, and app icon, i still need to be able to focus the window with the wid only. I tried that 2 days ago and while i can actually have the window be active, the code is not able to switch Space. I then looked for private APIs to switch space, but then remembered why i moved the windows previously: all the private APIs for Space manipulation are not actually moving to different Spaces. Instead they do logical assignement of windows to Spaces which result in graphical glitches. Do you think it's possible to focus a window on another Space only with its wid? I remember spending a few weeks on this last time, playing with the methods from yabai, but in the end i still had to add an AX call<https://github.com/lwouis/alt-tab-macos/blob/c21170d574c4e9bb2de1bc3a1f3873289720ae4f/src/logic/Window.swift#L143> to make it work — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#431 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABPDZV4HM4YRHXFF7QXPUVDR4J3ZNANCNFSM4O3PKSCA>.

[lwouis]

IIRC this function is sending a message to some apple.dock.Server port to implement that. I’m not sure if this is simply a mach call that any process could send, to implement the same feature, or if it is a protected xpc service.

Interesting. Maybe i could reuse that somehow. Could you please point me to the area of the codebase where this is happening? I don't remember seeing this in the past. Maybe the scripting addition is in another dedicated repo?

[koekeishiya]

Wrapper for code that I inject into the Dock: https://github.com/koekeishiya/yabai/blob/master/src/osax/payload.m#L812
Pattern to locate the function inside the Dock binary: https://github.com/koekeishiya/yabai/blob/master/src/osax/payload.m#L361
You can use IDA or Hopper and do search by bytes/hex sequence and paste the pattern to lookup the disassembled function.

[lwouis]

I followed your instructions and was able to get to that part of the Dock binary. It seems it's doing an XPC call. I guess that's game-over for using it without a scripting addition.

[koekeishiya]

I found this to be quite interesting so I looked into where this service is defined, and it is ran by TALagent (Transparent Application Lifecycle agent), and this is what the Dock is signalling when it wants to request focus for a window. The binary is located at /System/Library/CoreServices/talagent. The xpc path you refer to above is however only chosen if the application is registered with the property LSIsProxiedForTAL (check using _LSCopyApplicationInformation). If this attribute is false it will look up a CFMachPortRef using the process serial number, retrieve the associated mach_port_t and send a mach_msg, instead of contacting the talagent xpc service.

[lwouis]

This is a bit too difficult for me to keep investigating as I lack background in low-level languages, and their toolchains and practices.

Furthermore, I think there may be a simpler approach. When investigating the decompiled code of HyperSwitch, I see their fullscreen function:

Note how it uses AXRaise.

Now if I put a window on another Space, and I start HyperSwitch, I can then summon it. At this point I see the window listed. Our theory so far was that they are able to focus it without having access to the AXUIElement object. However, if I press alt-f on the thumbnail, HyperSwitch focuses, then fullscreens the window. That seems to indicate to me that they have the AXUIElement at this point.

Now the question is: how do they get the AXUIElement of windows from other Spaces. When HyperSwitch boots, there is no visual flickering, so they are no using a trick with the private Space APIs. I also checked if they have hidden windows in all Spaces are something like that but the AX call returns 0 window for HyperSwitch.

[lwouis]

@mparry @sk-gara note that I brought back the old behavior of teleporting the windows at startup. I figured it's better than nothing until we find a better approach

[koekeishiya]

Not sure how relevant this really is at this point, but I did some more digging for fun and figured I would share my findings.
The following code creates and sends a mach message to a target process that correctly makes it focus the specified window through its window_id only. However, the tricky part now is to actually be able to pass it along to the target process..

I tested and confirmed that it does work as-is (by running it from inside the Dock process, because it already has access to the mach_port_t instances). Unfortunately as far as I know there is no way to get the task (mach_port_t) to some other process without at least running as root. The Dock, however, is NOT running as root, so I assume there may still be some way to be able to achieve this both out of process and without root...

struct mach_msg_set_front_window {
    mach_msg_header_t header;
    NDR_record_t NDR_record;
    uint32_t window_id;
};

static bool set_front_window(mach_port_t port, uint32_t wid)
{
    struct mach_msg_set_front_window msg = {};

    msg.NDR_record = NDR_record;
    msg.window_id = wid;
    msg.header.msgh_bits = 0x13;
    msg.header.msgh_size = sizeof(struct mach_msg_set_front_window);
    msg.header.msgh_remote_port = port;
    msg.header.msgh_local_port = 0;
    msg.header.msgh_id = 0x17AE8;
    voucher_mach_msg_set(&msg.header);

    OSErr result = mach_msg(&msg.header, 0x11, 0x24, 0x0, 0x0, 0x7d0, 0x0);
    return result == KERN_SUCCESS;
}

[lwouis]

That's very interesting. This reminded me of this section of this retro-engineering repo, and this issue opened last year there.

There may be way indeed. This system doesn't seem very sound from a security standpoint. You have also explained you found a way to read the keyboard through userland security measures, so that sets the tone for macOS security.

That being said, are you not intrigued by how HyperSwitch seemingly is able to access AXUIElement from other spaces? I explored their app with Hopper for hours, and can't find the trick. They have code referring to osax scripting additions, and injecting things, but i think it may not be used. Their OCWindow class seems to be the main interest, but I only found normal code through asking the public AX API for the app, then for its windows, which is not enough for other-Space windows.

[lwouis]

I'll close this ticket as the functionality was brought back in the latest release, but I want to dig deeper and continue the conversation in this follow-up ticket: #447, where the goal is to avoid the current janky trick, and find a cooler trick, as HyperSwitch is doing.

Let's follow-up in #447