Can't get enigo working in Tauri app on MacOS #174

garrrikkotua · 2023-04-13T15:13:25Z

Describe your Question
Can't get enigo working in Tauri app on macOS.

Describe your Goal
I want to have enigo typing (key_sequence function) in Tauri app, but it doesn't work at all. I want it to type in the app itself and other apps too. This functionality works great on Windows.

Environment (please complete the following information):

OS: macOS
Rust: 1.68.1
Library Version: "0.1.2"

Note
I use set_activation_policy(tauri::ActivationPolicy::Accessory) in Tauri, so the app behaves like a spotlight window. Maybe it prevents some functionality

The text was updated successfully, but these errors were encountered:

pentamassiv · 2023-04-13T15:15:04Z

We already have an issue for Tauri: #153
Is your problem different?

garrrikkotua · 2023-04-13T15:25:47Z

Yeah, I think it is related. Basically key_sequence is a sequence of key_clicks, right?

But in my case the app doesn't crush, just nothing happens. I thought this might be due to some permissions on macOS, etc

pentamassiv · 2023-04-13T17:43:32Z

key_sequence is not just a sequence of key_click. key_sequence enters the whole text at the same time. It should probably be renamed to text_input or something like that. Could you please try key_click and see if it panics?

garrrikkotua · 2023-04-13T20:42:41Z

Yeah I will try later when I have mac :) Perhaps tommorrow

pentamassiv · 2023-09-01T21:35:57Z

I am not familiar with Tauri, but I had to grant permissions in a popup. Did you grant the application the Accessibility permissions? https://support.apple.com/guide/mac-help/allow-accessibility-apps-to-access-your-mac-mh43185/mac

pentamassiv · 2023-10-11T22:48:50Z

Is this still an issue or can I close it?

skyslide22 · 2023-12-09T03:55:44Z

i have the same issue, enigo does not fire anything
MacOS Sonoma 14.1.2 with M3
Enigo 0.1.3 & 2.0rc2 tested

#[tauri::command]
async fn switch_space(left: bool) {
    let mut enigo= Enigo::new(&Settings::default()).unwrap();
    enigo.key(Key::Control, Direction::Press);
    if left {
        enigo.key(Key::LeftArrow, Direction::Click);
    } else {
        enigo.key(Key::RightArrow, Direction::Click);
    }
    enigo.key(Key::Control, Direction::Release);
    println!("switch space left = {left}") // this is printed but key did not fire at all
}

A-Shleifman · 2024-02-05T02:02:46Z

I'm writing my first rust script now, so I have no idea what I'm doing, but maybe the problem is a level lower. My script is not using Tauri, but it's using tao, which is what Tauri runs on top of from what I've read.

use global_hotkey::{ hotkey::{HotKey, Code}, GlobalHotKeyEvent, GlobalHotKeyManager, HotKeyState };
use tao::event_loop::{ ControlFlow, EventLoopBuilder };
use enigo::{ Enigo, Keyboard, Settings };

fn main() {
    let event_loop = EventLoopBuilder::new().build();
    let mut enigo = Enigo::new(&Settings::default()).unwrap();
    let hotkey_manager = GlobalHotKeyManager::new().unwrap();

    enigo.text("Launched!").unwrap(); // 👈 this is working as expected

    let hotkey = HotKey::new(None, Code::F12);

    hotkey_manager.register(hotkey).unwrap();

    let global_hotkey_channel = GlobalHotKeyEvent::receiver();

    event_loop.run(move |_event, _, control_flow| {
        *control_flow = ControlFlow::Wait;

        if let Ok(event) = global_hotkey_channel.try_recv() {
            if hotkey.id() == event.id && event.state == HotKeyState::Pressed {
                println!("F12 Pressed!");

                enigo.text("input").unwrap(); // 👈 this doesn't do anything
            }
        }
    })
}

In this script enigo.text works outside tao's event_loop, but not inside. This could make hunting for the problem a bit easier.

thewh1teagle · 2024-05-06T23:05:10Z

I also facing the same issue with tauri app.
enigo works great outside of tauri, also it works with tauri in dev mode. but once I install the app and launch through applications then enigo keys doesn't work.
Production log:

tauri installed app logs

[2024-05-06T23:06:12Z TRACE tao::platform_impl::platform::app_delegate] Triggered `applicationDidFinishLaunching`
[2024-05-06T23:06:12Z TRACE tao::platform_impl::platform::window] Creating new window
[2024-05-06T23:06:12Z TRACE tao::platform_impl::platform::view] Triggered `viewDidMoveToWindow`
[2024-05-06T23:06:12Z TRACE tao::platform_impl::platform::view] Completed `viewDidMoveToWindow`
[2024-05-06T23:06:12Z TRACE tao::platform_impl::platform::view] Triggered `validAttributesForMarkedText`
[2024-05-06T23:06:12Z TRACE tao::platform_impl::platform::view] Completed `validAttributesForMarkedText`
[2024-05-06T23:06:12Z TRACE tao::platform_impl::platform::view] Triggered `validAttributesForMarkedText`
[2024-05-06T23:06:12Z TRACE tao::platform_impl::platform::view] Completed `validAttributesForMarkedText`
[2024-05-06T23:06:12Z TRACE tao::platform_impl::platform::view] Triggered `validAttributesForMarkedText`
[2024-05-06T23:06:12Z TRACE tao::platform_impl::platform::view] Completed `validAttributesForMarkedText`
[2024-05-06T23:06:12Z TRACE tao::platform_impl::platform::view] Triggered `validAttributesForMarkedText`
[2024-05-06T23:06:12Z TRACE tao::platform_impl::platform::view] Completed `validAttributesForMarkedText`
[2024-05-06T23:06:12Z TRACE tao::platform_impl::platform::view] Triggered `validAttributesForMarkedText`
[2024-05-06T23:06:12Z TRACE tao::platform_impl::platform::view] Completed `validAttributesForMarkedText`
[2024-05-06T23:06:12Z TRACE tao::platform_impl::platform::window] Locked shared state in `set_fullscreen`
[2024-05-06T23:06:12Z TRACE tao::platform_impl::platform::window] Unlocked shared state in `set_fullscreen`
[2024-05-06T23:06:12Z TRACE tao::platform_impl::platform::window_delegate] Triggered `windowDidBecomeKey:`
[2024-05-06T23:06:12Z TRACE tao::platform_impl::platform::window_delegate] Completed `windowDidBecomeKey:`
[2024-05-06T23:06:12Z TRACE tao::platform_impl::platform::view] Triggered `validAttributesForMarkedText`
[2024-05-06T23:06:12Z TRACE tao::platform_impl::platform::view] Completed `validAttributesForMarkedText`
[2024-05-06T23:06:12Z TRACE tao::platform_impl::platform::view] Triggered `validAttributesForMarkedText`
[2024-05-06T23:06:12Z TRACE tao::platform_impl::platform::view] Completed `validAttributesForMarkedText`
[2024-05-06T23:06:12Z TRACE tao::platform_impl::platform::view] Triggered `validAttributesForMarkedText`
[2024-05-06T23:06:12Z TRACE tao::platform_impl::platform::view] Completed `validAttributesForMarkedText`
[2024-05-06T23:06:12Z TRACE tao::platform_impl::platform::view] Triggered `validAttributesForMarkedText`
[2024-05-06T23:06:12Z TRACE tao::platform_impl::platform::view] Completed `validAttributesForMarkedText`
[2024-05-06T23:06:12Z TRACE tao::platform_impl::platform::view] Triggered `validAttributesForMarkedText`
[2024-05-06T23:06:12Z TRACE tao::platform_impl::platform::view] Completed `validAttributesForMarkedText`
[2024-05-06T23:06:12Z TRACE tao::platform_impl::platform::view] Triggered `viewDidMoveToWindow`
[2024-05-06T23:06:12Z TRACE tao::platform_impl::platform::view] Completed `viewDidMoveToWindow`
[2024-05-06T23:06:12Z TRACE tao::platform_impl::platform::app_delegate] Completed `applicationDidFinishLaunching`
[2024-05-06T23:06:13Z TRACE tao::platform_impl::platform::window_delegate] Triggered `windowDidResignKey:`
[2024-05-06T23:06:13Z TRACE tao::platform_impl::platform::window_delegate] Completed `windowDidResignKey:`
[2024-05-06T23:06:13Z TRACE tao::platform_impl::platform::app_delegate] Triggered `applicationSupportsSecureRestorableState`
[2024-05-06T23:06:13Z TRACE tao::platform_impl::platform::app_delegate] Completed `applicationSupportsSecureRestorableState`
[2024-05-06T23:06:16Z TRACE tao::platform_impl::platform::window_delegate] Triggered `windowDidBecomeKey:`
[2024-05-06T23:06:16Z TRACE tao::platform_impl::platform::window_delegate] Completed `windowDidBecomeKey:`
[2024-05-06T23:06:18Z TRACE tao::platform_impl::platform::window_delegate] Triggered `windowDidResignKey:`
[2024-05-06T23:06:18Z TRACE tao::platform_impl::platform::window_delegate] Completed `windowDidResignKey:`
[2024-05-06T23:06:23Z DEBUG enigo] using default settings
[2024-05-06T23:06:23Z DEBUG enigo::platform::macos_impl] �[93mconnection established on macOS�[0m
[2024-05-06T23:06:23Z DEBUG mobslide::cmd] Pressing VOL_UP
[2024-05-06T23:06:23Z DEBUG enigo::platform::macos_impl] �[93mkey(key: VolumeUp, direction: Click)�[0m
[2024-05-06T23:06:23Z DEBUG enigo::platform::macos_impl] special case for handling the VolumeUp key
[2024-05-06T23:06:23Z DEBUG enigo::platform::macos_impl] released all held keys
[2024-05-06T23:06:23Z DEBUG enigo] using default settings
[2024-05-06T23:06:23Z DEBUG enigo::platform::macos_impl] �[93mconnection established on macOS�[0m
[2024-05-06T23:06:23Z DEBUG mobslide::cmd] Pressing VOL_UP
[2024-05-06T23:06:23Z DEBUG enigo::platform::macos_impl] �[93mkey(key: VolumeUp, direction: Click)�[0m
[2024-05-06T23:06:23Z DEBUG enigo::platform::macos_impl] special case for handling the VolumeUp key
[2024-05-06T23:06:23Z DEBUG enigo::platform::macos_impl] released all held keys
[2024-05-06T23:06:25Z DEBUG enigo] using default settings
[2024-05-06T23:06:25Z DEBUG enigo::platform::macos_impl] �[93mconnection established on macOS�[0m
[2024-05-06T23:06:25Z DEBUG mobslide::cmd] Pressing VOL_DN
[2024-05-06T23:06:25Z DEBUG enigo::platform::macos_impl] �[93mkey(key: VolumeDown, direction: Click)�[0m
[2024-05-06T23:06:25Z DEBUG enigo::platform::macos_impl] special case for handling the VolumeDown key
[2024-05-06T23:06:25Z DEBUG enigo::platform::macos_impl] released all held keys
[2024-05-06T23:06:25Z DEBUG enigo] using default settings
[2024-05-06T23:06:25Z DEBUG enigo::platform::macos_impl] �[93mconnection established on macOS�[0m
[2024-05-06T23:06:25Z DEBUG mobslide::cmd] Pressing VOL_DN
[2024-05-06T23:06:25Z DEBUG enigo::platform::macos_impl] �[93mkey(key: VolumeDown, direction: Click)�[0m
[2024-05-06T23:06:25Z DEBUG enigo::platform::macos_impl] special case for handling the VolumeDown key
[2024-05-06T23:06:25Z DEBUG enigo::platform::macos_impl] released all held keys
[2024-05-06T23:06:31Z DEBUG enigo] using default settings
[2024-05-06T23:06:31Z DEBUG enigo::platform::macos_impl] �[93mconnection established on macOS�[0m
[2024-05-06T23:06:31Z DEBUG mobslide::cmd] Pressing VOL_UP
[2024-05-06T23:06:31Z DEBUG enigo::platform::macos_impl] �[93mkey(key: VolumeUp, direction: Click)�[0m
[2024-05-06T23:06:31Z DEBUG enigo::platform::macos_impl] special case for handling the VolumeUp key
[2024-05-06T23:06:31Z DEBUG enigo::platform::macos_impl] released all held keys
[2024-05-06T23:06:31Z DEBUG enigo] using default settings
[2024-05-06T23:06:31Z DEBUG enigo::platform::macos_impl] �[93mconnection established on macOS�[0m
[2024-05-06T23:06:31Z DEBUG mobslide::cmd] Pressing VOL_UP
[2024-05-06T23:06:31Z DEBUG enigo::platform::macos_impl] �[93mkey(key: VolumeUp, direction: Click)�[0m
[2024-05-06T23:06:31Z DEBUG enigo::platform::macos_impl] special case for handling the VolumeUp key
[2024-05-06T23:06:31Z DEBUG enigo::platform::macos_impl] released all held keys
[2024-05-06T23:06:34Z DEBUG enigo] using default settings
[2024-05-06T23:06:34Z DEBUG enigo::platform::macos_impl] �[93mconnection established on macOS�[0m
[2024-05-06T23:06:34Z DEBUG mobslide::cmd] Pressing VOL_DN
[2024-05-06T23:06:34Z DEBUG enigo::platform::macos_impl] �[93mkey(key: VolumeDown, direction: Click)�[0m
[2024-05-06T23:06:34Z DEBUG enigo::platform::macos_impl] special case for handling the VolumeDown key
[2024-05-06T23:06:34Z DEBUG enigo::platform::macos_impl] released all held keys
[2024-05-06T23:06:34Z DEBUG enigo] using default settings
[2024-05-06T23:06:34Z DEBUG enigo::platform::macos_impl] �[93mconnection established on macOS�[0m
[2024-05-06T23:06:34Z DEBUG mobslide::cmd] Pressing VOL_DN
[2024-05-06T23:06:34Z DEBUG enigo::platform::macos_impl] �[93mkey(key: VolumeDown, direction: Click)�[0m
[2024-05-06T23:06:34Z DEBUG enigo::platform::macos_impl] special case for handling the VolumeDown key
[2024-05-06T23:06:34Z DEBUG enigo::platform::macos_impl] released all held keys

tauri dev mode logs (works)

    Finished dev [unoptimized + debuginfo] target(s) in 3.65s
[2024-05-06T23:09:38Z TRACE tao::platform_impl::platform::app_delegate] Triggered `applicationDidFinishLaunching`
[2024-05-06T23:09:38Z TRACE tao::platform_impl::platform::window] Creating new window
[2024-05-06T23:09:38Z TRACE tao::platform_impl::platform::view] Triggered `viewDidMoveToWindow`
[2024-05-06T23:09:38Z TRACE tao::platform_impl::platform::view] Completed `viewDidMoveToWindow`
[2024-05-06T23:09:38Z TRACE tao::platform_impl::platform::view] Triggered `validAttributesForMarkedText`
[2024-05-06T23:09:38Z TRACE tao::platform_impl::platform::view] Completed `validAttributesForMarkedText`
[2024-05-06T23:09:38Z TRACE tao::platform_impl::platform::view] Triggered `validAttributesForMarkedText`
[2024-05-06T23:09:38Z TRACE tao::platform_impl::platform::view] Completed `validAttributesForMarkedText`
[2024-05-06T23:09:38Z TRACE tao::platform_impl::platform::view] Triggered `validAttributesForMarkedText`
[2024-05-06T23:09:38Z TRACE tao::platform_impl::platform::view] Completed `validAttributesForMarkedText`
[2024-05-06T23:09:38Z TRACE tao::platform_impl::platform::view] Triggered `validAttributesForMarkedText`
[2024-05-06T23:09:38Z TRACE tao::platform_impl::platform::view] Completed `validAttributesForMarkedText`
[2024-05-06T23:09:38Z TRACE tao::platform_impl::platform::view] Triggered `validAttributesForMarkedText`
[2024-05-06T23:09:38Z TRACE tao::platform_impl::platform::view] Completed `validAttributesForMarkedText`
[2024-05-06T23:09:38Z TRACE tao::platform_impl::platform::window] Locked shared state in `set_fullscreen`
[2024-05-06T23:09:38Z TRACE tao::platform_impl::platform::window] Unlocked shared state in `set_fullscreen`
[2024-05-06T23:09:38Z TRACE tao::platform_impl::platform::view] Triggered `validAttributesForMarkedText`
[2024-05-06T23:09:38Z TRACE tao::platform_impl::platform::view] Completed `validAttributesForMarkedText`
[2024-05-06T23:09:38Z TRACE tao::platform_impl::platform::view] Triggered `validAttributesForMarkedText`
[2024-05-06T23:09:38Z TRACE tao::platform_impl::platform::view] Completed `validAttributesForMarkedText`
[2024-05-06T23:09:38Z TRACE tao::platform_impl::platform::view] Triggered `validAttributesForMarkedText`
[2024-05-06T23:09:38Z TRACE tao::platform_impl::platform::view] Completed `validAttributesForMarkedText`
[2024-05-06T23:09:38Z TRACE tao::platform_impl::platform::view] Triggered `validAttributesForMarkedText`
[2024-05-06T23:09:38Z TRACE tao::platform_impl::platform::view] Completed `validAttributesForMarkedText`
[2024-05-06T23:09:38Z TRACE tao::platform_impl::platform::view] Triggered `validAttributesForMarkedText`
[2024-05-06T23:09:38Z TRACE tao::platform_impl::platform::view] Completed `validAttributesForMarkedText`
[2024-05-06T23:09:38Z TRACE tao::platform_impl::platform::view] Triggered `viewDidMoveToWindow`
[2024-05-06T23:09:38Z TRACE tao::platform_impl::platform::view] Completed `viewDidMoveToWindow`
[2024-05-06T23:09:38Z TRACE tao::platform_impl::platform::app_delegate] Completed `applicationDidFinishLaunching`
[2024-05-06T23:09:38Z TRACE tao::platform_impl::platform::window_delegate] Triggered `windowDidBecomeKey:`
[2024-05-06T23:09:38Z TRACE tao::platform_impl::platform::window_delegate] Completed `windowDidBecomeKey:`

🌼   daisyUI 4.10.5
├─ ✔︎ 2 themes added             https://daisyui.com/docs/themes
╰─ ★ Star daisyUI on GitHub     https://github.com/saadeghi/daisyui

[2024-05-06T23:09:42Z TRACE tao::platform_impl::platform::window_delegate] Triggered `windowDidResignKey:`
[2024-05-06T23:09:42Z TRACE tao::platform_impl::platform::window_delegate] Completed `windowDidResignKey:`
[2024-05-06T23:09:44Z DEBUG enigo] using default settings
[2024-05-06T23:09:44Z DEBUG enigo::platform::macos_impl] connection established on macOS
[2024-05-06T23:09:44Z DEBUG mobslide::cmd] Pressing VOL_UP
[2024-05-06T23:09:44Z DEBUG enigo::platform::macos_impl] key(key: VolumeUp, direction: Click)
[2024-05-06T23:09:44Z DEBUG enigo::platform::macos_impl] special case for handling the VolumeUp key
[2024-05-06T23:09:44Z DEBUG enigo::platform::macos_impl] released all held keys
[2024-05-06T23:09:46Z DEBUG enigo] using default settings
[2024-05-06T23:09:46Z DEBUG enigo::platform::macos_impl] connection established on macOS
[2024-05-06T23:09:46Z DEBUG mobslide::cmd] Pressing VOL_UP
[2024-05-06T23:09:46Z DEBUG enigo::platform::macos_impl] key(key: VolumeUp, direction: Click)
[2024-05-06T23:09:46Z DEBUG enigo::platform::macos_impl] special case for handling the VolumeUp key
[2024-05-06T23:09:46Z DEBUG enigo::platform::macos_impl] released all held keys
[2024-05-06T23:09:46Z DEBUG enigo] using default settings
[2024-05-06T23:09:46Z DEBUG enigo::platform::macos_impl] connection established on macOS
[2024-05-06T23:09:46Z DEBUG mobslide::cmd] Pressing VOL_UP
[2024-05-06T23:09:46Z DEBUG enigo::platform::macos_impl] key(key: VolumeUp, direction: Click)
[2024-05-06T23:09:46Z DEBUG enigo::platform::macos_impl] special case for handling the VolumeUp key
[2024-05-06T23:09:46Z DEBUG enigo::platform::macos_impl] released all held keys
[2024-05-06T23:09:47Z DEBUG enigo] using default settings
[2024-05-06T23:09:47Z DEBUG enigo::platform::macos_impl] connection established on macOS
[2024-05-06T23:09:47Z DEBUG mobslide::cmd] Pressing VOL_DN
[2024-05-06T23:09:47Z DEBUG enigo::platform::macos_impl] key(key: VolumeDown, direction: Click)
[2024-05-06T23:09:47Z DEBUG enigo::platform::macos_impl] special case for handling the VolumeDown key
[2024-05-06T23:09:47Z DEBUG enigo::platform::macos_impl] released all held keys
[2024-05-06T23:09:48Z DEBUG enigo] using default settings
[2024-05-06T23:09:48Z DEBUG enigo::platform::macos_impl] connection established on macOS
[2024-05-06T23:09:48Z DEBUG mobslide::cmd] Pressing VOL_DN
[2024-05-06T23:09:48Z DEBUG enigo::platform::macos_impl] key(key: VolumeDown, direction: Click)
[2024-05-06T23:09:48Z DEBUG enigo::platform::macos_impl] special case for handling the VolumeDown key
[2024-05-06T23:09:48Z DEBUG enigo::platform::macos_impl] released all held keys

cargo.toml

enigo = { version = "0.2.0" }

Update

Found the cause, it's permission error

For manage that permission you need to open settings -> accessibility

screenshot

Then click the + button to add your app from application and allow it to control the computer
If you have already the app it's possibly the dev mode app, remove it and add again the installed one.
Once you remove it and reopen the app it should open the following prompt to ask for the permission:

screenshot

When the permission allowed enigo should work.
By the way, it will be nice if enigo can detect that the app doesn't have that required permission and show error

https://stackoverflow.com/questions/6933510

Checking it is simple as

permission.rs

use std::{error::Error, ptr};
use accessibility_sys::{kAXTrustedCheckOptionPrompt, AXIsProcessTrustedWithOptions};
use core_foundation_sys::dictionary::{CFDictionaryAddValue, CFDictionaryCreateMutable};
use core_foundation_sys::base::{CFRelease, TCFTypeRef};
use core_foundation_sys::number::{kCFBooleanFalse, kCFBooleanTrue};


fn check_accessibility(ask_if_not_allowed: bool) -> Result<bool, Box<dyn Error>> {
    let is_allowed;
    unsafe {
        let options =
            CFDictionaryCreateMutable(ptr::null_mut(), 0, std::ptr::null(), std::ptr::null());
        let key = kAXTrustedCheckOptionPrompt;
        let value = if ask_if_not_allowed {kCFBooleanTrue} else {kCFBooleanFalse};
        if !options.is_null() {
            CFDictionaryAddValue(
                options,
                key.as_void_ptr(),
                value.as_void_ptr(),
            );
            is_allowed = AXIsProcessTrustedWithOptions(options);
            CFRelease(options as *const _);
        } else {
            return Err("options is null".into());
        }
    }
    Ok(is_allowed)
}

fn main() {
    let is_allowed = check_accessibility(true).unwrap();
    println!("Accessibility permission enabled: {}", is_allowed);
}

braden-w · 2024-07-21T02:04:31Z

Adding to @thewh1teagle's solution, I just wanted to leave a quick guide for anyone facing similar issues in the future.

First, I created an accessibility.rs file. Note that I created two functions to use in our application:

is_macos_accessibility_enabled: Based off @thewh1teagle's solution, which returns whether accessibility is enabled or not. It takes in a parameter, ask_if_not_allowed, which toggles prompting the user with a native system prompt to enable accessibility if not already enabled.
open_apple_accessibility: This function opens the accessibility page directly, inspired from this gist with macOS. It is the equivalent of running open "x-apple.systempreferences:com.apple.preference.security?Privacy_Accessibility" in shell.

accessibility.rs:

use accessibility_sys::{kAXTrustedCheckOptionPrompt, AXIsProcessTrustedWithOptions};
use core_foundation_sys::base::{CFRelease, TCFTypeRef};
use core_foundation_sys::dictionary::{
    CFDictionaryAddValue, CFDictionaryCreateMutable, __CFDictionary,
};
use core_foundation_sys::number::{kCFBooleanTrue,kCFBooleanFalse};
use std::process::Command;
use std::ptr;

#[tauri::command]
pub fn is_macos_accessibility_enabled(ask_if_not_allowed: bool) -> Result<bool, &'static str> {
    let options = create_options_dictionary(ask_if_not_allowed)?;
    let is_allowed = unsafe { AXIsProcessTrustedWithOptions(options) };
    release_options_dictionary(options);
    Ok(is_allowed)
}

fn create_options_dictionary(ask_if_not_allowed: bool) -> Result<*mut __CFDictionary, &'static str> {
    unsafe {
        let options = CFDictionaryCreateMutable(ptr::null_mut(), 0, ptr::null(), ptr::null());
        if options.is_null() {
            return Err("Failed to create options dictionary");
        }
        let key = kAXTrustedCheckOptionPrompt;
        let value = if ask_if_not_allowed {kCFBooleanTrue} else {kCFBooleanFalse};
        CFDictionaryAddValue(
            options,
            key.as_void_ptr(),
            value.as_void_ptr(),
        );
        Ok(options)
    }
}

fn release_options_dictionary(options: *mut __CFDictionary) {
    unsafe {
        CFRelease(options as *const _);
    }
}

#[tauri::command]
pub async fn open_apple_accessibility() -> Result<(), String> {
    Command::new("open")
        .arg("x-apple.systempreferences:com.apple.preference.security?Privacy_Accessibility")
        .status()
        .map_err(|e| format!("Failed to execute command: {}", e))
        .and_then(|status| {
            if status.success() {
                Ok(())
            } else {
                Err(format!("Command failed with status: {}", status))
            }
        })
}

Here's how I used it in main.rs:

// Prevents additional console window on Windows in release, DO NOT REMOVE!!
#![cfg_attr(not(debug_assertions), windows_subsystem = "windows")]

#[cfg(target_os = "macos")]
mod accessibility;

#[cfg(target_os = "macos")]
use accessibility::{is_macos_accessibility_enabled, open_apple_accessibility};

use tauri::{CustomMenuItem, Manager};
use tauri::{SystemTray, SystemTrayEvent, SystemTrayMenu};


fn main() {
    let quit = CustomMenuItem::new("quit".to_string(), "Quit");

    let tray_menu = SystemTrayMenu::new().add_item(quit);

    let builder = tauri::Builder::default()
        .system_tray(SystemTray::new().with_menu(tray_menu))
        .on_system_tray_event(|app, event| match event {
            SystemTrayEvent::LeftClick {
                position: _,
                size: _,
                ..
            } => {
                app.emit_all("toggle-recording", ()).unwrap();
            }
            SystemTrayEvent::MenuItemClick { id, .. } => match id.as_str() {
                "quit" => {
                    std::process::exit(0);
                }
                _ => {}
            },
            _ => {}
        });

    #[cfg(target_os = "macos")]
    let builder = builder.invoke_handler(tauri::generate_handler![
        write_text,
        set_tray_icon,
        open_apple_accessibility,
        is_macos_accessibility_enabled,
    ]);

    #[cfg(not(target_os = "macos"))]
    let builder = builder.invoke_handler(tauri::generate_handler![write_text, set_tray_icon,]);

    builder
        .run(tauri::generate_context!())
        .expect("error while running tauri application");
}

use enigo::{Enigo, Keyboard, Settings};

#[tauri::command]
fn write_text(text: String) -> Result<(), String> {
    let mut enigo = Enigo::new(&Settings::default()).unwrap();
    enigo.text(&text).map_err(|e| e.to_string())
}

#[tauri::command]
async fn set_tray_icon(recorder_state: String, app_handle: tauri::AppHandle) -> Result<(), String> {
    let icon = match recorder_state.as_str() {
        "IDLE" => include_bytes!("../icons/recorder_state/studio_microphone.png").to_vec(),
        "RECORDING" => include_bytes!("../icons/recorder_state/red_large_square.png").to_vec(),
        "LOADING" => include_bytes!("../icons/recorder_state/arrows_counterclockwise.png").to_vec(),
        _ => return Err("Invalid state. Must be IDLE, RECORDING, or LOADING.".to_string()),
    };
    app_handle
        .tray_handle()
        .set_icon(tauri::Icon::Raw(icon))
        .unwrap();
    Ok(())
}

To avoid including accessibility features in non-macOS platforms, I used:

#[cfg(target_os = "macos")] in the import statements
#[cfg(target_os = "macos")] and #[cfg(not(target_os = "macos"))] in main() .

Usage

Here's how I used it in my Tauri application:

Check if accessibility is enabled

const isAccessibilityEnabled = await invoke<boolean>(
  "is_macos_accessibility_enabled",
	// Set to true to ask the user to enable accessibility if it's not enabled
  { askIfNotAllowed: false }
)

// Handle case if accessibility is not enabled

I recommend setting askIfNotAllowed to false. Instead, show a custom message to the user to enable it manually. You can include a video and link to the system preferences so the user can allow accessibility.

if (!isAccessibilityEnabled) {
	// note the `yield*` is there because I'm using the Effect-TS package
	yield* toast({
		variant: 'warning',
		title: 'Please enable or re-enable accessibility to paste transcriptions!',
		description: 'Accessibility must be enabled or re-enabled for Whispering after install or update. Follow the link below for instructions.',
		action: {
			label: 'Open Directions',
			onClick: () => goto('/macos-enable-accessibility'),
		},
	});
	return;
}

In this example, I display a warning toast when accessibility is not enabled, where the action button leads the user to a page with directions for how to (re-)enable accessibility.

Just like @thewh1teagle mentioned, if you already have the app enabled, it's possible it's the dev mode app—you need to remove it and add the installed one again.

On the page with directions, I include this button to enable opening MacOS Accessibility.

<button
	onclick={() => invoke('open_apple_accessibility')}
>
	Open MacOS Accessibility Settings
</button>

and a screen recording for enabling and re-enabling accessibility. You can see a preview here.

thewh1teagle · 2024-07-21T02:08:57Z

2. running open "x-apple.systempreferences:com.apple.preference.security?Privacy_Accessibility" in shell.

Thanks for sharing!
It would be great to have an option to open accessibility page without shell commands, as it's not recommend to run shell commands from desktop app in general, for instance AV usually detect shell commands as malware behavior

braden-w · 2024-07-21T02:12:29Z

@thewh1teagle agreed, I tried to find other options to avoid it, but decided to bite the bullet on this one just for added convenience for users!

Thank you again for your solution, it saved me many hours of headache 🙏

thewh1teagle · 2024-07-21T02:59:33Z

@thewh1teagle agreed, I tried to find other options to avoid it, but decided to bite the bullet on this one just for added convenience for users!

Thank you again for your solution, it saved me many hours of headache 🙏

You can use the following to open accessibility with OS's API's directly:

main.rust

/*
[dependencies]
accessibility-sys = { version = "0.1.3" }
cocoa = "0.25.0"
core-foundation-sys = { version = "0.8.6" }
objc = "0.2.7"
*/

use std::error::Error;

pub fn open_accessibility() {
    use cocoa::base::id;
    use objc::{self, class, msg_send, sel, sel_impl};
    
    let url = "x-apple.systempreferences:com.apple.preference.security?Privacy_Accessibility";
    const UTF8_ENCODING: usize = 4;
    unsafe {
        let ns_string: id = msg_send![class!(NSString), alloc];
        let ns_string: id = msg_send![ns_string,
            initWithBytes: url.as_ptr()
            length: url.len()
            encoding: UTF8_ENCODING];
        let _: () = msg_send![ns_string, autorelease];

        let ns_url: id = msg_send![class!(NSURL), alloc];
        let ns_url: id = msg_send![ns_url, initWithString: ns_string];
        let _: () = msg_send![ns_url, autorelease];

        let shared_workspace: id = msg_send![class!(NSWorkspace), sharedWorkspace];
        let _: bool = msg_send![shared_workspace, openURL: ns_url];
    }
}

pub fn check_accessibility(ask_if_not_allowed: bool) -> Result<bool, Box<dyn Error>> {
    use accessibility_sys::{kAXTrustedCheckOptionPrompt, AXIsProcessTrustedWithOptions};
    use core_foundation_sys::base::{CFRelease, TCFTypeRef};
    use core_foundation_sys::dictionary::{CFDictionaryAddValue, CFDictionaryCreateMutable};
    use core_foundation_sys::number::{kCFBooleanFalse, kCFBooleanTrue};
    use std::ptr;

    let is_allowed;
    unsafe {
        let options =
            CFDictionaryCreateMutable(ptr::null_mut(), 0, ptr::null(), ptr::null());
        let key = kAXTrustedCheckOptionPrompt;
        let value = if ask_if_not_allowed {
            kCFBooleanTrue
        } else {
            kCFBooleanFalse
        };
        if !options.is_null() {
            CFDictionaryAddValue(options, key.as_void_ptr(), value.as_void_ptr());
            is_allowed = AXIsProcessTrustedWithOptions(options);
            CFRelease(options as *const _);
        } else {
            return Err("options is null".into());
        }
    }
    Ok(is_allowed)
}


fn main() {
    let allowed = check_accessibility(false).unwrap();
    if !allowed {
        open_accessibility();
        println!("Open Accessibility");
    } else {
        println!("Allowed")
    }
    
}

pentamassiv · 2024-08-01T16:47:19Z

Thank you both for investigating the issue. I should definitely add a check if the permissions were granted. Hopefully I can find some time within the next two weeks

pentamassiv · 2024-09-09T22:07:08Z

Thank you for the code snippets, enigo will now check if the application has the required permissions on macOS. If not, it will ask the user to grant the permissions. You can change this default behavior by creating a Settings struct with open_prompt_to_get_permissions set to false. The prompt for the user will be displayed asynchronously and an error will be returned. You will have to handle a retry yourself.

garrrikkotua added the question label Apr 13, 2023

pentamassiv mentioned this issue Apr 24, 2023

key_sequence() method doesn't take line return into account. #175

Closed

pentamassiv added the macOS macOS specific label Aug 31, 2023

pentamassiv mentioned this issue Sep 9, 2024

macos: Check if the application has the permission to simulate input #324

Merged

pentamassiv closed this as completed in #324 Sep 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can't get enigo working in Tauri app on MacOS #174

Can't get enigo working in Tauri app on MacOS #174

garrrikkotua commented Apr 13, 2023

pentamassiv commented Apr 13, 2023

garrrikkotua commented Apr 13, 2023

pentamassiv commented Apr 13, 2023

garrrikkotua commented Apr 13, 2023

pentamassiv commented Sep 1, 2023

pentamassiv commented Oct 11, 2023

skyslide22 commented Dec 9, 2023

A-Shleifman commented Feb 5, 2024

thewh1teagle commented May 6, 2024 •

edited

Loading

braden-w commented Jul 21, 2024 •

edited

Loading

thewh1teagle commented Jul 21, 2024

braden-w commented Jul 21, 2024 •

edited

Loading

thewh1teagle commented Jul 21, 2024

pentamassiv commented Aug 1, 2024

pentamassiv commented Sep 9, 2024 •

edited

Loading

Can't get enigo working in Tauri app on MacOS #174

Can't get enigo working in Tauri app on MacOS #174

Comments

garrrikkotua commented Apr 13, 2023

pentamassiv commented Apr 13, 2023

garrrikkotua commented Apr 13, 2023

pentamassiv commented Apr 13, 2023

garrrikkotua commented Apr 13, 2023

pentamassiv commented Sep 1, 2023

pentamassiv commented Oct 11, 2023

skyslide22 commented Dec 9, 2023

A-Shleifman commented Feb 5, 2024

thewh1teagle commented May 6, 2024 • edited Loading

Update

braden-w commented Jul 21, 2024 • edited Loading

Usage

Check if accessibility is enabled

thewh1teagle commented Jul 21, 2024

braden-w commented Jul 21, 2024 • edited Loading

thewh1teagle commented Jul 21, 2024

pentamassiv commented Aug 1, 2024

pentamassiv commented Sep 9, 2024 • edited Loading

thewh1teagle commented May 6, 2024 •

edited

Loading

braden-w commented Jul 21, 2024 •

edited

Loading

braden-w commented Jul 21, 2024 •

edited

Loading

pentamassiv commented Sep 9, 2024 •

edited

Loading