Decoupled architecture: separate screenshot collection and processing

[tleyden]

This app seems to have two high level pieces:

1. Collect screenshots in the background
2. Do something with those screenshots

The screenshot collection part could be factored out into its own library so it could be used by any app that needed to collect screenshots in the background. A quick search shows that an XPC service might be the right IPC mechanism for doing that on OSX.

Functionality like:

• Skip identical screens
• Only grab the content of the frontmost application

could be implemented by the screenshot grabbing XPC service.

If other apps adopted the screenshot grabbing library, then the burden of fixing bugs like Started recording, put computer to sleep, reopened - no longer recording could be shared among those apps.

On the downside, splitting out a separate library would have costs, like maintaining two separate building / packaging pipelines.

[jasonjmcghee]

Only running ocr if the current image changed more than some small tolerance is a great optimization / low hanging fruit that would reduce energy usage.

[tleyden]

I think someone else mentioned this in another issue, but doing a lazy approach to OCR would probably be helpful too. For example, before searching, it could OCR the backlog of images.

And maybe the underlying VisionKit algorithms are more efficient if you pass batches of images rather than a single image, similar to training on a GPU where you typically pass batches of 64 images to the GPU.

[jasonjmcghee]

It could be very useful to decouple this as you describe. Could be even better to use https://github.com/jasonjmcghee/xrem as this agent to be cross platform, and have a different ui per platform, or common ui for non-mac.

This would involve performance benchmarking to compare rusty-tesseract to Apple VisionKit

[tleyden]

Xrem looks promising! 👀

For a cross-platform UI framework, I would check out Tauri. Tauri allows you to call rust plugins from the Javascript based UI, and with swift-rs it's pretty straightforward to make calls from vue.js -> rust -> swift -> VisionKit

[jasonjmcghee]

Yes, I started a discussion on lobste.rs and that was the conclusion i drew as well: https://lobste.rs/s/bt4enq/future_rem_cross_platform_gui_rust_2024#c_pmprqm

The repo now has tauri + status icon foundation, but not much there yet still