[META] Fix Problems Relating to macOS Notarization/Signing

[bluebandit21]

Introduction

Etterna is currently distributed as an unsigned app on all platforms, including macOS.
Windows throws some mildly-scary looking warnings up to users (which most people are conditioned to bypassing anyway), but macOS takes many active measures to severely limit the functionality of unsigned apps for security reasons.

At this point, many bugs exist and have been encountered by users stemming indirectly from this problem, and they will only become greater and harder/impossible to work around in the future.

App Translocation:

By default, macOS prevents apps from accessing resources relative to the app's location.
This is to prevent a class of attacks known as "dylib hijacking", wherein an attacker maliciously replaces external resources used by an app, which can potentially cause even signed intact apps to exhibit unintended malicious behavior. [1]

To fix this, recent versions of macOS do not always launch apps with a current working directory equal to where the app actually lives on disk. Instead, apps are sometimes launched from a read-only dynamically created volume under /private/var/ containing the app and any internal resources. This process is known as "App Translocation", and occurs when certain conditions are met, including the launching of an unsigned app with the com.apple.quarantine xattr set (which occurs automatically when apps are downloaded through Safari) [2] [3] This breaks Etterna because we intentionally store the Songs and Saves directories relative to Etterna.app itself, instead of within ~/Library/Application\ Support like Apple intends us to.

We currently work around this in an incredibly hacky way, by explicitly checking if we are undergoing app translocation, locating the real location of Etterna.app if so, and directly removing the com.apple.quarantine xattr from our own app ourselves before then killing and relaunching Etterna. See:

etterna/src/Core/Platform/PlatformMac.mm

Lines 24 to 69 in 9a0b9fb

	// Apple API Translocation variables
	// Apple introduced app translocation in macOS Sierra v10.12, which prevents apps from accessing files outside
	// their own .app directory (via a relative reference). This is a problem for Etterna since all files are accessed with
	// relative references to where the .app directory is located. Apple moves the app to a secure location, then
	// attempts to run the program. Etterna fails right away as it requires those local files. The code in init()
	// will load Apple's private security API, load the functions, and call on them to untranslocate by removing
	// the quarantine restriction on the original binary location. Apple will not accept the app into their app store
	// unless this code is removed.
	void (*mySecTranslocateIsTranslocatedURL)(CFURLRef path, bool *isTranslocated, CFErrorRef* __nullable error);
	CFURLRef __nullable (*mySecTranslocateCreateOriginalPathForURL)(CFURLRef translocatedPath, CFErrorRef * __nullable error);
	namespace Core::Platform {
	void init(){
	// Load Apple private security API
	void *handle = dlopen("/System/Library/Frameworks/Security.framework/Security", RTLD_LAZY);
	if(!handle){
	Locator::getLogger()->info("Unable to load security framework!\nSkipping translocation check");
	return;
	}
	mySecTranslocateIsTranslocatedURL = reinterpret_cast<void (*)(CFURLRef, bool *, CFErrorRef *)>(dlsym(handle, "SecTranslocateIsTranslocatedURL"));
	mySecTranslocateCreateOriginalPathForURL = reinterpret_cast<CFURLRef __nullable (*)(CFURLRef, CFErrorRef * __nullable)>(dlsym(handle, "SecTranslocateCreateOriginalPathForURL"));
	if(!mySecTranslocateIsTranslocatedURL || !mySecTranslocateCreateOriginalPathForURL){
	Locator::getLogger()->info("Unable to find security framework translocation functions!\nSkipping translocation check");
	return;
	}
	// Check if we are translocated
	bool isTranslocated = false;
	mySecTranslocateIsTranslocatedURL((__bridge CFURLRef)[NSURL fileURLWithPath:[[NSBundle mainBundle] bundlePath]], &isTranslocated, nullptr);
	if(isTranslocated) {
	Locator::getLogger()->info("App is translocated! Removing com.apple.quarantine extended attribute");
	// Get original app location
	auto untranslocatedURL = (NSURL *)mySecTranslocateCreateOriginalPathForURL(static_cast<CFURLRef>(NSBundle.mainBundle.bundleURL), nullptr);
	// Remove quarantine flag
	std::system(fmt::format("xattr -rd com.apple.quarantine {}", untranslocatedURL.path.UTF8String).c_str());
	// Reopen ignoring other instances
	std::system(fmt::format("open -n -a {}", untranslocatedURL.path.UTF8String).c_str());
	// Close this instance
	std::exit(0);
	}
	}

This workaround is bad for a number of reasons:

1. It is fragile, in that it relies upon the internal function SecTranslocateCreateOriginalPathForURL in the Security framework [4]
   This is certainly something Apple doesn't want us doing and relies upon an internal function that could change or disappear in the future.
2. The roundaboutness of the way we try to do this has caused problems

• Sometimes, removing the quarantine xattr from only Etterna.app itself doesn't seem to be enough, and it must be removed from additional resources like crashpad_handler: [Feature Request]: Improve first-time launch macOS experience #1128
• Sometimes, we fail to untranslocate ourselves for whatever reason (e.g. Etterna mistakenly being launched from a read-only volume), and the way our current code responds is by repeatedly opening new Etterna instances which then kill themselves after opening others in an eternal loop: [Bug]: Fix translocation removal behavior when unexpected things occur #1174

3. It appears macOS Ventura/Sonoma may have once again changed how the quarantine process works -- I (bluebandit21) had a Discord DM conversation helping someone with problems they encountered with our unquarantining process; see also [5]
   This might make our approach stop working, as it appears only certain trusted apps (e.g. Terminal) can interact with the protected com.apple.provenance xattr
   TODO: Expand this further/confirm?

Input Handling

Currently, we use a horribly outdated API for handling low-level input on macOS, as described by #1260 and other issues/discussions.

This API requires elevated permissions to be allowed for use by an app, namely the Input Monitoring permission, without which certain API calls fail with a lack-of-permissions error. (TODO: Expand, link to relevant detailed sources)

Unfortunately, the system for handling per-app permissions seems to somehow interact with whether the app is signed or not; sometimes the com.apple.quarantine xattr seems to have to be removed before it's even possible to grant Etterna permissions to Input Monitoring in the first place! (TODO: This is possibly also related to us not having a valid CFBundleID; investigate further...)
See: https://gist.github.com/bluebandit21/d34d7b0b4f9137c34c167e607db63c03

Native M-Series (Apple Silicon) Builds

To ease the transition when Apple switched from Intel's x86_64 chips to their own in-house ARM (aarch64) chips, Apple created the library Rosetta v2 which transparently and efficiently lets Apps compiled for the x86_64 architecture run on aarch64, via a JIT compilation layer. [6] This fortunately means that existing Etterna apps built for x86_64 do not encounter any more problems than we already do when trying to run on M-series machines.

However, we definitely want to get native ARM macOS builds working at some point for two main reasons:

• One, we take a moderate, but not insignificant, performance hit by operating through the rosetta v2 translation layer.
• Two, eventually Apple will stop support for x86_64 apps in the future, and the only way to run apps on modern Apple computers will be by compiling them targeting ARM. When Apple released Rosetta v1 for the PowerPC to Intel transition, they supported PowerPC apps from 2006 until 2011, or five years. [7]
  Apple released Rosetta V2 in 2020; if we assumed the same five-year lifespan for it, we'd expect them to drop support sometime in 2025, less than two years away from the time of this issue's creation!

Why is this of potential concern w.r.t. notarization/signing? Because modern versions of macOS require any apps running natively on M-series to be signed and notarized to run!

Beginning in macOS 10.14.5, software signed with a new Developer ID certificate and all new or updated kernel extensions must be notarized to run. Beginning in macOS 10.15, all software built after June 1, 2019, and distributed with Developer ID must be notarized. However, you aren’t required to notarize software that you distribute through the Mac App Store because the App Store submission process already includes equivalent security checks. [8]

Conclusion:

We should probably actually figure out how to notarize ourselves on Etterna!!
The most straightforward route would probably be to have a single individual developer for Etterna (most likely @jameskr97 or @bluebandit21 ) pay for a personal Apple Developer license at the cost of $99/year and sign all releases intended for public distribution. [9]
This would likely have to be a manual process, or we'd have to very carefully modify the CI to make it possible for this signing to occur without leaking the personal signing information of the said developer...

Another possible way would be establishing an Organization Apple Developer account for Etterna, also at the cost of $99/year. This obviously has the relative downside that we'd have to somehow generate this money specifically for Etterna, while the "Individual random developer" route might be "free" in the sense that we might just have access a developer that already pays this cost and lets us use their signing identity at no additional cost to either us or them.
However, it is theoretically possible we could get this fee completely waived for us by applying for a non-profit fee waiver.[10]
This would require Etterna to have a legal Non-Profit status, which might be much more work than the potential benefit it provides.

[bluebandit21]

@jameskr97 Could I ask you to quickly double-check this hecking essay of an issue post for correctness?
I believe everything I'm saying is correct and should be well-supported by the many citations I've included inline, but always good to have a second set of eyes :)

[bluebandit21]

Note: I snapshotted all of the resources I link to on archive.org, so they should be available there if Apple takes down/moves random documentation pages as they like to do :)