[OldNotes] AAP 2020 Year in Review
I haven't really blogged about my ongoing project called Android Audio Plugin Framework = AAP here. I didn't because it's not quite ready for the public uses. So it may die at any stage. But it is good to record and publish what is going on there (in fact I did so in Japanese for a while, but without speaking it loudly). It is literally, an audio plugin framework for Android. Like VST, AU, or LV2.
This post has premises on very basics on what an audio plugin framework is in general. I skip explanation on that.
Developing another audio plugin framework is not cool and it's best to avoid that. But we need one for Android anyways. There should be feature parity in Android for iOS AudioUnit. Here's how it is different from others, in primary aspect:
In Android, all apps are isolated to each process with its own access rights. To make DAWs and audio plugins diverse, they have to be published by respective developers and hence as different apps. Otherwise it will be "only one DAW/host ecosystem rules", which is not what I want.
In fact AU v3 is quite similar to AAP in terms of app process model (well, it is the other way, AAP is quite similar to AUv3).
The overall design is inevitable but goes against general audio app design. DAWs in general manage audio plugins in one process (the application process itself) and crash together with plugins. Later on, some DAWs started offering plugin containers like Google Chrome did back in 200x. Bitwig Studio is famous for that. Ardour team didn't like the idea. Some other DAWs seem to provide intermediate solution like one single audio engine process apart from the main application. Tracktion Waveform11 has such a feature (if you have crashy plugins, you'd see a lot of "Audio engine crashed" message dialogs), but while its usability is good enough I would take it rather as a single process model here. Unlike Bitwig Studio, they don't have to provide the feature to separate multiple plugin processes.
To connect the plugins we resort to Android Binder IPC mechanism, which is known as good in performance and potential support for realtime priority inheritance in the future (which is disabled in general at /dev/binder now), but comes with a bunch of mutex_locks, which is in general no-go for professional audio. We may revisit this area in the future.
There are many issues to sort out for a new plugin framework. The hardest hurdle is to get many working plugins. Another issue is to not break the API, which is annoying for plugin and host developers.
Fortunately, the former is getting less problematic year by year. What audio plugins do can be mostly cross platform, and there are many developers who switched to such solutions. The most popular cross-platform audio development framework is probably JUCE. iPlug2 is another option with long history (considering that it is from WDL). They are there to ease per-platform and per-plugin-framework development.
With all recent development in CMake support and VST3 support in Linux, JUCE is becoming (again) a truly usable cross-platform audio plugin development framework. You can develop VST v2/v3 or AU plugins in their standard distribution, and there are some community effort that supports LV2. And JUCE already supports Android, just not for audio plugins (because there is no plugin framework, which is why I am working on this project).
JUCE audio plugin format API comes with some abstraction, and it is fairly easy to extend that to our own framework. Once we developed a bridge to JUCE API and a bit of build tweaks, we can import a lot of JUCE based plugins in our world. I have already imported a handful of plugins in aap-juce repo (which is under ongoing refactoring these days). And what is important at a time is, we don't really expose our framework API internals to those plugin developers at all. They are not even aware of it. We can make API changes until we provide stable plugin API.
The most important remaining issue is to connect a plugin UI from a DAW (host), but the audio parameters can be retrieved and states can be store and restored.
LV2 is another bunch of low-hanging fruits (well, it might not be that low, up to plugins). There are couple of popular audio plugins or can-be-a-plugin audio apps on Linux desktop, and they are in general portable to Android, unless they rely on some desktop technology (like access to local files). I have ported LV2 toolkits to Android world, which is somehow listed on the official wiki, and aap-lv2 repo contains the support library and some tool to import metadata, as well as ports such as Fluidsynth (I wrote tiny code for it to work as an LV2 plugin), sfizz, and Guitarix (all repos linked from there). No UI again, but since LV2 plugin offers its parameter list via manifest, it can still work, to some extent.
(VST3 is another option. Steinberg has made vst3sdk partially portable enough to build some of their plugin examples on Android for a while. Last time I tried it, they had dependency on std::filesystem so I quickly dumped the idea. Since Android NDK r22 supports std::filesystem, it may be working now. Depending on local filesystem means it most unlikely works as is on Android though.)
Plugin metadata compatibility is somewhat different story, but that is relatively a minor issue, and schema changes (runtime compatibility) is easier to manage than code changes.