Run Alda in the browser

[daveyarwood]

My primary motivation is to have a "Try It" area on the Alda website, similar to what you find on the Haskell website, where you can enter snippets of Alda code, press a "play" button and hear the result.

There was also a request (#391) for playable demos of the example code in the Alda documentation, which I think is a great idea.

I've been thinking about either compiling the Alda client (written in Go) to WebAssembly, or using GopherJS to compile it to JavaScript. That should allow us to parse Alda scores in the browser and hopefully send OSC messages.

We'll also need to implement the player process (written in Kotlin) in the browser. It looks like it's possible to compile Kotlin to JavaScript, so maybe that will help. Although, I used Java inter-op quite heavily, both to use the JVM MIDI synthesizer/sequencer and to do sophisticated things with concurrency. So, I suspect that it might be challenging to try to reuse the same code base to generate a player process in JS. We might end up needing to write a port in JavaScript from scratch instead, which wouldn't be the end of the world. Ideally, we could use the same code base, though, so that we can avoid the two implementations drifting apart over time.

[severak]

This is exactly what I was trying to do while implementing alda-js. Maybe we should cooperate on that. 😉

There are three parts you need to implement:

• parser - takes ALDA score and parse it
• sequencer - creates sequence of MIDI events from parsed score
• player - plays that MIDI events

I have already implemented parts of parser, heavily borrowing from an MML parser. I am already able to parse some of the examples, but it turned out that ALDA syntax is somewhat more complicated than I expected. This part should be definitely ported from original implementation.

Sequencer part can be probably ported too. If you can create standard MIDI file you can already play it in javascript.

There is a good library called JZZ.js for dealing with all the MIDI stuff.

The player part is possible to implement in javascript too. There are several Web Audio based synthesisers I am even working on one. You basically need General MIDI synthesiser and good news is there is already one. However some sounds are somewhat crappy, but this just implementation detail.

[daveyarwood]

One complication is that the Alda client does not produce a MIDI file. It sends a bunch of OSC messages to the player process, and the player then uses those messages to build up a MIDI sequence in memory. At that point, the player process can either play the sequence or export it to a MIDI file. All of that would need to be implemented in the JS part of the port. (I'm hoping that at least some of this work could be done by compiling some of the Kotlin player code to JS.)

[daveyarwood]

I'd love to collaborate on this, BTW! I'm excited that you're also interested in making this dream a reality. :)

[daveyarwood]

Have you looked at MIDI.js? It looks kind of promising, and there is support for loading soundfonts, so we could make the instrument quality sound better.

[severak]

These older libs/projects like MIDI.js tends to be broken sometimes (because WebAudio/WebMIDI API changed in the meantime). This needs to be checked.

I can contribute javascript parts.

[daveyarwood]

@severak Maybe a good first step would be to look into JS libraries for receiving and parsing OSC messages. I could come up with a way to mock out the OSC messages that the Alda client sends, which could allow you to start working on the JS side of things until I have time (or someone else has time!) to dig into compiling the Go client to WASM or JS.

To give you an idea of the kinds of messages the Alda client sends, I wrote some good documentation here about the Alda OSC API.

There is a bunch of OSC message parsing code here in the Kotlin player that I'd ideally like to reuse by compiling the Kotlin to JS, so that we don't have competing implementations of the player process (JVM vs. JS in browser), but as a prototype, it might make sense to just whip up something simple in plain JS and see if we can get something working.

[severak]

@daveyarwood can be those OSC messages exported as text file? If that is possible, conversion to MIDI and playing is easy then.

[daveyarwood]

I can definitely make that happen. OSC messages are basically just a bunch of bytes, so instead of sending them to the player process, we could print them to stdout or dump them into a file.

When I have a little time, I'll see if I can hack in that change on a branch, so that when you use alda play, instead of sending the OSC messages, it simply prints them to stdout. (I'm tempted to even add this as a feature of Alda. It might be useful for debugging.)

[UlyssesZh]

Definitely a great idea! I used to encounter such a senario: I want to write a blog on music theory, and I want to demonstrate some concepts by using some short snippets of music. If I can just play music in the browser (best if in a similar UI to the HTML audio element) according to the Alda codes hard-coded in JavaScript/HTML, that would be extremely helpful! Hope alda-js or something can meet this need.

[daveyarwood]

@severak I made a quick branch of Alda where instead of sending OSC messages to a player process, it just dumps the OSC bundle to stdout in raw binary form. You can download the build here.

After extracting the alda executable for your OS/arch from the zip file and (if you're on Mac/Linux) using chmod +x to mark it as executable, you can use alda play to play a file or a string of code, and it will print the OSC bundle binary data to stdout:

$ ~/Downloads/alda play -c 'piano: c d e'
#bundle#{ /track/1/midi/patch,ii/system/tempo,ifB$/track/1/midi/volume,iid$/track/1/midi/panning,ii@0/track/1/midi/note,iiiii<E0/track/1/midi/note,iiiii>E0/track/1/midi/note,iiiii@E/system/play,/system/shutdown,i,

[severak]

@daveyarwood progress report:

• downloaded your modified branch and generated some OSC files
• examined it using text editor (oh no, this is binary format )
• figured out how to parse it using https://github.com/colinbdclark/osc.js

Now I need to implement translation to MIDI score and then I will be able to play it in browser (using built-in windows MIDI device for now).

[severak]

Translation from OSC to MIDI score is somewhat implemented - see source code and demo page.

It can play in Firefox if you have browser extensions from jazz-soft.net.

In Chrome it has some problems with permissions but you can at least download generated MIDI files.

[daveyarwood]

Nice, good progress!

Do you think it's possible for this to work without the user needing to install a browser extension? I'd really like for anyone who shows up to alda.io to be able to try the language right there in their browser, right away.

[severak]

Hi Dave,
I finally found out, how to implement soundfonts. There is soundfont editor called Polyphone which can translate soundfonts into SFZ. SFZ has major advantage - it's text based (which means it's easily parsed) and it can be loaded on demand (no need to load full soundfont). Stay tuned, another demo will be with soundfont.

[daveyarwood]

I've started looking into running the Alda parser/compiler/OSC generator in the browser.

I tried Browsix because it looks really compelling (a custom OS/kernel in the browser that I can simply run alda in without modifications? Sign me up!), but it isn't at all clear how to use it, and it seems to be early days still.

I tried GopherJS, but ran into some weird errors that I had a hard time debugging. That alone made me want to move on to another approach, which is to compile to WebAssembly using official Go tooling.

I think the WebAssembly approach is going well so far! I'll post here with updates as I have them. I think once I've got something working, I'll go ahead and deploy it to the Alda website in some way that isn't end-user visible yet - like the Alda client functionality could be available to use somehow in the browser (maybe by opening the JS console and running functions that I've defined in the global scope). That way, folks like @severak who are interested in working on the JS side of this can play around with Alda's actual OSC output in the browser!

[daveyarwood]

I couldn't stop there, so I've started tinkering on a branch with compiling the Alda player Kotlin source code to JavaScript. It's going pretty well so far, although I think it'll probably take a while. I'll keep chipping away at it in my free time. At this point, I've at least done enough to be confident that we can have a single code base where the compiler will help us avoid the two implementations (CLI/JVM vs browser/JS) growing too far apart.

@severak The work you've done in vanilla JS has been a valuable reference for me about how to actually do the whole "playing MIDI in the browser" part! I ended up doing a bunch of reading and exploring today about the Web MIDI API, JZZ, and MIDI.js. Some thoughts:

• The last commit in the MIDI.js repo was way back in 2015, which does not give me confidence that it will work well in 2022.
• JZZ is more actively maintained, like @severak mentioned, and that alone makes it more appealing to me.
• My top choice would be the Web MIDI API, since it's an official web spec, but at the moment, it isn't well supported. It's not currently available in Firefox or Safari.
  • We could use the Web MIDI API anyway and include a little message saying that you must use Chrome in order to use the live demo, but it would be way better if the Alda web REPL "just worked" on all modern browsers, and it seems like JZZ might be a way to make that happen.
• I was actually able to get JZZ to work without installing any plugins! Here is the code:

  alda/player/test-page/index.html

  Lines 46 to 62 in 18649e2

  	<!--
  	Temporary JZZ test code.
  	TODO: Bundle JZZ with alda-player.js?
  	-->
  	<script src="https://cdn.jsdelivr.net/npm/jzz"></script>
  	<script src="https://cdn.jsdelivr.net/npm/jzz-synth-tiny"></script>
  	<script>
  	function test_jzz() {
  	let tiny = JZZ.synth.Tiny();
  	const randomProgramNumber = Math.floor(Math.random() * 128);
  	tiny.program(0, randomProgramNumber).noteOn(0, 'C5', 127)
  	.wait(500).noteOn(0, 'E5', 127)
  	.wait(500).noteOn(0, 'G5', 127)
  	.wait(500).noteOff(0, 'C5').noteOff(0, 'E5').noteOff(0, 'G5');
  	}
  	</script>
  	<button onclick="test_jzz()">click me to make sound</button>

  It uses the JZZ-synth-Tiny plugin, which implements a basic GM synth using the Web Audio API. It sounds less than amazing, but I think it will be good enough for an in-browser Alda demo. Maybe in the future, we'll discover a way to make it sound nicer, but for now, I'm eager to at least get something working out there!

[severak]

@daveyarwood Nice to see that my experiments were valueable to you. I will look into your progress when I find time to do so.

JZZ is capable to act as Web MIDI API shim. I just use native JZZ API because I find it to be more comfortable.

Then you need to use some GM implementation, either your platform has decent one or you need to include your own (e.g. JZZ Tiny Synth). I wanted to implement one (see Heartbeat GM) but it seems
this is not an easy task.

However I am looking forward to see Alda working in web environment.

[daveyarwood]

Heartbeat GM is cool, nice work! ❤️

I noticed the Web MIDI shim capability too, but when I tried it, it didn't quite work for one reason or another. I forget exactly what the issue was, but it seems like it's missing the full functionality of the Web MIDI API. And I figure that as long as I'm using JZZ, I may as well just use the JZZ API, you know?

[gedw99]

great that you got it outputting to WASM @daveyarwood
this is really exciting now...

Maybe you should consider GIO which is pure golang and outputs wasm ( browser), desktop and mobile for making advanced GUI's. YOu get full control. It can play sound too.

Links:
https://gioui.org/
https://github.com/gioui/gio-example has various examples. The Component one is decent...

I know another golang dev on github that is also building a music editor and player with gio in golang.
if you want I can try to find it.

[gedw99]

here is the example music visualiser in golang.
https://github.com/fiwippi/currents

It is designed to play on a arduino though.. ) but anyway shows the audio working.

it uses https://github.com/gen2brain/malgo to play the audio.

[daveyarwood]

It's good to know about these libraries! I think the Go part of the "Alda in the browser" project is already complete at this point, and the remaining work is to port the (Kotlin) player process to JS in the browser. I'm still chipping away at that in my free time. ⛏️

[gedw99]

ok great.

i also wanted to just show you this..
this is similar to alda because it has a DSL that drives the GIo gui.
Its pumping out graphics. Alda is pumping out music. same same archi but different medium.

https://github.com/ajstarks/decksh/blob/master/cmd/decksh/test.dsh

[gedw99]

i used https://github.com/hajimehoshi/oto with gio. plays audio on any device

all native plugins for gio are here: https://github.com/gioui/gio-x

you just use oto in the same way.

[severak]

@daveyarwood where can I download WASM/javascript version of Alda?

I just started fiddling with music software again so I decided to look into this.

[daveyarwood]

The latest version of alda.wasm can always be found at: https://alda.io/assets/wasm/alda.wasm

(EDIT: Actually, that's not totally true. It's the latest version of alda.wasm that was available at the time that the website was last updated. You can see what version of Alda this corresponds to by running Alda.VERSION in the JS console after loading the WASM, or you can do it in the JS console at https://alda.io)

The website build includes a step where we download the latest published version of alda.wasm from the Alda releases API, and include the alda.asm file as an asset on the website.

[severak]

While reasearching something other I looked into this and found out what is missing here. Actually not much.

I am going to implement it. Stay tuned!

[severak]

This is big - WE HAVE IT!

I have connected all bits and pieces and now we have finally textarea where you can enter Alda code and have it played by browser - see here (and source code). It's based on your WASM version of Alda, my hackish OSC2MIDI converter and JZZ lib (with Tiny synth) for playing that midi.

We just need to polish it and pack it for Alda homepage.

On Firefox I am able to select buit-in Microsoft GS Wavetable Synth which sounds a bit nicer than Tiny synth. In theory it's possible to use some soundfont playing javascript contraption or my own Heartbeat GM once it has more programs implemented.

[gedw99]

this is freaking awesome !!

[gedw99]

it works in safari on my MAC btw && safari on IOS
https://severak.github.io/alda-js/

[daveyarwood]

Fantastic work, @severak ! It works out of the box for me on both Chrome and Firefox, and even on Chrome on my Android phone!

Long term, I am working on a branch of Alda where the Kotlin code produces both the CLI program alda-player that we have today, as well as a JavaScript version that runs in the browser, working with alda.wasm (just like what your code is doing). The advantage of doing it this way is that it will be easier to keep the JavaScript version up to date with changes to the CLI version. As new features become available in Alda itself, it will be easier for us to also include them in the JS version on the website.

My progress on this has been very slow (I'm super busy these days between work and family!), so I'm happy to see you have a working version in the meantime! I think there would be value in adapting your code to integrate with the Alda website in the short term, and in the long term, I can work towards finishing the Kotlin-JS version to replace it. I'll probably end up stealing a lot of your code in the process :)