Create a AssemblyScript Resolver

[willemneal]

What package is covered by this investigations?

https://github.com/AssemblyScript/assemblyscript: TypeScript -> Wasm compiler.

Describe the goal of the investigation

Create portable tool that uses yarn to handle fetching dependencies. Since AssemblyScript is compiled, a cool feature would be to fetch dependencies as needed. This way you don't need to download the full package, but rather only the files and dependencies that get used.

Investigation report

As plugins already support an abstracted filesystem, it would be great to have this work in a browser setting.

I'm new to Plug'n Play would it be useful in this setting?

Any advice on where to start?

[arcanis]

Hey! That sounds interesting. I have some questions to better understand the ramifications (I've only heard about AssemblyScript so far, and never tried it myself, so please bear with me 😄):

• Who would you see as the primary audience? People writing Javascript and consuming AssemblyScript packages, or people who intend to build AssemblyScript binaries?

  • In the first case, since AssemblyScript compiles down to WebAssembly, I think people would typically consume directly the WebAssembly package. First-class WebAssembly support in Yarn would be very interesting, and I know that some actors are following this with interest.

  • The second case looks particularly interesting for the AssemblyScript case, as you'd typically prefer to depend on the sources in order to get their types (I think?). Unless the types are shipped as part of the generated bundle (which would thus contain wasm + .d.ts). That's the second question:

• How are dependencies currently handled in AssemblyScript? Are they loaded from one place in particular? How does AssemblyScript supports import? How would a library expose its types to a dependent project?

  • I've seen this issue which seems to imply that there is some at least partial support. How good is it? What could we do to make it better from a developer experience perspective? cc @MaxGraey

[willemneal]

Thanks for the quick reply. To answer your second set of question first, we currently have a node-like resolution scheme that searches node_modules or passed in paths, from the current location to the /node_modules, etc. We look for a package.json that contains an ascMain field, which points to the modules entry. If it isn't found then we look for a assembly/index.ts. Then when parsing the file, we associate all imports to the location we found it so that we can start looking from there when they are resolved.

As for types, this is something I'm particularly interested to add. Currently we have to use static linking when compiling, meaning that we indeed do need all sources referenced at compile time. However, WebAssembly Interface Types provides a way to import types used by exports. Whereas currently the only types you can have are floats and ints. This also means it will be easier to stitch together wasm binaries at load time.

In the short term yarn would help the compiler to either to set up the files in an abstract environment that the compiler shares before compiling. (Currently the compiler has arguments for read, write, and list files). Or while it is compiling and resolving a file, it would ask yarn to resolve/fetch the dependency.

As for your first case, this seems the most promising: https://github.com/CraneStation/wasmtime-demos/tree/master/nodejs

With the Interface Types I mentioned you create adapters for encoding and decoding types via an abstract interface. So then you can very easily import a wasm module, not that must be compiled and then run with the WebAssembly API, but rather node or any other native language runtime will see the module as if it was imported by the language itself.

Another aspect is that currently the compiler is pretty fast, so really it would be nice to have a node like interface that compiles and caches the wasm before executing it with something like https://github.com/wasmerio/wasmer-js

For now let's say I wanted to go with the first way that yarn could help, which is to just run yarn install in a sandboxed environment, how would I go about that?

[arcanis]

I see, thanks for the explanation!

In the short term yarn would help the compiler to either to set up the files in an abstract environment that the compiler shares before compiling. (Currently the compiler has arguments for read, write, and list files). Or while it is compiling and resolving a file, it would ask yarn to resolve/fetch the dependency.

Yarn has a concept of linkers. They are used to declare how packages should be laid out on the disk. In the case of the v2 we register Javascript packages into the PnP map that we store in the .pnp.js file, but back in the v1 we would have generated a node_modules directory instead.

This mechanism allow you to do pretty much what you want regarding the way packages are stored. You could for example generate a ascDeps.json file that would list all top-level packages available that list the ascMain field you mention, and their location. Then the compiler would just read this file and know where to find the dependencies.

For now let's say I wanted to go with the first way that yarn could help, which is to just run yarn install in a sandboxed environment, how would I go about that?

• First I'd look at the two following files: Linker.ts and Installer.ts. They describe the general idea behind linkers, and plugins that want to provide their own need to implement those interfaces. A good example is PnpLinker.ts which as I mentioned aggregates all the packages in the dependency tree in a single .pnp.js file.

• Then you'd settle on a languageName value. It's a package.json field that we plan to use in order to detect the linker that should be used with a specific package (I don't think we'll support one package being installed by multiple linkers - in this case it would be two different packages).

  • Note that the NpmResolver current hardcodes the node language name. We'll need to change that into manifest.languageName ?? 'node' so that packages can define their own.

• Then you must write the plugin that defines the linker. We have a tutorial that shows the basics, and you can look at the plugin-pnp for a better idea. Note that you also can use the @yarnpkg/builder package to automatically build a plugin like Yarn would expect. It's not very well documented at the moment, but all the plugins inside our repo use it so you should have plenty of examples.

• If the compiler is written in Node, maybe the easiest way would be to start by copy-pasting the existing PnP linker and slightly adapt it to suit your needs:

  • Replacing the supportsPackage implementation to cover only AssemblyScript packages (the regular PnpLinker will also have to be tweaked to only cover node, btw).

  • You might want to force all packages to be "unplugged" by returning true here. To give you some context, an unplugged package is extracted from its zip archive and stored on the disk uncompressed.

    • The PnP loader is able to load files directly from the archives so we usually don't need to unpack them (except when they have build scripts, obviously), but in your case I guess the compiler cannot do it yet hence why you'd want to unplug everything.

    • In order for the Node and AssemblyScript package files not to collide, we'll likely want to make the unplugged folders different. I think we'd better move .yarn/unplugged into .yarn/unplugged/pnp, and this way you'll be free to use .yarn/unplugged/pnp-asc or similar.

  • You'd rename the .pnp.js file to something like .pnp.asc.js.

• And finally, in your compiler, you'd require('./.pnp.asc.js'), and query Yarn for the location of your packages through the standard PnP API. In most cases, it would amount to using resolveToUnqualified to get the base resolution (which you'd then run through your own resolver to resolve the ascMain field).

Note that the linker API is very new (we only have a single linker at the moment), so it's likely we'll have to change some pieces. That's not a problem, that's why we're still in beta 🙂

And finally, I'm often available on our Discord channel and will be happy to answer any further question! GitHub is fine too, of course.

[willemneal]

Thanks for all the info. I'll dive into soon. One last question: is resolveToUnqualified synchronous?

[arcanis]

One last question: is resolveToUnqualified synchronous?

Yep. It doesn't even query the filesystem.

[willemneal]

Beautiful! This is perfect!