Using XS native Compartments for Agoric Vat Workers

[kriskowal]

[update 2024-04-29]

Agoric currently bundles contracts in a pre-compiled JSON envelope to avoid the runtime cost of loading and using Babel to transform ESM to a form that SES can digest. We could instead rely on the native XS ModuleSource behavior on XS if the native Compartment were sufficiently similar to the SES emulation of Compartment.

• Change the Agoric/Endo bundle format such that it contains original sources instead of “precompiled” sources.
  • Refactor Bundling, Compartment Mapper, and SES together such that existing applications continue to function in Node.js and XSnap except using the XS native Compartment and ModuleSource instead of Babel. In the current world, bundleSource applies the module-to-program transform unconditionally. In the new world, bundleSource would capture original sources, but importBundle will use the Babel transform on Node.js and native ModuleSource on XS.
    • Export native XS ModuleSource in lieu of StaticModuleRecord on XS #2250
    • SES: Adapt Native XS Compartment to SES Compartment #2251
      • refactor(ses): Retool for module descriptors #2321
      • feat(ses): Module descriptors for XS parity #2345
      • refactor(compartment-mapper): Pivot to module descriptors #2350
      • feat(ses): Opt out of namespace boxes #2380
      • feat(ses): Opt out of namespace boxes #2380
      • feat(module-source): Add XS specific variant #2470
      • feat(ses): Add XS variant of shim #2471
    • Integration tests for XS Node.js bundler parity #2252
      • Either use Rollup for Node.js ModuleSource shim preamble or for test262-harness to support Node.js import preamble
      • Generate bundles with untransformed original sources instead of precompiled sources. #2295
      • feat(compartment-mapper): JSON module support for makeBundle #2352
      • feat(ses,module-source): Add ModuleSource shim #2463
      • test: Introduce XS CI workflow #2465
    • Test parity and integrity between XS and SES Compartment #2259
      • feat(hardened262): Preliminary harness #2593
    • Move Babel dependency from bundleSource to importBundle #2254
      • Using an "imports" directive in @endo/import-bundle such that it only entrains @endo/evasive-transforms on Node.js and not with the "xs" tag.
        • Compartment Mapper: Support "imports" directive in package.json #2203

Enables #2116, #2117

It follows that we will be able to close when we have adequate parity with native XS for virtual module source on XS #1066

---

[update 2023-12-19]

Agoric currently bundles contracts in a pre-compiled JSON envelope to avoid the runtime cost of loading and using Babel to transform ESM to a form that SES can digest. We could instead rely on the native XS ModuleSource behavior on XS if the native Compartment were sufficiently similar to the SES emulation of Compartment.

• establish 262 style tests that prove that we have adequate parity between XS Compartment and SES Comaprtment. This is likely to require changes to SES and a specialized thin variant of SES just for use on XS.
• alter import bundle to take a dependency on @endo/static-module-record on Node.js but rely on the native ModuleSource constructor when running on XS. Differentiate these cases with a tag like endo in the package.json "exports" or "imports" property so that the SwingSet worker bootstrap for XS doesn’t entrain Babel when we generate that bundle.
• alter compartment mapper to embed original sources instead of precompiled sources

---

[update 2020-01-21]

Agoric currently runs contracts in an XSnap worker using the SES shim for dynamic module-loading Compartment support. This works for now, and Moddable has made considerable progress toward making their native Compartment implementation a better alternative. To run a contract in a native Compartment, we would not need to use the bundleSource “censorship evasive transforms” that mitigate the weaknesses in the ses (hardened JavaScript shim) emulation of ESM (ECMAScript modules).

This issue stands open to track native compartment support. At time of writing, the ball is in our court. We need:

• to move XSnap to the Endo repository so we can run common tests against both SES shim and XS native
• adjust the SES shim to conditionally reveal the native Compartment when it’s available
• get a test262 runner working in Endo CI Ava Test 262 runner #1001
• converge on implementation differences that we have agreed upon in design SES: Compartment: Rename importHook to loadHook #420, Sync spec and XS interpretation of Date(), Date.now(), and Math.random() #934

In the original description below, we are running the stopgap in production, and the longshot is not such a long shot anymore.

---

There are two avenues to running contract archives on Moddable’s XS. Both involve creating an XS binary for hosting dynamically-loaded modules from an archive (presumed Zip), including both CommonJS and ECMAScript modules. In both scenarios, this binary would implement Endo, ideally reüsing as much of its internals without modification as possible, so we have parity between running on Node.js with SES-Shim and running on XS without.

1. Longshot: Fill out the XS Compartment API (C) such that it can incorporate modules at run-time.
2. Stopgap: Emulate run-time modules using XS’s Compartment evaluator.

The Longshot is the most desirable long-term outcome. XS should in principle not need any part of the SES shim, since it’s a SES-only runtime. Also, our friends at Moddable are committed to implementing the Compartment specification, though the specification remains negotiable. To pursue this option, we may need to engage more closely with the XS project.

The Stopgap is possible (modulo risks) without major changes to XS, and largely reusing work we have already done on SES-Shim. We would need to implement the dynamic module loading feature as an xs-compartment-shim. This would reüse the transform-module Babel shim. We would need to factor some code out of the SES compartment-shim, presumably into the make-importer project Michael sketched.

The work break-down for the Longshot includes:

1. XS Compartment would need to implement the StaticModuleRecord constructor, so ECMAScript modules can be parsed and analyzed.
2. XS Compartment would need to implement importHook.
3. On the first pass, importHook would need to be able to return a static module record provided by the above StaticModuleRecord and the Compartment and StaticModuleRecord can communicate behind the scenes with internal slots or weak maps to hide implementation details.
4. On the second pass, importHook ould also support CommonJS, JSON, and potentially other modules by permitting the importHook to return implementations of static module record with the currently unspecified { imports, execute(exports, compartment, resolvedImports) } interface. This is possible as of feat(endo): Add CommonJS and JSON module support #397 on Node.js and would be necessary for parity, though we could constrain the transitive dependencies of contracts to be limited to the currently rare ESM format.
5. XS Compartment would need to implement resolveHook. There is some wiggle room here since we actually just need every Compartment to use our particular resolveHook that works like Node.js's resolver, but where every full specifier is either package-local (starting with . or ..) or from another package. With this hook, resolve('./aux.js', './main.js') is ./aux.js, whereas resolve('../foo.js', './main.js') is an error since it escapes the package root. Folks who have implemented resolution for Node.js do not use this math since Node.js effectively executes within a single compartment where module specifiers are fully qualified and resolution may involve IO. This would be antithetical to compartmentalizing third-party packages like LavaMoat.
6. XS Compartment and SES-shim Compartment would need to settle some minor differences or Endo would need to feature-test and work-around them. Notably SES-shim exposes import (which returns a promise) but not the importNow. This is likely a case where the specification and the SES shim need to change, since distinguishing importNow will no longer beuseful when the Compartment specification meets the new top-level-await.

The work break-down for Stopgap includes:

1. We would need to factor the Compartment constructor out of SES-shim such that it can be used by Endo in XS without the lockdown shim.
2. We would need a tool that turns an application that uses Node.js-style packages into a single namespace (so they can be precompiled into an XS binary) and a JavaScript" kernel that partitions the packages into child compartments and executes them in topological order. This is because the XS Compartment only supports pre-compiled ESM and does not support Node.js-style packaging. The “kernel” must be a single package of ECMASCript modules.
3. The transitive dependencies of the XS binary application would need to all be ECMAScript modules. If we got the longshot, we could write a much smaller bootstrap and use Endo both for the application-runner (which depends on a CommonJS Zip library and Babel) and the application.
4. We will need to port a Zip library to ESM.
5. We will need to ensure that Babel can run under ESM, instead of the CommonJS projection of its ESM implementation.

[dckc]

... This is because the XS Compartment only supports pre-compiled ESM and does not support Node.js-style packaging. The “kernel” must be a single package of ECMASCript modules.

I wonder if that's quite true. I'm not sure what "Node.js-style packaging" means exactly. XS can import modules by reading .js from the filesystem at runtime. (I was in the past under the impression that it cannot, and I spread this misinformation around a bit, but I was wrong. IOU pointers to details)

I have incomplete understanding of importHook and a few other terms above, so I'm not sure how to share what I know about this topic. A meeting might be cost-effective.

[kriskowal]

Thanks @dckc. Importing off the filesystem might be useful in some special cases. It would be useful to see an example of, say, writing a file to disk then importing it.

[dckc]

here's hoping I find time to cook up such an example. (I'm pretty sure I've done it before... maybe I still have it around...)

Meanwhile, for reference: fxLoadScript and fxLoadModule a few lines previous.

[dckc]

here's hoping I find time to cook up such an example. (I'm pretty sure I've done it before... maybe I still have it around...)

found it: https://gist.github.com/dckc/cbbd3e8469723b342cc90799ace7a287
from Jan 23 Agoric/agoric-sdk#426 (comment)

[kriskowal]

The stopgap I described here has been working for some time now. Moddable has also recently made incredible strides toward making the longshot work too. I’m revising the title and description so that this issue continues to track native compartment support. #848 is related.