Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(import-bundle): Support endoScript #2398

Merged
merged 3 commits into from
Aug 31, 2024
Merged

feat(import-bundle): Support endoScript #2398

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4af8ebf
feat(import-bundle): Support endoScript
kriskowal Aug 2, 2024
91a7351
docs(import-bundle): Update bundle source formats, including endoScript
kriskowal Aug 2, 2024
029330f
docs(import-bundle): Wrap lines
kriskowal Aug 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions packages/import-bundle/NEWS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
User-visible changes to `@endo/import-bundle`:

# Next release

- Adds support for `endoScript` format bundles.

# v1.2.1 (2024-08-01)

- Fixes support for `inescapableGlobalProperties` in the `endoZipBase64` format
Expand Down
106 changes: 85 additions & 21 deletions packages/import-bundle/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# import-bundle

`importBundle` is an async function that evaluates the bundles created by `bundle-source`, turning them back into callable functions:
`importBundle` is an async function that evaluates the bundles created by
`bundle-source`, turning them back into callable functions:

```js
const bundle = await bundleSource('path/to/bundle.js');
Expand All @@ -10,21 +11,53 @@ const namespace = await importBundle(bundle);
const { default, namedExport1, namedExport2 } = namespace;
```

This must be run in a SES environment: you must install SES before importing `@endo/import-bundle`. The conventional way to do this is to import a module (e.g. `@endo/init`) which does `import 'ses'; lockdown();`.
This must be run in a SES environment: you must install SES before importing
`@endo/import-bundle`.
The conventional way to do this is to import a module (e.g. `@endo/init`) which
does `import 'ses'; lockdown();`.

The bundle will be loaded into a new Compartment, which does not have access to platform globals like `document` or `Fetch` or `require`. The bundle is isolated to only having access to powerless JavaScript facilities and whatever endowments you provide.
The bundle will be loaded into a new Compartment, which does not have access to
platform globals like `document` or `Fetch` or `require`.
The bundle is isolated to only having access to powerless JavaScript facilities
and whatever endowments you provide.

Each call to `importBundle` creates a new `Compartment`. The globals of the new Compartment are frozen before any bundle code is evaluated, to enforce ocap rules.
Each call to `importBundle` creates a new `Compartment`.
The globals of the new Compartment are frozen before any bundle code is
evaluated, to enforce ocap rules.

## Module Formats

The source can be bundled in a variety of "formats". By default, `bundleSource` uses a format named `getExports`, in which the source tree is linearized into a single CommonJS-style string, and wrapped in a callable function that provides the `exports` and `module.exports` context to which the exports can be attached. `importBundle` recognizes this format, and invokes the function thus defined, to return the namespace object.
The source can be bundled in a variety of "formats".

A more sophisticated format is named `nestedEvaluate`. In this mode, the source tree is converted into a table of evaluable strings, one for each original module. This table is then encoded and wrapped as before. The evaluation process uses a separate evaluator call for each module, providing an opportunity to attach a distinct `sourceMap` to each one. This preserves relative filenames in subsequent debugging information and stack traces.
By default, `bundleSource` uses a format named `endoZipBase64`, in which the
source modules and a "compartment map" are captured in a Zip file and base-64
encoded.
The compartment map describes how to construct a set of [Hardened
JavaScript](https://hardenedjs.org) compartments and how to load and link the
source modules between them.

To set a base prefix for these relative filenames, provide the `filePrefix` option.
The `endoScript` format captures the sources as a single JavaScript program
that completes with the entry module's namespace object.

Note that the `nestedEvaluate` format requires an endowment named `require`, although it will only be called if the source tree imported one of the few modules on the `bundle-source` "external" list.
The `getExport` format captures the sources as a single CommonJS-style string,
and wrapped in a callable function that provides the `exports` and
`module.exports` context to which the exports can be attached.

More sophisticated than `getExport` is named `nestedEvaluate`.
In this mode, the source tree is converted into a table of evaluable strings,
one for each original module.
This table is then encoded and wrapped as before.
The evaluation process uses a separate evaluator call for each module,
providing an opportunity to attach a distinct `sourceMap` to each one.
This preserves relative filenames in subsequent debugging information and stack
traces.

To set a base prefix for these relative filenames, provide the `filePrefix`
option.

Note that the `nestedEvaluate` format receives a global endowment named
`require`, although it will only be called if the source tree imported one of
the few modules on the `bundle-source` "external" list.

## Options

Expand All @@ -34,19 +67,50 @@ Note that the `nestedEvaluate` format requires an endowment named `require`, alt
const namespace = await importBundle(bundle, options, powers);
```

The most common option is `filePrefix`, which can be provided for `nestedEvaluate`-format bundles. This sets the source filename of the top-level module inside the bundle, as used in debugging messages (like the stack traces displayed in errors). The other modules will append a suffix to this filename, based upon their location within the original source tree.

Another common option is `endowments`, which provides names that will be available everywhere in the evaluated sources. By default, the bundle will only get access to the standard JavaScript primordials (`Array`, `Object`, `Map`, etc). It will not get `document`, `window`, `Request`, `process`, `require`, or even `console` unless you provide them as endowments, giving you full control over what the loaded bundle can do.

The `bundle-source` tool has a small number of module names marked as "external". These modules are not bundled into the source (copied from the filesystem where `bundleSource` was called). Instead, the bundler injects a call to `require()` for each external module that was imported from somewhere in original source graph. This let the final evaluation environment control what these imports get, rather than the original source tree.

To support these "external" imports, you will need to provide a `require` endowment that can honor any such names. In addition, the `nestedEvaluate` format always needs a `require` endowment (although it will only be called if the original sources imported one of the "external" names).

For debugging purposes, you should probably provide a `console` endowment. See `makeConsole.js` in the SwingSet source tree for inspiration.

The rest of the `options` are passed through to the `Compartment` constructor, which currently only accepts `transforms`. For more information, see the `compartment-shim` docs in the SES repository. Note that `transforms` is defined to be an array of objects which each have a `rewrite` method.

Note that `sloppyGlobalsMode` is only accepted by the Compartment's `evaluate` method, not the Compartment constructor itself, and thus cannot be supplied to `importBundle`. To use `sloppyGlobalsMode`, you will probably want to create a Compartment directly (and not freeze its globals).
The most common option is `filePrefix`, which can be provided for
`nestedEvaluate`-format bundles.
This sets the source filename of the top-level module inside the bundle, as
used in debugging messages (like the stack traces displayed in errors).
The other modules will append a suffix to this filename, based upon their
location within the original source tree.

Another common option is `endowments`, which provides names that will be
available everywhere in the evaluated sources.
By default, the bundle will only get access to the standard JavaScript
primordials (`Array`, `Object`, `Map`, etc).
It will not get `document`, `window`, `Request`, `process`, `require`, or even
`console` unless you provide them as endowments, giving you full control over
what the loaded bundle can do.

The `bundle-source` tool has a small number of module names marked as
"external".
These modules are not bundled into the source (copied from the filesystem where
`bundleSource` was called).
Instead, the bundler injects a call to `require()` for each external module
that was imported from somewhere in original source graph.
This let the final evaluation environment control what these imports get,
rather than the original source tree.

To support these "external" imports, you will need to provide a `require`
endowment that can honor any such names.
In addition, the `nestedEvaluate` format always needs a `require` endowment
(although it will only be called if the original sources imported one of the
"external" names).

For debugging purposes, you should probably provide a `console` endowment.
See `makeConsole.js` in the SwingSet source tree for inspiration.

The rest of the `options` are passed through to the `Compartment` constructor,
which currently only accepts `transforms`.
For more information, see the `compartment-shim` docs in the SES repository.
Note that `transforms` is defined to be an array of objects which each have a
`rewrite` method.

Note that `sloppyGlobalsMode` is only accepted by the Compartment's `evaluate`
method, not the Compartment constructor itself, and thus cannot be supplied to
`importBundle`.
To use `sloppyGlobalsMode`, you will probably want to create a Compartment
directly (and not freeze its globals).

## Source maps

Expand Down
31 changes: 22 additions & 9 deletions packages/import-bundle/src/index.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,8 @@ export async function importBundle(bundle, options = {}, powers = {}) {
);
}

let compartment;

const { moduleFormat } = bundle;
if (moduleFormat === 'endoZipBase64') {
const { endoZipBase64 } = bundle;
Expand All @@ -76,8 +78,8 @@ export async function importBundle(bundle, options = {}, powers = {}) {
return namespace;
}

let c;
const { source, sourceMap } = bundle;
let { source } = bundle;
const { sourceMap } = bundle;
if (moduleFormat === 'getExport') {
// The 'getExport' format is a string which defines a wrapper function
// named `getExport()`. This function provides a `module` to the
Expand All @@ -87,6 +89,7 @@ export async function importBundle(bundle, options = {}, powers = {}) {
// (making it an expression). We also want to append the `sourceMap`
// comment so `evaluate` can attach useful debug information. Finally, to
// extract the namespace object, we need to invoke this function.
source = `(${source})\n${sourceMap || ''}`;
} else if (moduleFormat === 'nestedEvaluate') {
// The 'nestedEvaluate' format is similar, except the wrapper function
// (now named `getExportWithNestedEvaluate`) wraps more than a single
Expand All @@ -98,17 +101,27 @@ export async function importBundle(bundle, options = {}, powers = {}) {
// `filePrefix`, which will be used as the sourceMap for the top-level
// module. The sourceMap name for other modules will be derived from
// `filePrefix` and the relative import path of each module.
endowments.nestedEvaluate = src => c.evaluate(src);
endowments.nestedEvaluate = src => compartment.evaluate(src);
source = `(${source})\n${sourceMap || ''}`;
} else if (moduleFormat === 'endoScript') {
// The 'endoScript' format is just a script.
} else {
Fail`unrecognized moduleFormat '${moduleFormat}'`;
}

c = new CompartmentToUse(endowments, {}, { transforms });
harden(c.globalThis);
const actualSource = `(${source})\n${sourceMap || ''}`;
const namespace = c.evaluate(actualSource)(filePrefix);
// namespace.default has the default export
return namespace;
compartment = new CompartmentToUse(endowments, {}, { transforms });
harden(compartment.globalThis);
const result = compartment.evaluate(source);
if (moduleFormat === 'endoScript') {
// The completion value of an 'endoScript' is the namespace.
// This format does not curry the filePrefix.
return result;
} else {
// The 'getExport' and 'nestedEvaluate' formats curry a filePrefix.
const namespace = result(filePrefix);
// namespace.default has the default export
return namespace;
}
}

/*
Expand Down
25 changes: 24 additions & 1 deletion packages/import-bundle/test/import-bundle.test.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ test('test import archive', async t => {
const endowments = { console };
const b1EndoZip = await makeArchive(
read,
new URL('bundle1.js', import.meta.url).toString(),
new URL('bundle1.js', import.meta.url).href,
);
const b1EndoZipBase64 = encodeBase64(b1EndoZip);
const b1EndoZipBase64Bundle = {
Expand All @@ -90,6 +90,15 @@ test('test import archive', async t => {
await testBundle1(t, b1EndoZipBase64Bundle, 'endoZipBase64', endowments);
});

test('test import script', async t => {
const endowments = { console };
const b1EndoScriptBundle = await bundleSource(
url.fileURLToPath(new URL('bundle1.js', import.meta.url)),
'endoScript',
);
await testBundle1(t, b1EndoScriptBundle, 'endoScript', endowments);
});

test('test missing sourceMap', async t => {
function req(what) {
console.log(`require(${what})`);
Expand Down Expand Up @@ -152,6 +161,20 @@ test('inescapable global properties, nested evaluate format', async t => {
t.is(ns.default, 42);
});

test('inescapable global properties, script format', async t => {
const bundle = await bundleSource(
url.fileURLToPath(new URL('export-inescapable-global.js', import.meta.url)),
'endoScript',
);

const ns = await importBundle(bundle, {
inescapableGlobalProperties: {
inescapableGlobalValue: 42,
},
});
t.is(ns.default, 42);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not really part of this PR, but should these tests exercise the "inescapability" of inescapableGlobalProperties ? Or did we decide that only the lexical-scope additions needed to be inescapable?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like transforms being inescapable might be important too

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test fixture is a module that runs in a Compartment and creates a child Compartment without options. So, all of these are verifying inescapability at least one level down.

export default new Compartment().evaluate('inescapableGlobalValue');

});

test('inescapable global properties, zip base64 format', async t => {
const bundle = await bundleSource(
url.fileURLToPath(new URL('export-inescapable-global.js', import.meta.url)),
Expand Down
Loading