🤖 Store execution outputs in `Outputs` #1903

agoose77 · 2025-03-07T10:59:19Z

This PR closes #1674 by changing our internal representation of cell outputs so that they are children of the Outputs node.

Note

Part of initiative #1026

Notable Changes

Address individual output children with labels ${block.identifier}-outputs-${N}
Add new parent outputs node with identifier ${block.identifier}-outputs

Tasks

Check that .docx exports still export matplotlib figures
Check that outputs can still be embedded
Check that individual output nodes can be embedded
Check that individual outputs are still minified
Check that JATS exports still export matplotlib figures

changeset-bot · 2025-03-07T10:59:23Z

🦋 Changeset detected

Latest commit: f1a7a44

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 13 packages

Name	Type
mystmd	Minor
myst-directives	Patch
myst-transforms	Patch
myst-spec-ext	Patch
myst-execute	Patch
myst-cli	Minor
myst-common	Patch
myst-config	Patch
myst-frontmatter	Patch
myst-parser	Patch
myst-roles	Patch
myst-to-html	Patch
myst-migrate	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

rowanc1

A few comments as I am looking now.

packages/myst-cli/src/transforms/outputs.ts

packages/myst-execute/tests/execute.yml

packages/myst-execute/tests/run.spec.ts

packages/mystmd/tests/notebook-fig-embed/outputs/index.json

rowanc1 · 2025-03-07T15:17:17Z

Thoughts on a testing plan for this:

thebe can attach to the right place, there are likely going to be changes in the theme due to this
contents are still minified and downloaded correctly (e.g. images are a network request, etc.)
docx, typst, latex, JATS still export matplotlib images and latex
mystmd docs work for the embed and interactive components
bokeh example, holoviews, plotly, sympy
placeholders work

packages/mystmd/tests/notebook-fig-embed/outputs/index.json

packages/myst-transforms/src/blocks.spec.ts

rowanc1 · 2025-03-07T15:31:28Z

packages/myst-transforms/src/blocks.ts

+      const outputsNode = select('outputs', block) as GenericNode | undefined;
+      if (outputsNode !== undefined && !outputsNode.identifier) {
+        // Label outputs node
+        outputsNode.identifier = `${block.identifier}-output`;


I think that these ID changes may need to be taken into account in the upgrade/downgrade script for thebe to continue working.

Let's keep an eye on that

agoose77 · 2025-04-01T10:18:40Z

packages/myst-cli/src/transforms/code.spec.ts

              result = mdast.children[0].children[0].visibility;
              break;
            case 'output':
-              if (!has_output && target == 'output') {


target is known to be output here.

stevejpurves · 2025-06-25T09:46:09Z

packages/myst-directives/src/code.ts

-      type: 'output',
+    const outputs = {
+      type: 'outputs',
      id: nanoid(),


@agoose77 adding this immediately enables compute in figures & embeds in many of the test cases. If there is a place in myst-execute where this should be added I suggest we do. If you could please do that, or if it's trivial, point me to the line to change.

Hmm, not quite following - are you saying that it's broken currently?

We need this change here for in-browser compute.

On this PR jupyter-book/myst-theme#571 you mentioned that myst-execute does not add the id, so I'm just flagging that we need one. Adding the id here seems to be enough, but if there is a further change needed in myst-execute, in line with your comment that should also be changed.

I'm in a conference atm. I need to check the status quo — that comment is about output.id not outputs.id. I think the intention is that the id remainsper-outputs, somyst-executeshould still work (it preserves the fields of theoutputsnode, and theid` is pre-populated).

IIRC myst-theme in that PR now accesses the outputs.id via context, so that the output renderer has access to it.

stevejpurves · 2025-06-26T10:36:16Z

Great - @agoose77 I think this is good to go pending test fixes, and is working against the theme PR jupyter-book/myst-theme#571

but I also note the testing points from @rowanc1 above:

docx, typst, latex, JATS still export matplotlib images and latex
mystmd docs work for the embed and interactive components

The second will work after th content there is updated to the new form as the PR above confirms external emdbedding and referencing works as before.

We don't need to delete nodes without children: they will simply vanish when lifted

agoose77 · 2025-07-30T09:34:38Z

@rowanc1 FYI I've added a warning for calls to migrate without to.

rowanc1 · 2025-07-30T13:26:11Z

I like that! Thanks for the heads up.

agoose77 force-pushed the agoose77/feat-future-ast-outputs branch 3 times, most recently from cec055e to 6870f57 Compare March 7, 2025 11:05

rowanc1 reviewed Mar 7, 2025

View reviewed changes

agoose77 commented Mar 7, 2025

View reviewed changes

packages/mystmd/tests/notebook-fig-embed/outputs/index.json Outdated Show resolved Hide resolved

agoose77 commented Mar 7, 2025

View reviewed changes

packages/mystmd/tests/notebook-fig-embed/outputs/index.json Outdated Show resolved Hide resolved

agoose77 commented Mar 7, 2025

View reviewed changes

packages/mystmd/tests/notebook-fig-embed/outputs/index.json Outdated Show resolved Hide resolved

rowanc1 reviewed Mar 7, 2025

View reviewed changes

packages/myst-transforms/src/blocks.spec.ts Outdated Show resolved Hide resolved

rowanc1 reviewed Mar 7, 2025

View reviewed changes

agoose77 mentioned this pull request Mar 21, 2025

Support new AST format in myst-theme jupyter-book/myst-theme#569

Open

agoose77 mentioned this pull request Apr 1, 2025

Improve capabilities of build-time execution #1026

Open

agoose77 force-pushed the agoose77/feat-future-ast-outputs branch 2 times, most recently from d8a8684 to 6f0d05b Compare April 1, 2025 10:05

agoose77 commented Apr 1, 2025

View reviewed changes

agoose77 force-pushed the agoose77/feat-future-ast-outputs branch from 2060521 to 5857844 Compare April 1, 2025 10:20

agoose77 mentioned this pull request Apr 1, 2025

[myst] Implement support for structured code-cell outputs in MyST AST 2i2c-org/infrastructure#5836

Closed

3 tasks

agoose77 marked this pull request as ready for review April 2, 2025 10:36

agoose77 changed the title ~~🤖 Store execution outputs in Outputs~~ [[⚠️ Do Not Merge ⚠️]] 🤖 Store execution outputs in Outputs Apr 2, 2025

agoose77 force-pushed the agoose77/feat-future-ast-outputs branch from 826b957 to 455e0d8 Compare April 2, 2025 12:49

agoose77 mentioned this pull request Apr 3, 2025

☢️ DO NOT MERGE ☢️ Prepare for outputs AST change jupyter-book/myst-theme#571

Open

16 tasks

agoose77 changed the title ~~[[⚠️ Do Not Merge ⚠️]] 🤖 Store execution outputs in Outputs~~ ⚠️ DO NOT MERGE ⚠️ 🤖 Store execution outputs in Outputs Apr 3, 2025

agoose77 mentioned this pull request Apr 10, 2025

✍️ Parse Markdown and LaTeX outputs into AST #1961

Open

agoose77 changed the title ~~⚠️ DO NOT MERGE ⚠️ 🤖 Store execution outputs in Outputs~~ ⚠️ DO NOT MERGE ⚠️ — 🤖 Store execution outputs in Outputs Apr 10, 2025

agoose77 added the blocked Waiting for another task to be compled label May 23, 2025

stevejpurves reviewed Jun 25, 2025

View reviewed changes

stevejpurves added enhancement New feature or request and removed blocked Waiting for another task to be compled labels Jun 26, 2025

agoose77 and others added 23 commits July 30, 2025 10:22

fix: handle IDs

2af0e14

fix: return type

aaf93af

fix: selectAll → select

9b7d135

refactor: move user expression lowering into function

910977a

refactor: nest declaration

1772491

fix: rename outputs

d602880

refactor: simplify conditions

827d287

We don't need to delete nodes without children: they will simply vanish when lifted

test: fix tests

102d918

fix: outputs for JATS

15e0ead

fix: use outputs- for all prefixes

5edaa47

fix: outputs for JATS

7bd35c3

refactor: simplify test for code cells

8ad105d

fix: early exit non-matplotlib

9dc505a

test: partially fix JATS test

df0a268

fix: final JATS test

19ace17

test: fix myst-execute tests

ad0af31

test: fix test titles

47cf042

test: fix missing children

5664c04

test: more fixes

f8a97cb

test: fix test titles

db80c54

fix: handle visibility properly

88cc3e0

fix: update test cases

ad44fad

🪪 adding id to outputs enables in-browser compute for figures and embeds

421bd10

agoose77 force-pushed the agoose77/feat-future-ast-outputs branch from fccf681 to 421bd10 Compare July 30, 2025 09:22

agoose77 added 2 commits July 30, 2025 10:24

feat: bump version

901041c

feat: add warning for unbounded migration

7be7c0b

agoose77 added 2 commits July 30, 2025 14:58

test: bump version in test

9493999

test: bump version

f1a7a44

Uh oh!

🤖 Store execution outputs in Outputs #1903

Are you sure you want to change the base?

🤖 Store execution outputs in Outputs #1903

Uh oh!

Conversation

agoose77 commented Mar 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Notable Changes

Tasks

Uh oh!

changeset-bot bot commented Mar 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

rowanc1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rowanc1 commented Mar 7, 2025 • edited by stevejpurves Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rowanc1 Mar 7, 2025

Choose a reason for hiding this comment

Uh oh!

agoose77 Apr 1, 2025

Choose a reason for hiding this comment

Uh oh!

agoose77 Apr 1, 2025

Choose a reason for hiding this comment

Uh oh!

stevejpurves Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

agoose77 Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

stevejpurves Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

agoose77 Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

agoose77 Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

stevejpurves commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

agoose77 commented Jul 30, 2025

Uh oh!

rowanc1 commented Jul 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

🤖 Store execution outputs in `Outputs` #1903

🤖 Store execution outputs in `Outputs` #1903

agoose77 commented Mar 7, 2025 •

edited

Loading

changeset-bot bot commented Mar 7, 2025 •

edited

Loading

rowanc1 commented Mar 7, 2025 •

edited by stevejpurves

Loading

stevejpurves Jun 25, 2025 •

edited

Loading

stevejpurves commented Jun 26, 2025 •

edited

Loading