Results disappear from the drawer when using async and org >= 9.7

[cvb]

Hi, it seems that some breaking changes have been made in the new versions of th org mode. They break async execution of the src blocks. Potentialy this also breaks some data after the RESULTS: when result disappears, but I wasn't able to reproduce that, so maybe it was my fault.

Here is the minimal example

#+PROPERTY: header-args:jupyter-python :kernel allms :session test

#+begin_src jupyter-python :async yes :results scalar
import logging
import time
logging.basicConfig(level=logging.INFO)
log = logging.getLogger(__name__)
for i in range(10):
    log.info("here")
    time.sleep(0.1)

{1: 2}
#+end_src

#+RESULTS:

When executing the block, I observe that the results section is initially populated with log records. However, upon completion of the execution, the results section is cleared, leaving it empty. Notably, this issue does not occur when reducing the number of iterations in the loop or when removing the final result ({1: 2}) from the code block.

In the Messages I see this

Error in ZMQ subprocess filter: (jupyter-publisher-subscribers-had-errors (jupyter-publisher-subscribers-had-errors (args-out-of-range #<buffer /Users/cvb/tmp/test.org> 331 521)))

I tried to track the source of the error and it seems to be this call org-element-interpret-data. Which is called from here. I put some extra message calls and I was able to see that jupyter-org--append-stream-result appending data just fine, and then in the end jupyter-org--insert-result fails.

It looks like jupyter-org--wrap-result-maybe generates something wrong, but I couldn't understand it, that is just too deep in the org for me.

This problem disappears if I downgrade org to 9.6.30.

I tested this with the minimal emacs config and master branch of the jupyter.

Here is the description of my minimal setup to reproduce.
minimal.el:

(require 'package)
(add-to-list 'package-archives '("melpa" . "https://melpa.org/packages/"))
(add-to-list 'package-archives '("org-elpa" . "http://orgmode.org/elpa/"))
(add-to-list 'package-archives '("gnu" . "https://elpa.gnu.org/packages/"))
(add-to-list 'package-archives '("nongnu" . "https://elpa.nongnu.org/nongnu/"))

(package-initialize)

(package-refresh-contents)

(package-install 'websocket)
(package-install 'simple-httpd)
(package-install 'zmq)
(package-install 'org)


(add-to-list 'load-path "~/tmp/jupyter")
(require 'jupyter)

(org-babel-do-load-languages
   'org-babel-load-languages
   '((emacs-lisp . t)
     (julia . t)
     (python . t)
     (jupyter . t)))

Launched like emacs -Q -l minimal.el --init-dir tmpemacs

[nnicandro]

It looks like I've found the issue. It's because in Org >= 9.7 there is the concept of computing the deferred value of an element. In Emacs-Jupyter once the {1: 2} result is received it deletes the example-block element from the buffer (which holds the logging output) and then attempts to re-insert that element along with the {1: 2} result but wrapped in a drawer this time.

Since the example-block was deleted, org-element-interpret-data can't compute the deferred value since it has been deleted from the buffer by the time it is called.

Not sure why it works when the element that contains the logging lines is fixed-width since that is also deleted in the same way as the example-block case when the {1:2} result comes in. The reason why you get an example-block at 10 lines is due to org-babel-min-lines-for-block-output by the way.

A simple solution would be to force the computation of the deferred value, e.g. by calling (org-element-property :value ...) before deleting the element from the buffer, I'll have to think more about this. Thanks for opening up the issue.

[cvb]

You are right, I made a super simple change resolving deferred, and now it works as expected cvb@b770e2a . I'll keep using my branch, maybe I'll see some other issues with this solutiuon.

The reason why you get an example-block at 10 lines is due to org-babel-min-lines-for-block-output by the way.

Thanks for the tip.