Possible bug in eglot--format-markup?

[garyo]

With most recent eglot and emacs (as of today) I'm getting an error in my python-mode buffers. While eglot is trying to show hover info, it gets a treesit error.

I'm using eldoc and treesit a bunch of other stuff so I'm not sure where the real bug is.
But basically it looks like treesit is trying to fontify a temp python-mode buffer that just contains

(class) openfx

and getting an error. Somehow markdown mode is involved too; see the stack trace.

The markup arg passed to eglot--format-markup is:

(:kind "markdown" :value "```python\n(class) openfx\n```")

I'm using pyright-langserver as the LSP server; it's possible it is returning malformed markup or something that eglot doesn't expect. Here's a bit from the eglot events log in case it's useful:

[jsonrpc] e[10:38:36.232] <-- textDocument/hover[6] 
(:jsonrpc "2.0" :id 6 :result
          (:contents
           (:kind "markdown" :value "```python\n(class) openfx\n```")
           :range
           (:start (:line 7 :character 6) :end (:line 7 :character 12))))

I note that the following patch appears to fix it for me:

modified   eglot.el
@@ -1813,7 +1813,7 @@ Doubles as an indicator of snippet support."
             (message-log-max nil)
             match)
         (ignore-errors (delay-mode-hooks (funcall mode)))
-        (font-lock-ensure)
+        (ignore-errors (font-lock-ensure))
         (goto-char (point-min))
         (let ((inhibit-read-only t))
           (when (fboundp 'text-property-search-forward) ;; FIXME: use compat

The stack trace looks like this:

Debugger entered--Lisp error: (treesit-query-error "Node type error at" 271 "[\"as\" \"assert\" \"async\" \"await\" \"break\" \"case\" \"class\" \"continue\" \"def\" \"del\" \"elif\" \"else\" \"except\" \"exec\" \"finally\" \"for\" \"from\" \"global\" \"if\" \"import\" \"lambda\" \"match\" \"nonlocal\" \"pass\" \"print\" \"raise\" \"return\" \"try\" \"while\" \"with\" \"yield\" \"and\" \"in\" \"is\" \"not\" \"or\" \"not in\"] @font-lock-keyword-face ((identifier) @font-lock-keyword-face (#match \"\\\\`self\\\\'\" @font-lock-keyword-face))" "Debug the query with `treesit-query-validate'")
  treesit--font-lock-fontify-region-1(#<treesit-node module in 1-17> #<treesit-compiled-query> 1 17 nil nil)
  treesit-font-lock-fontify-region(1 17 nil)
  font-lock-fontify-syntactically-region(1 17 nil)
  font-lock-default-fontify-region(1 17 nil)
  font-lock-fontify-region(1 17)
  #f(compiled-function (beg end) #<bytecode -0x147d5368d5be984b>)(1 17)
  font-lock-ensure()
  (save-current-buffer (set-buffer (get-buffer-create (concat " markdown-code-fontification:" (symbol-name lang-mode)))) (let ((inhibit-modification-hooks nil)) (delete-region (point-min) (point-max)) (insert string " ")) (if (eq major-mode lang-mode) nil (funcall lang-mode)) (font-lock-ensure) (setq pos (point-min)) (while (setq next (next-single-property-change pos 'face)) (let ((val (get-text-property pos 'face))) (if val (progn (put-text-property (+ start (1- pos)) (1- (+ start next)) 'face val markdown-buffer)))) (setq pos next)))
  (let ((string (buffer-substring-no-properties start end)) (modified (buffer-modified-p)) (markdown-buffer (current-buffer)) pos next) (remove-text-properties start end '(face nil)) (save-current-buffer (set-buffer (get-buffer-create (concat " markdown-code-fontification:" (symbol-name lang-mode)))) (let ((inhibit-modification-hooks nil)) (delete-region (point-min) (point-max)) (insert string " ")) (if (eq major-mode lang-mode) nil (funcall lang-mode)) (font-lock-ensure) (setq pos (point-min)) (while (setq next (next-single-property-change pos 'face)) (let ((val (get-text-property pos 'face))) (if val (progn (put-text-property (+ start ...) (1- ...) 'face val markdown-buffer)))) (setq pos next))) (add-text-properties start end '(font-lock-fontified t fontified t font-lock-multiline t)) (set-buffer-modified-p modified))
  (progn (let ((string (buffer-substring-no-properties start end)) (modified (buffer-modified-p)) (markdown-buffer (current-buffer)) pos next) (remove-text-properties start end '(face nil)) (save-current-buffer (set-buffer (get-buffer-create (concat " markdown-code-fontification:" (symbol-name lang-mode)))) (let ((inhibit-modification-hooks nil)) (delete-region (point-min) (point-max)) (insert string " ")) (if (eq major-mode lang-mode) nil (funcall lang-mode)) (font-lock-ensure) (setq pos (point-min)) (while (setq next (next-single-property-change pos 'face)) (let ((val (get-text-property pos ...))) (if val (progn (put-text-property ... ... ... val markdown-buffer)))) (setq pos next))) (add-text-properties start end '(font-lock-fontified t fontified t font-lock-multiline t)) (set-buffer-modified-p modified)))
  (if (fboundp lang-mode) (progn (let ((string (buffer-substring-no-properties start end)) (modified (buffer-modified-p)) (markdown-buffer (current-buffer)) pos next) (remove-text-properties start end '(face nil)) (save-current-buffer (set-buffer (get-buffer-create (concat " markdown-code-fontification:" (symbol-name lang-mode)))) (let ((inhibit-modification-hooks nil)) (delete-region (point-min) (point-max)) (insert string " ")) (if (eq major-mode lang-mode) nil (funcall lang-mode)) (font-lock-ensure) (setq pos (point-min)) (while (setq next (next-single-property-change pos 'face)) (let ((val ...)) (if val (progn ...))) (setq pos next))) (add-text-properties start end '(font-lock-fontified t fontified t font-lock-multiline t)) (set-buffer-modified-p modified))))
  (let ((lang-mode (if lang (markdown-get-lang-mode lang) markdown-fontify-code-block-default-mode))) (if (fboundp lang-mode) (progn (let ((string (buffer-substring-no-properties start end)) (modified (buffer-modified-p)) (markdown-buffer (current-buffer)) pos next) (remove-text-properties start end '(face nil)) (save-current-buffer (set-buffer (get-buffer-create (concat " markdown-code-fontification:" ...))) (let ((inhibit-modification-hooks nil)) (delete-region (point-min) (point-max)) (insert string " ")) (if (eq major-mode lang-mode) nil (funcall lang-mode)) (font-lock-ensure) (setq pos (point-min)) (while (setq next (next-single-property-change pos ...)) (let (...) (if val ...)) (setq pos next))) (add-text-properties start end '(font-lock-fontified t fontified t font-lock-multiline t)) (set-buffer-modified-p modified)))))
  markdown-fontify-code-block-natively("python" 11 26)
  (if (and markdown-fontify-code-blocks-natively (or (setq lang (markdown-code-block-lang)) markdown-fontify-code-block-default-mode)) (markdown-fontify-code-block-natively lang start end) (add-text-properties start end '(face markdown-pre-face)))
  (let* ((start (match-beginning 0)) (end (match-end 0)) (bol-prev (progn (goto-char start) (if (bolp) (line-beginning-position 0) (line-beginning-position)))) (eol-next (progn (goto-char end) (if (bolp) (line-beginning-position 2) (line-beginning-position 3)))) lang) (if (and markdown-fontify-code-blocks-natively (or (setq lang (markdown-code-block-lang)) markdown-fontify-code-block-default-mode)) (markdown-fontify-code-block-natively lang start end) (add-text-properties start end '(face markdown-pre-face))) (font-lock-append-text-property bol-prev eol-next 'face 'markdown-code-face) (add-text-properties bol-prev start '(invisible markdown-markup)) (add-text-properties end eol-next '(invisible markdown-markup)))
  (progn (let* ((start (match-beginning 0)) (end (match-end 0)) (bol-prev (progn (goto-char start) (if (bolp) (line-beginning-position 0) (line-beginning-position)))) (eol-next (progn (goto-char end) (if (bolp) (line-beginning-position 2) (line-beginning-position 3)))) lang) (if (and markdown-fontify-code-blocks-natively (or (setq lang (markdown-code-block-lang)) markdown-fontify-code-block-default-mode)) (markdown-fontify-code-block-natively lang start end) (add-text-properties start end '(face markdown-pre-face))) (font-lock-append-text-property bol-prev eol-next 'face 'markdown-code-face) (add-text-properties bol-prev start '(invisible markdown-markup)) (add-text-properties end eol-next '(invisible markdown-markup))))
  (unwind-protect (progn (let* ((start (match-beginning 0)) (end (match-end 0)) (bol-prev (progn (goto-char start) (if (bolp) (line-beginning-position 0) (line-beginning-position)))) (eol-next (progn (goto-char end) (if (bolp) (line-beginning-position 2) (line-beginning-position 3)))) lang) (if (and markdown-fontify-code-blocks-natively (or (setq lang (markdown-code-block-lang)) markdown-fontify-code-block-default-mode)) (markdown-fontify-code-block-natively lang start end) (add-text-properties start end '(face markdown-pre-face))) (font-lock-append-text-property bol-prev eol-next 'face 'markdown-code-face) (add-text-properties bol-prev start '(invisible markdown-markup)) (add-text-properties end eol-next '(invisible markdown-markup)))) (set-match-data saved-match-data t))
  (let ((saved-match-data (match-data))) (unwind-protect (progn (let* ((start (match-beginning 0)) (end (match-end 0)) (bol-prev (progn (goto-char start) (if ... ... ...))) (eol-next (progn (goto-char end) (if ... ... ...))) lang) (if (and markdown-fontify-code-blocks-natively (or (setq lang ...) markdown-fontify-code-block-default-mode)) (markdown-fontify-code-block-natively lang start end) (add-text-properties start end '(face markdown-pre-face))) (font-lock-append-text-property bol-prev eol-next 'face 'markdown-code-face) (add-text-properties bol-prev start '(invisible markdown-markup)) (add-text-properties end eol-next '(invisible markdown-markup)))) (set-match-data saved-match-data t)))
  (save-excursion (let ((saved-match-data (match-data))) (unwind-protect (progn (let* ((start (match-beginning 0)) (end (match-end 0)) (bol-prev (progn ... ...)) (eol-next (progn ... ...)) lang) (if (and markdown-fontify-code-blocks-natively (or ... markdown-fontify-code-block-default-mode)) (markdown-fontify-code-block-natively lang start end) (add-text-properties start end '...)) (font-lock-append-text-property bol-prev eol-next 'face 'markdown-code-face) (add-text-properties bol-prev start '(invisible markdown-markup)) (add-text-properties end eol-next '(invisible markdown-markup)))) (set-match-data saved-match-data t))))
  (progn (save-excursion (let ((saved-match-data (match-data))) (unwind-protect (progn (let* ((start ...) (end ...) (bol-prev ...) (eol-next ...) lang) (if (and markdown-fontify-code-blocks-natively ...) (markdown-fontify-code-block-natively lang start end) (add-text-properties start end ...)) (font-lock-append-text-property bol-prev eol-next 'face 'markdown-code-face) (add-text-properties bol-prev start '...) (add-text-properties end eol-next '...))) (set-match-data saved-match-data t)))) t)
  (if (funcall matcher last) (progn (save-excursion (let ((saved-match-data (match-data))) (unwind-protect (progn (let* (... ... ... ... lang) (if ... ... ...) (font-lock-append-text-property bol-prev eol-next ... ...) (add-text-properties bol-prev start ...) (add-text-properties end eol-next ...))) (set-match-data saved-match-data t)))) t))
  markdown-fontify-code-blocks-generic(markdown-match-gfm-code-blocks 29)
  markdown-fontify-gfm-code-blocks(29)
  font-lock-fontify-keywords-region(1 29 nil)
  font-lock-default-fontify-region(1 29 nil)
  font-lock-fontify-region(1 29)
  #f(compiled-function (beg end) #<bytecode -0x147d5368d5be984b>)(1 29)
  font-lock-ensure()
  (let ((inhibit-message t) (message-log-max nil) match) (condition-case nil (progn (progn (make-local-variable 'delay-mode-hooks) (let ((delay-mode-hooks t)) (funcall mode)))) (error nil)) (font-lock-ensure) (goto-char (point-min)) (let ((inhibit-read-only t)) (if (fboundp 'text-property-search-forward) (progn (while (setq match (text-property-search-forward 'invisible)) (delete-region (let* (...) (progn ... ...)) (let* (...) (progn ... ...))))))) (string-trim (buffer-string)))
  (progn (set (make-local-variable 'markdown-fontify-code-blocks-natively) t) (insert string) (let ((inhibit-message t) (message-log-max nil) match) (condition-case nil (progn (progn (make-local-variable 'delay-mode-hooks) (let ((delay-mode-hooks t)) (funcall mode)))) (error nil)) (font-lock-ensure) (goto-char (point-min)) (let ((inhibit-read-only t)) (if (fboundp 'text-property-search-forward) (progn (while (setq match (text-property-search-forward ...)) (delete-region (let* ... ...) (let* ... ...)))))) (string-trim (buffer-string))))
  (unwind-protect (progn (set (make-local-variable 'markdown-fontify-code-blocks-natively) t) (insert string) (let ((inhibit-message t) (message-log-max nil) match) (condition-case nil (progn (progn (make-local-variable 'delay-mode-hooks) (let (...) (funcall mode)))) (error nil)) (font-lock-ensure) (goto-char (point-min)) (let ((inhibit-read-only t)) (if (fboundp 'text-property-search-forward) (progn (while (setq match ...) (delete-region ... ...))))) (string-trim (buffer-string)))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))
  (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (set (make-local-variable 'markdown-fontify-code-blocks-natively) t) (insert string) (let ((inhibit-message t) (message-log-max nil) match) (condition-case nil (progn (progn (make-local-variable ...) (let ... ...))) (error nil)) (font-lock-ensure) (goto-char (point-min)) (let ((inhibit-read-only t)) (if (fboundp 'text-property-search-forward) (progn (while ... ...)))) (string-trim (buffer-string)))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer))))
  (let ((temp-buffer (generate-new-buffer " *temp*" t))) (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (set (make-local-variable 'markdown-fontify-code-blocks-natively) t) (insert string) (let ((inhibit-message t) (message-log-max nil) match) (condition-case nil (progn (progn ... ...)) (error nil)) (font-lock-ensure) (goto-char (point-min)) (let ((inhibit-read-only t)) (if (fboundp ...) (progn ...))) (string-trim (buffer-string)))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))))
  (let ((string x614) (mode x616)) (let ((temp-buffer (generate-new-buffer " *temp*" t))) (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (set (make-local-variable 'markdown-fontify-code-blocks-natively) t) (insert string) (let ((inhibit-message t) (message-log-max nil) match) (condition-case nil (progn ...) (error nil)) (font-lock-ensure) (goto-char (point-min)) (let (...) (if ... ...)) (string-trim (buffer-string)))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer))))))
  (progn (ignore (null x617)) (let ((string x614) (mode x616)) (let ((temp-buffer (generate-new-buffer " *temp*" t))) (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (set (make-local-variable ...) t) (insert string) (let (... ... match) (condition-case nil ... ...) (font-lock-ensure) (goto-char ...) (let ... ...) (string-trim ...))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))))))
  (let* ((x616 (car-safe x615)) (x617 (cdr-safe x615))) (progn (ignore (null x617)) (let ((string x614) (mode x616)) (let ((temp-buffer (generate-new-buffer " *temp*" t))) (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (set ... t) (insert string) (let ... ... ... ... ... ...)) (and (buffer-name temp-buffer) (kill-buffer temp-buffer))))))))
  (progn (ignore (consp x615)) (let* ((x616 (car-safe x615)) (x617 (cdr-safe x615))) (progn (ignore (null x617)) (let ((string x614) (mode x616)) (let ((temp-buffer (generate-new-buffer " *temp*" t))) (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn ... ... ...) (and ... ...))))))))
  (let* ((x614 (car-safe val)) (x615 (cdr-safe val))) (progn (ignore (consp x615)) (let* ((x616 (car-safe x615)) (x617 (cdr-safe x615))) (progn (ignore (null x617)) (let ((string x614) (mode x616)) (let ((temp-buffer ...)) (save-current-buffer (set-buffer temp-buffer) (unwind-protect ... ...))))))))
  (progn (ignore (consp val)) (let* ((x614 (car-safe val)) (x615 (cdr-safe val))) (progn (ignore (consp x615)) (let* ((x616 (car-safe x615)) (x617 (cdr-safe x615))) (progn (ignore (null x617)) (let ((string x614) (mode x616)) (let (...) (save-current-buffer ... ...))))))))
  (let* ((val (if (stringp markup) (list markup 'gfm-view-mode) (list (plist-get markup :value) (let* ((val ...)) (cond (... ...) (... ...) (t ...))))))) (progn (ignore (consp val)) (let* ((x614 (car-safe val)) (x615 (cdr-safe val))) (progn (ignore (consp x615)) (let* ((x616 (car-safe x615)) (x617 (cdr-safe x615))) (progn (ignore (null x617)) (let (... ...) (let ... ...))))))))
  eglot--format-markup((:kind "markdown" :value "```python\n(class) openfx\n```"))
  mapconcat(eglot--format-markup ((:kind "markdown" :value "```python\n(class) openfx\n```")) "\n")
  eglot--hover-info((:kind "markdown" :value "```python\n(class) openfx\n```") (:start (:line 7 :character 6) :end (:line 7 :character 12)))

[joaotavora]

More of a markdown-mode problem right? Anyway, what happens is that Eglot delegates this fontification to markdown-mode who then eventually delegates it to Emacs major modes. In in Emacs a lot of new "tree-sitter" modes with slightly new semantics and breakage are appearing, so you get the error in Eglot. Put a condition-case-unless-debug there instead of the ignore-errors

[garyo]

Thanks. Revised patch:

modified   eglot.el
@@ -1813,7 +1813,9 @@ Doubles as an indicator of snippet support."
             (message-log-max nil)
             match)
         (ignore-errors (delay-mode-hooks (funcall mode)))
-        (font-lock-ensure)
+        (condition-case-unless-debug oops
+            (font-lock-ensure)
+          (error (eglot--warn (error-message-string oops))))
         (goto-char (point-min))
         (let ((inhibit-read-only t))
           (when (fboundp 'text-property-search-forward) ;; FIXME: use compat

[joaotavora]

Looks good. You should open a M-x report-emacs-bug and CC me and, if you can, to involve the markdown-mode and python-ts-mode authors in this discussion. Attach the patch there, thanks in advance.

Another problem that I saw is that even when I use c++-ts-mode, cc-mode will still get loaded on markdown-mode's behalf. Maybe markdown-mode should try to use the same default mode.