Do not resolve during annotate, enrich documentation with details (Fix #3201) (Fix #3905)

[wyuenho]

Problem

When using typescript-language-server, the initial call to textDocument/completion does not return any detail or documentation for any of the completion items. I suppose the reason for this is many Javascript signatures are extremely long, often they are 5x to 10x longer than the label, they are unreadable when displaying beside the label on one line, so the server forces the client to make completionItem/resolve requests to resolve the item detail and documentation individually, and it's up to the client to prepend the signature to the documentation, as is done in VS Code.

VS Code Typescript

This approach presents a problem to lsp-mode in that the CAPF function caches the partial completion item response as a text property on each candidate string, and when a completion frontend such as company or corfu calls lsp-completion--annotate to get a suffix, every call will issue an async completionItem/resolve request to modify the cached completion item in place while returning just a kind or an empty string initially, depending on some variables. This means the first completion popup will only have the kinds or simply no suffix at all, and then on the next refresh after a selection change, in the case of company, all of the candidates in the pop up will suddenly be annotated, and in the case of corfu, the previous selection will suddenly be annotated. In both cases the popup width will suddenly expand greatly, often times as wide as the window size. This is fundamentally because lsp-mode assumes the partial completion item response from textDocument/completion is meant to be used the same way as the fully resolved completion item response from completionItem/resolve.

This PR reimplements lsp-completion--make-item, lsp-completion--annotate and lsp-completion--get-documentation to separate the two different usages. In addition, the signature from detail is now prepended to the document if it has not been prepended by the language server already.

LSP ts-ls

LSP pyright

LSP gopls

LSP rust-analyzer

LSP jdtls

[kiennq]

This is fundamentally because lsp-mode assumes the partial completion item response from textDocument/completion is meant to be used the same way as the fully resolved completion item response from completionItem/resolve.

This PR reimplements lsp-completion--make-item, lsp-completion--annotate and lsp-completion--get-documentation to separate the two different usages.

The implementation in this PR relies on the auto-documentation being automatically triggered. I would like to avoid that and always have the candidate resolved as it's displayed. The annotation update is called for displayed candidates and is a good function to trigger resolving asynchronously.

Also, we can configure the client's capability to not have partial completion item responses at all. The reason why we make it have partial completion item responses is to make the completion list return as quickly as possible without unnecessary and/or large item property strings. The other properties can be retrieved later with completionItem/resolve and should be treated as updated completion items.

Please see #4591 for the issue with completion items not being resolved without the document's update as well.

To avoid the width suddenly changing, I think the user can disable lsp-completion-show-detail. Alternatively, we can trigger a candidate list rendering refresh when the completionItem/resolve is done. The second approach will make lsp-mode behave like VS Code. I would prefer that if it's easy to do.

In addition, the signature from detail is now prepended to the document if it has not been prepended by the language server already.

This is a good feature. I agree we should do this.

[wyuenho]

I think we are beginning to see this one-size-fits-all lsp-completion-at-point implementation fail.

The implementation in this PR relies on the auto-documentation being automatically triggered. I would like to avoid that and always have the candidate resolved as it's displayed.

The documentation is already resolved synchronously in HEAD, I didn't change this, I only changed detail resolution. Detail should be resolved when it's needed, which is largely determined by the server. Even when detail is added to resolveSupport, I have not seen a server honor it by skipping this property in the response of textDocument/completion from what I can see.

I would like to avoid that and always have the candidate resolved as it's displayed. The annotation update is called for displayed candidates and is a good function to trigger resolving asynchronously.

The problem is exactly because the first time the candidates are displayed, they may not be resolved, there is also no guarantee they will be resolved the next time they are displayed, or the third time, they will be resolved whenever the server feels like it's time to send back a response because, asynchronicity, so what ends up happening is the annotation appearing in the completion popup erratically.

Also, we can configure the client's capability to not have partial completion item responses at all. The reason why we make it have partial completion item responses is to make the completion list return as quickly as possible without unnecessary and/or large item property strings.

Nobody is arguing with that, for languages where it makes sense, like TypeScript or Python, the language servers often do not send down detail and documentation in textDocument/completion anyway. The reason why these 2 properties have always supported lazy resolution is because they can be slow to generate or are extremely long for some languages. In general, you don't need to specify detail and documentation in resolveSupport, the servers will decide to send them in textDocument/completion when it makes sense, so most of the time, they have no effect anyway. The only exception is JDTLS, where you have to specify documentation to get the docs, but otherwise I have not seen a server skipping detail in textDocument/completion just because you've specified detail in resolveSupport.

The other properties can be retrieved later with completionItem/resolve and should be treated as updated completion items.

Fine, but they should not affect how the completion candidate list is displayed, but only how text are inserted or replaced, and displaying documentation. Resolving for insertion, replacement, indentation etc are already done in the exit function. If you want to speed up insertion in case resolution in the exit function is slow, you can call lsp-completion--resolve-async in lsp-completion-at-point for each item. This has the possibly of spamming the server, and I suppose JDTLS might not like that, so the alternative could be supporting itemDefaults. Regardless, this is a separate issue that requires experimentation in a separate PR. My only concern is the annotation function should not use the resolved item. Detail retrieved from completionItem/resolve should only be prepended to the documentation. This should lay the ground work for further optimization, e.g. itemDefaults.

To avoid the width suddenly changing, I think the user can disable lsp-completion-show-detail.

This is crazy. Are you suggesting that every user should adjust this defcustom buffer-local in mode hooks as opposed to simply shipping with a default behavior that makes sense for the vast majority if not all cases?

Alternatively, we can trigger a candidate list rendering refresh when the completionItem/resolve is done. The second approach will make lsp-mode behave like VS Code. I would prefer that if it's easy to do.

CAPF is pull-based. How do you "trigger a refresh" of all the completion frontends now and in the future? Also, what does it have to do with VS Code?

[kiennq]

In general, you don't need to specify detail and documentation in resolveSupport, the servers will decide to send them in textDocument/completion when it makes sense, so most of the time, they have no effect anyway. The only exception is JDTLS, where you have to specify documentation to get the docs, but otherwise I have not seen a server skipping detail in textDocument/completion just because you've specified detail in resolveSupport.

The rust-analyzer (nightly) is supporting that and will skip detail and document if it's specified in resolveSupport. That's also the issue in #4591.

The spec from LSP said that

By default, the request can only delay the computation of the detail and documentation properties. Since 3.16.0, the client can signal that it can resolve more properties lazily. This is done using the completionItem#resolveSupport client capability which lists all properties that can be filled in during a ‘completionItem/resolve’ request. All other properties (usually sortText, filterText, insertText and textEdit) must be provided in the textDocument/completion response and must not be changed during resolve.

So, from 3.16, instead of the default detail and document, more properties can be lazily resolved, and it's entirely depended on the language server to support that. I will not be surprised if there's new language server that takes advantage of that and implement lazy-resolving as much as possible.

I think we are beginning to see this one-size-fits-all lsp-completion-at-point implementation fail.
This is crazy. Are you suggesting that every user should adjust this defcustom buffer-local in mode hooks as opposed to simply shipping with a default behavior that makes sense for the vast majority if not all cases?

I think as long as we provide enough customization for the user, it would be okay as there's no one-size-fits-all solution. The default should be as close to the VsCode behavior as possible. So, if the VsCode doesn't do the candidate annotation (which Emacs does) then we should configure lsp-completion-show-detail as nil by default instead. And this defcustom is not buffer-local btw.
Although I would argue that since showing the detail right beside the candidate has been a default configuration for a long time, suddenly change it to nil might cause confusion.

The problem is exactly because the first time the candidates are displayed, they may not be resolved, there is also no guarantee they will be resolved the next time they are displayed, or the third time, they will be resolved whenever the server feels like it's time to send back a response because, asynchronicity, so what ends up happening is the annotation appearing in the completion popup erratically.

I'm not sure but the behavior of showing document pop can be argued as erratically as well, as it suddenly appears, blocking since the user will experience hang if the server is slow to return the result, unlike lsp-completion--resolve-async. The blocking can be justified if the user triggers that intentionally, but it would be hammering if it's triggered automatically, for example due to company-posframe-quickhelp-delay or company-auto-update-doc.
If we think of the annotation is something that will be filled asynchronously, then it's suddenly filled at a later time while the user is browsing the candidate list will not be surprised at all. Perhaps show more visual indicators for that (a loading gif?? or place holder for annotation string) would help?

CAPF is pull-based. How do you "trigger a refresh" of all the completion frontends now and in the future? Also, what does it have to do with VS Code?

This would be capf but rather company-mode or corfu. If they have method to support refresh their candidate lists, we can use that. The lsp-mode will try to default to what VsCode is configured by default, so if they do lazy-resolving for candidate annotation then we should follow that (I haven't checked this btw).

I think that your main argument is that we shouldn't treat the resolved completion item and the original completion item as a same entity and always use the original completion even if it's lacking information. My counterargument is that they're the same and we should use the latest information if possible. The reason is that it provides more information to the user.
I would invite other maintainers (@yyoncho @ericdallo @jcs090218 ...) to chime in and provide their opinions on this as well.

Btw, here is an example behavior with a place-holder on annotation string.

The code change

(defun lsp-completion--annotate (item)
  "Annotate ITEM detail."
  (-let* (((&plist 'lsp-completion-item completion-item
                   'lsp-completion-resolved resolved)
           (text-properties-at 0 item))
         ((&CompletionItem :detail? :kind? :label-details?) completion-item))
    (lsp-completion--resolve-async item #'ignore)

    (concat (when lsp-completion-show-detail
              (if resolved
                  (when detail? (concat " " (s-replace-regexp "\r" "" detail?)))
                " <loading...>"))
            (when (and lsp-completion-show-label-description label-details?)
              (when-let* ((description (and label-details? (lsp:label-details-description label-details?))))
                (format " %s" description)))
            (when lsp-completion-show-kind
              (when-let* ((kind-name (and kind? (aref lsp-completion--item-kind kind?))))
                (format " (%s)" kind-name))))))

[wyuenho]

The rust-analyzer (nightly) is supporting that and will skip detail and document if it's specified in resolveSupport. That's also the issue in #4591.

Ok, this PR works just as well. When the initial partial completion item has no detail, after a resolution, the detail will be prepended to the document.

So, from 3.16, instead of the default detail and document, more properties can be lazily resolved, and it's entirely depended on the language server to support that. I will not be surprised if there's new language server that takes advantage of that and implement lazy-resolving as much as possible.

So load them when you need them, I don't know why we keep circling back to this. This PR has nothing to do with these other lazily resolved properties. It's already done in the exit function.

The default should be as close to the VsCode behavior as possible. So, if the VsCode doesn't do the candidate annotation (which Emacs does) then we should configure lsp-completion-show-detail as nil by default instead. And this defcustom is not buffer-local btw.

I believe the central issue here is, ts-ls doesn't always return detail in the response of textDocument/completion, and when it does, the detail is not a type signature, so the user shouldn't even attempt to set lsp-completion-show-detail to nil, either buffer-locally or worse, globally. When it doesn't return any detail, the popup width jumps erratically after resolving the first item. When it does return some detail, there's no easy and efficient way to tell what's in it. The only acceptable default is leave lsp-completion-show-detail to t globally, and deal with showing the type in the document after resolution.

I'm not sure but the behavior of showing document pop can be argued as erratically as well, as it suddenly appears, blocking since the user will experience hang if the server is slow to return the result, unlike lsp-completion--resolve-async.

I don't understand this sentence. Can you rephrase? The response to textDocument/completion is designed to be fast and is often tuned to be fast by default by the language servers, hence the lazy properties. It's perfectly fine to block here.

The blocking can be justified if the user triggers that intentionally, but it would be hammering if it's triggered automatically, for example due to company-posframe-quickhelp-delay or company-auto-update-doc.

Yes, that's why lsp-completion--get-documentation sends a blocking request to completionItem/resolve. There's nothing wrong with it, what wrong is you are attempting to resolve the entire item in the annotation function. It doesn't matter whether you do it synchronously or asynchronously, it just shouldn't happen as you are interfering how the server intends how the completion list is displayed.

If spamming the server is a problem, these completion frontend should implement debounce with run-with-idle-timer. This should not be a responsibility for lsp-mode. I've tried this PR on the server that's really sensitive to spamming - JDTLS, it's perfectly fine, as the total number of requests is actually reduced by not resolving until absolutely necessary. What's in HEAD however, is much worse as for company, the annotation function is called when constructing the candidate list for all candidates on the first page after the first selection change. The PR actually improves on this situation by really resolving the detail when needed.

If we think of the annotation is something that will be filled asynchronously, then it's suddenly filled at a later time while the user is browsing the candidate list will not be surprised at all.

Yes, which is already handled when corfu-popupinfo/corfu-info/company-quickhelp etc calls lsp-completion--get-documentation. The VS Code behavior is to prepend the detail to the documentation after resolving, and leave the completion popup alone. (Edit: when "Show More" is active)

This would be capf but rather company-mode or corfu. If they have method to support refresh their candidate lists, we can use that.

I beg you please don't even try this. I'm working on corfu-pixel-perfect, it does have the ability to refresh, but it's a little complicated for vanilla corfu. I think company box has this ability as well, but not sure about company. Basically, don't do this as this is highly dependent on third-party packages. You don't need it, and the outcome is undesirable as the width will either erratically expand or the candidate line is squished and truncated in all sorts of ways.

The lsp-mode will try to default to what VsCode is configured by default, so if they do lazy-resolving for candidate annotation then we should follow that

VS Code doesn't change the popup list after selecting a different candidate and showing the documentation popup...

Btw, here is an example behavior with a place-holder on annotation string.

I don't think this is VS Code's behavior...

[wyuenho]

Ok here's more information. It turns out, VS Code remembers the last value of ^SPC (Show more or less), and the way to change it is hidden in a hint in the status bar which is off by default.

When "Show More" is active, the detail is prepended to the documentation. When "Show Less" is active, the detail is rendered on the popup menu on selection if it is not in the response from textDocument/completion. When the user selects a different completion item, the detail on the last selection will be removed. This means that VS Code does make requests to completionItem/resolve on selection, but the response from textDocument/completion and completionItem/resolve are still used differently.

In addition, if a completion item has no detail from textDocument/completion, but has detail but no documentation from completionItem/resolve, the user cannot change from "Show Less" to "Show More" when selecting that item. The user must select another item that has documentation before he can change back to "Show More".

In order to accomplish this in lsp-mode, we will need to cooperate with completion frontends, I guess this is where your idea of "refresh" comes in. What we can do is, keep the separation of unresolved and resolved completion item as done in this PR , do not resolve async when the annotation function is called, but instead, the completion frontends should call lsp-completion--resolve[-async] so they can surgically refresh the completion item on selection change. Does this make sense? This is a UI problem that lsp-completion-mode should not try to solve all by itself.

[wyuenho]

This is how reverse engineering VS Code's behavior results in in corfu-pixel-perfect when combined with this PR.

[wyuenho]

More reasons to separate the unresolved and resolved completion item: the details for the same label can be different in the responses in textDocument/completion and completionItem/resolve.

textDocument/completion

{
      "data": {
        "cacheId": 964
      },
      "detail": "@nestjs/common/utils/shared.utils",
      "kind": 6,
      "label": "isObject",
      "sortText": "�16",
      "textEdit": {
        "insert": {
          "end": {
            "character": 3,
            "line": 2
          },
          "start": {
            "character": 0,
            "line": 2
          }
        },
        "newText": "isObject",
        "replace": {
          "end": {
            "character": 3,
            "line": 2
          },
          "start": {
            "character": 0,
            "line": 2
          }
        }
      }
    }

completionItem/resolve

[Trace - 09:50:15 PM] Received response 'completionItem/resolve - (20)' in 534ms.
Result: {
  "additionalTextEdits": [
    {
      "newText": "import { isObject } from '@nestjs/common/utils/shared.utils';\n\n",
      "range": {
        "end": {
          "character": 0,
          "line": 0
        },
        "start": {
          "character": 0,
          "line": 0
        }
      }
    }
  ],
  "data": {
    "entryNames": [
      {
        "data": {
          "exportMapKey": "8 4590 isObject ",
          "exportName": "isObject",
          "fileName": "/Users/wyuenho/nest/packages/common/utils/shared.utils.ts",
          "moduleSpecifier": "@nestjs/common/utils/shared.utils"
        },
        "name": "isObject",
        "source": "@nestjs/common/utils/shared.utils"
      }
    ],
    "file": "/Users/wyuenho/nest/packages/core/repl/index.ts",
    "line": 3,
    "offset": 4
  },
  "detail": "Auto import from '@nestjs/common/utils/shared.utils'\nconst isObject: (fn: any) => fn is object",
  "kind": 6,
  "label": "isObject",
  "sortText": "�16",
  "textEdit": {
    "insert": {
      "end": {
        "character": 3,
        "line": 2
      },
      "start": {
        "character": 0,
        "line": 2
      }
    },
    "newText": "isObject",
    "replace": {
      "end": {
        "character": 3,
        "line": 2
      },
      "start": {
        "character": 0,
        "line": 2
      }
    }
  }
}

[wyuenho]

I've just changed back to always resolve when getting the documentation as the resolved detail may be different from unresolved detail, even when both are non-empty. This should be the last bit required to reverse engineer VS Code's auto-completion UI.