ruff.toml ignored inside subdir with jupyter-lsp

[ctcjab]

We are reliably reproducing this in jupyterlab but I think it's an issue with python-lsp-ruff, so I'm starting by reporting this here (apologies if I'm mistaken).

Reproduction Steps:

1. uv venv && . .venv/bin/activate
2. uv pip install jupyterlab jupyterlab-lsp python-lsp-ruff ruff [1]
3. Create a pylsp config like the following:

$ cat << EOF >.venv/share/jupyter/lab/settings/overrides.json
{
  "@jupyter-lsp/jupyterlab-lsp:plugin": {
    "language_servers": {
      "pylsp": {
        "priority": 50,
        "serverSettings": {
          "pylsp.plugins.flake8.enabled": false,
          "pylsp.plugins.pycodestyle.enabled": false,
          "pylsp.plugins.pyflakes.enabled": false,
          "pylsp.plugins.pylint.enabled": false,
          "pylsp.plugins.ruff.enabled": true,
          "pylsp.plugins.ruff.formatEnabled": true,
          "pylsp.plugins.ruff.lineLength": 120,
          "pylsp.plugins.yapf.enabled": false
        }
      }
    }
  }
}
EOF

4. jupyter lab --notebook-dir=/tmp
5. Create a top-level notebook with the code import os and confirm that it triggers error F401, as exected
6. Create a top-level ruff.toml with ignore = ["F401"] in the [lint] section
7. Reload the notebook from disk and confirm that error F401 is no longer triggered, as expected
8. Close the notebook
9. Move both the notebook and the ruff.toml into any subdirectory
10. Open the notebook from inside the subdir
11. Observe that the ruff.toml is ignored and error F401 is triggered again
12. The same bug occurs when using pyproject.toml instead of (or in addition to) ruff.toml

Screen capture of running through this repro:
https://github.com/user-attachments/assets/8cb0710a-0ebf-4279-8633-ab9a13a39901

[1] Click to see `pip freeze` output

❯ uv pip freeze
Using Python 3.12.9 environment at: jup-venv
anyio==4.8.0
appnope==0.1.4
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
arrow==1.3.0
asttokens==3.0.0
async-lru==2.0.4
attrs==25.1.0
babel==2.17.0
beautifulsoup4==4.13.3
bleach==6.2.0
cattrs==24.1.2
certifi==2025.1.31
cffi==1.17.1
charset-normalizer==3.4.1
comm==0.2.2
debugpy==1.8.13
decorator==5.2.1
defusedxml==0.7.1
docstring-to-markdown==0.15
executing==2.2.0
fastjsonschema==2.21.1
fqdn==1.5.1
h11==0.14.0
httpcore==1.0.7
httpx==0.28.1
idna==3.10
ipykernel==6.29.5
ipython==9.0.1
ipython-pygments-lexers==1.1.1
isoduration==20.11.0
jedi==0.19.2
jinja2==3.1.6
json5==0.10.0
jsonpointer==3.0.0
jsonschema==4.23.0
jsonschema-specifications==2024.10.1
jupyter-client==8.6.3
jupyter-core==5.7.2
jupyter-events==0.12.0
jupyter-lsp==2.2.5
jupyter-server==2.15.0
jupyter-server-terminals==0.5.3
jupyterlab==4.3.5
jupyterlab-lsp==5.1.0
jupyterlab-pygments==0.3.0
jupyterlab-server==2.27.3
lsprotocol==2023.0.1
markupsafe==3.0.2
matplotlib-inline==0.1.7
mistune==3.1.2
nbclient==0.10.2
nbconvert==7.16.6
nbformat==5.10.4
nest-asyncio==1.6.0
notebook-shim==0.2.4
overrides==7.7.0
packaging==24.2
pandocfilters==1.5.1
parso==0.8.4
pexpect==4.9.0
platformdirs==4.3.6
pluggy==1.5.0
prometheus-client==0.21.1
prompt-toolkit==3.0.50
psutil==7.0.0
ptyprocess==0.7.0
pure-eval==0.2.3
pycparser==2.22
pygments==2.19.1
python-dateutil==2.9.0.post0
python-json-logger==3.3.0
python-lsp-jsonrpc==1.1.2
python-lsp-ruff==2.2.2
python-lsp-server==1.12.2
pyyaml==6.0.2
pyzmq==26.2.1
referencing==0.36.2
requests==2.32.3
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rpds-py==0.23.1
ruff==0.9.10
send2trash==1.8.3
setuptools==75.8.2
six==1.17.0
sniffio==1.3.1
soupsieve==2.6
stack-data==0.6.3
terminado==0.18.1
tinycss2==1.4.0
tornado==6.4.2
traitlets==5.14.3
types-python-dateutil==2.9.0.20241206
typing-extensions==4.12.2
ujson==5.10.0
uri-template==1.3.0
urllib3==2.3.0
wcwidth==0.2.13
webcolors==24.11.1
webencodings==0.5.1
websocket-client==1.8.0

[jhossbach]

Can you increase the verbosity and send the output? See https://github.com/python-lsp/python-lsp-ruff?tab=readme-ov-file#debugging

[jhossbach]

Actually, I misunderstood the problem. python-lsp-ruff does not actually load the configuration from the ruff.toml, it only looks for it to prevent pylsp from accidentally overwriting the project settings with the lsp settings:

python-lsp-ruff/pylsp_ruff/plugin.py

Lines 743 to 745 in 041c695

	ruff_toml = find_parents(
	workspace.root_path, document_path, ["ruff.toml", ".ruff.toml"]
	)

So this is not an issue of pylsp or pylsp-ruff, but rather ruff not finding the ruff.toml. Could it be that the CWD in jupyterlab is set to the parent directory and ruff therefore cannot see the ruff.toml?
I am not sure if this is something that should be changed in the first place, since reading a configuration file in a subdirectory below the CWD would probably break some configs for people?
Also not sure who to blame here, either ruff or jupyterlab-lsp?

[ctcjab]

Could it be that the CWD in jupyterlab is set to the parent directory and ruff therefore cannot see the ruff.toml?

If that were the case, I would not expect this to work, but it does:

❯ cat subdir/foo.py 
import os

❯ cat subdir/ruff.toml 
[lint]
ignore = ["F401"]

❯ ruff check subdir/foo.py 
All checks passed!

❯ mv subdir/ruff.toml{,.disabled} 

❯ ruff check subdir/foo.py 
subdir/foo.py:1:8: F401 [*] `os` imported but unused
  |
1 | import os
  |        ^^ F401
  |
  = help: Remove unused import: `os`

Found 1 error.
[*] 1 fixable with the `--fix` option.

I.e. ruff itself respects ruff.toml files in subdirectories of the CWD when checking adjacent (or descendent) sources.

Can you increase the verbosity and send the output? See https://github.com/python-lsp/python-lsp-ruff?tab=readme-ov-file#debugging

I tried for a long time and still could not figure out how to do this. :(

Thanks for any help looking into this.

[jhossbach]

python-lsp-ruff parses the python file to ruff via stdin and gives the relative path to the file via --stdin-filename for ruff to find the correct configuration file. I did some more digging and found that here the notebook in the virtual documents directory is used, therefore ignoring the ruff.toml in the subdirectory. In other words:

cat <...>| python -m ruff check --stdin-filename=subdir/Untitled.ipynb --extension=ipynb:python

instead of

<cat ...>| python -m ruff check --stdin-filename=.virtual_documents/subdir/Untitled.ipynb --extension=ipynb:python

I think this can be achieved by removing the virtual documents dir from the filename: https://jupyterlab-lsp.readthedocs.io/en/latest/Configuring.html#virtual-documents-dir
I'll look into it.

[jhossbach]

I did some playing around. The issue can be solved trivially if JP_LSP_VIRTUAL_DIR is not set to a custom directory. Otherwise pylsp never knows the real path, only the path to the virtual document. In this case, I am not sure how to handle this from pylsp or pylsp-ruff's side.
I will publish the changes soon. I think a better alternative would be for jupyter-lsp to copy the pyproject.toml or the ruff.toml to the virtual documents directory.
@krassowski would these files be possible to be tracked?

[ctcjab]

More of the users I support have been hitting astral-sh/ruff#17117 lately. It'd be great if a ruff.toml at the root of the repo could be used to work around issues like that. I'm new to these codebases but if this is a ~1-line fix and you could point me toward where to make the changes, I'd be happy to try to submit a PR. Thanks 🙏

[jhossbach]

Assuming you have not set the JP_LSP_VIRTUAL_DIR env variable, you could place a ruff.toml with the required config into .virtual_documents. Soft-linking should also work.
I don't think the problem is fixable from pylsps side AFAIK, since the LSP-server never gets information about the true CWD, only the virtual directory and its contents

One solution would be to copy/link the ruff.toml to the virtual directory, such that when calling ruff with --stdin-filename pointing to the virtual directory, ruff will look inside that directory for configurations.
Or to give pylsp the path to the actual directory. Not sure what is easier to implement.
I will open up an issue on jupyterlab-lsp soon (or you can do it)

[ctcjab]

Hi @jhossbach, thanks for the response and sorry for the delay.

The issue can be solved trivially if JP_LSP_VIRTUAL_DIR is not set to a custom directory.

I can confirm JP_LSP_VIRTUAL_DIR is unset (including when I perform the reproduction steps I listed above), so I don't think this is due to setting JP_LSP_VIRTUAL_DIR to a custom directory, so it's not solvable by ceasing to do so.

Assuming you have not set the JP_LSP_VIRTUAL_DIR env variable, you could place a ruff.toml with the required config into .virtual_documents. Soft-linking should also work.

Unfortunately using .virtual_documents/ruff.toml is not going to work here, for a few reasons. Primarily, we need to preserve the invariant that ruff always uses the ruff.toml maintained in version control alongside the code that ruff is linting. This way, CI's pass/fail result is a pure function of the contents of the repo at the revision under test. (Creating a ruff.toml symlink in .virtual_documents doesn't work either as there are multiple clones that users work with, each of which has a ruff.toml that needs to be honored.)

I will open up an issue on jupyterlab-lsp soon (or you can do it)

Would you mind opening the issue, since I think you understand the problem better? (I searched https://github.com/jupyter-lsp/jupyterlab-lsp/issues but don't see one there yet.) Thank you very much for your help with this.

[ctcjab]

(As an aside, I hit this very nasty bug while testing this and lost a lot of work 😿.)