Sphinx-Needs does not work with sphinx-autodoc-typehints

[marten-de-vries]

While running Sphinx-Needs, I ran into the following error:

<file>:docstring of <mod_a>.<mod_b>.<classname>.<methodname>:1: WARNING: Need could not be created: A need with ID '<ID>' already exists. [needs.create_need]

Sphinx-Needs is correct: the need is parsed twice during the build process:

1. By sphinx-autodoc-typehints. This extension will convert Python type signatures and 'insert' them into the docstring in the format Sphinx expects. To do so, it needs to figure out where the newly parsed parameters need to go. And to do that, it parses the whole doc string using a freshly instantiated RSTParser.
2. The normal way, when sphinx.ext.autodoc parses the doc string.

One workaround I've found is to use the fact that sphinx-autodoc-typehints doesn't specify the file name in its temporary parser. Simply inserting a check for that by replacing directives/need.py line 58 with the following works:

    def run(self) -> Sequence[nodes.Node]:
        file = self.get_source_info()[0]
        if not file:
            return []

I'm unsure whether that is the best way to fix the issue, though.

Thank you for Sphinx-Needs!

[danwos]

It's a tough one, as I'm against including workarounds for other extensions in our code.
There may be other use cases, where the file is not given, but Sphinx-Needs needs to create the needed objects.

Is there no way to solve this problem in sphinx-autodoc-typehints?

[marten-de-vries]

I quite understand. It's annoying that this bug is in the interaction between the two extensions, without one of them being clearly 'at fault'.

Is there no way to solve this problem in sphinx-autodoc-typehints?

I don't know to be honest. I could propose to them to not cause doc strings to be parsed twice, but I suspect that would mean having to partly duplicate RSTParser's functionality in sphinx-autodoc-typehints (probably poorly). That seems like a bug-prone approach, and quite heavy on maintenance.

How does sphinx-needs hook into RSTParser? Having sphinx-autodoc-typehints temporarily disable that might be more promising. Although if it requires a mechanism that seems too specific to sphinx-needs I might get hit by a similar 'no workarounds for other extensions' policy there.

[chrisjsewell]

Heya

How does sphinx-needs hook into RSTParser?

Well, it doesn't; it just uses the standard docutils/sphinx infrastructure which already parses the documentation

it parses the whole doc string using a freshly instantiated RSTParser

and this is why this is so problematic; it is basically overriding what sphinx does and will not integrate well with any extension

[marten-de-vries]

and this is why this is so problematic; it is basically overriding what sphinx does and will not integrate well with any extension

• I've opened an issue in the sphinx-autodoc-typehints repo: _parser.parse() causes issues if directives with side-effects are active tox-dev/sphinx-autodoc-typehints#510

Looking into it more, though, I'm not sure if they have good other options. There is no way to construct a docutils-based restructuredtext parser at all without your directives being triggered, as those are stored as global docutils state (app.add_directive() calls docutils.register_directive()).

Also, for most extensions this isn't a problem as directives that are not idempotent are rare.