[WIP] Further improvements to textmate based parser

[apozharski]

Better classdef attributes and validator function parsing based on improvements to MATLAB-language-grammar in:

• Allow for function calls within brace validation lists  mathworks/MATLAB-Language-grammar#86
• Fix broken attribute parsing in classdef. mathworks/MATLAB-Language-grammar#88
• Correct metaclass parsing mathworks/MATLAB-Language-grammar#90

and a textmate-grammar-python pr which bumps the MATLAB-language-grammar (which will be created once Mathworks pulls in the improvements).

Further improvements:

• parsing docstrings for functions

[joeced]

The tests, (see https://github.com/sphinx-contrib/matlabdomain/actions/runs/10058995696/job/27803269388?pr=258#step:4:106) are failing as setup.py does not require the textmate grammar package.

[apozharski]

Yep I have just been running pytest locally to get through the true errors. Unfortunately it looks like there are fundamental problems with the textmate grammar that I am not sure are solvable due to the limitations of the textmate grammar. In particular the syntax for multiline defaults is I believe unparseable. @watermarkhu I am unsure if you have any insights that I do not here.

An example of the problem:

prop = [1;
              2;
              3]; % a prop

in which one cannot unambiguously identify which of the ; actually ends the property default.

[joeced]

It's ok for the test to fail for now. We'll get there eventually. For the default value of properties the current code starts here:

matlabdomain/sphinxcontrib/mat_types.py

Line 1275 in b081e7c

if self._tk_eq(idx, (Token.Punctuation, "=")):

and uses more than 50 lines to collect it. I hope that textmate grammar can do it in less :)

[watermarkhu]

Yep I have just been running pytest locally to get through the true errors. Unfortunately it looks like there are fundamental problems with the textmate grammar that I am not sure are solvable due to the limitations of the textmate grammar. In particular the syntax for multiline defaults is I believe unparseable. @watermarkhu I am unsure if you have any insights that I do not here.

An example of the problem:

prop = [1;
              2;
              3]; % a prop

in which one cannot unambiguously identify which of the ; actually ends the property default.

There are two ways to tackle this.

1. Further edits to the MATLAB grammar: The MATLAB grammar can be further improved to catch these cases. In this case, we can add a dummy newline scope to the array and cell scopes.

2. Pre-parser:. Similar to the current case, there is MATLAB's line continuations ..., which is allowed to be inserted almost anywhere. I started on adding the line-continuation scope to places that I encountered them, but there are many, many more.

   • Fix for Line continuations on anonymous functions mathworks/MATLAB-Language-grammar#56
   • Fix for lost scope in case of line continuation with comment mathworks/MATLAB-Language-grammar#69
   • Fixes for multiple function definitions mathworks/MATLAB-Language-grammar#73

   Just for the sake of getting the tokens, I added a pre-parser to simply remove line-continuations from the source file. While this method is far from ideal, it is another possibility.

[apozharski]

This is replaced with #261