Account for problems being fixed in Python 3.12 #110

alexmojaki · 2023-06-30T20:02:26Z

Closes #109

Seems that several things have significantly improved in 3.12!

alexmojaki · 2023-06-30T20:11:02Z

GHA no longer supports Python 2.7: actions/setup-python#672

While there are ways we could continue testing 2.7, this seems like a good time to let it go. Anyone using 2.7 can still use an older version of asttokens (i.e. the current one) and they should be fine. Newer versions of asttokens have generally been motivated by newer versions of Python anyway, as with this PR.

hroncok · 2023-07-01T11:23:22Z

Using astroid from git HEAD and this PR, we get this in Fedora:

=================================== FAILURES ===================================
__________________________ TestAstroid.test_fixture9 ___________________________

self = <astroid.builder.AstroidBuilder object at 0x7f4ba936e720>
data = '_\nclass Aaaa(base):', modname = '', path = None

    def _data_build(
        self, data: str, modname: str, path: str | None
    ) -> tuple[nodes.Module, rebuilder.TreeRebuilder]:
        """Build tree node from data and add some informations."""
        try:
>           node, parser_module = _parse_string(data, type_comments=True)

/usr/lib/python3.12/site-packages/astroid/builder.py:176: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

data = '_\nclass Aaaa(base):', type_comments = True

    def _parse_string(
        data: str, type_comments: bool = True
    ) -> tuple[ast.Module, ParserModule]:
        parser_module = get_parser_module(type_comments=type_comments)
        try:
>           parsed = parser_module.parse(data + "\n", type_comments=type_comments)

/usr/lib/python3.12/site-packages/astroid/builder.py:482: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = ParserModule(unary_op_classes={<class 'ast.UAdd'>: '+', <class 'ast.USub'>: '-', <class 'ast.Not'>: 'not', <class 'ast...<class 'ast.Store'>: <Context.Store: 2>, <class 'ast.Del'>: <Context.Del: 3>, <class 'ast.Param'>: <Context.Store: 2>})
string = '_\nclass Aaaa(base):\n', type_comments = True

    def parse(self, string: str, type_comments: bool = True) -> ast.Module:
>       return ast.parse(string, type_comments=type_comments)

/usr/lib/python3.12/site-packages/astroid/_ast.py:26: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

source = '_\nclass Aaaa(base):\n', filename = '<unknown>', mode = 'exec'

    def parse(source, filename='<unknown>', mode='exec', *,
              type_comments=False, feature_version=None):
        """
        Parse the source into an AST node.
        Equivalent to compile(source, filename, mode, PyCF_ONLY_AST).
        Pass type_comments=True to get back type comments where the syntax allows.
        """
        flags = PyCF_ONLY_AST
        if type_comments:
            flags |= PyCF_TYPE_COMMENTS
        if feature_version is None:
            feature_version = -1
        elif isinstance(feature_version, tuple):
            major, minor = feature_version  # Should be a 2-tuple.
            if major != 3:
                raise ValueError(f"Unsupported major version: {major}")
            feature_version = minor
        # Else it should be an int giving the minor version for 3.x.
>       return compile(source, filename, mode, flags,
                       _feature_version=feature_version)
E         File "<unknown>", line 2
E           class Aaaa(base):
E                            ^
E       IndentationError: expected an indented block after class definition on line 2

/usr/lib64/python3.12/ast.py:52: IndentationError

The above exception was the direct cause of the following exception:

self = <tests.test_astroid.TestAstroid testMethod=test_fixture9>

>   def test_fixture9(self): self.verify_fixture_file('astroid/module2.py')

tests/test_mark_tokens.py:181: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_mark_tokens.py:140: in verify_fixture_file
    tested_nodes = m.verify_all_nodes(self)
tests/tools.py:107: in verify_all_nodes
    rebuilt_node = test_case.parse_snippet(text, node)
tests/test_mark_tokens.py:804: in parse_snippet
    return self.module.parse('_\n' + text).body[1]
/usr/lib/python3.12/site-packages/astroid/builder.py:305: in parse
    return builder.string_build(code, modname=module_name, path=path)
/usr/lib/python3.12/site-packages/astroid/builder.py:146: in string_build
    module, builder = self._data_build(data, modname, path)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <astroid.builder.AstroidBuilder object at 0x7f4ba936e720>
data = '_\nclass Aaaa(base):', modname = '', path = None

    def _data_build(
        self, data: str, modname: str, path: str | None
    ) -> tuple[nodes.Module, rebuilder.TreeRebuilder]:
        """Build tree node from data and add some informations."""
        try:
            node, parser_module = _parse_string(data, type_comments=True)
        except (TypeError, ValueError, SyntaxError) as exc:
>           raise AstroidSyntaxError(
                "Parsing Python code failed:\n{error}",
                source=data,
                modname=modname,
                path=path,
                error=exc,
            ) from exc
E           astroid.exceptions.AstroidSyntaxError: Parsing Python code failed:
E           expected an indented block after class definition on line 2 (<unknown>, line 2)

/usr/lib/python3.12/site-packages/astroid/builder.py:178: AstroidSyntaxError
_________________________ TestAstroid.test_sys_modules _________________________

self = <astroid.builder.AstroidBuilder object at 0x7f4ba81a0b90>
data = '_\ndef create_module(self, spec):', modname = '', path = None

    def _data_build(
        self, data: str, modname: str, path: str | None
    ) -> tuple[nodes.Module, rebuilder.TreeRebuilder]:
        """Build tree node from data and add some informations."""
        try:
>           node, parser_module = _parse_string(data, type_comments=True)

/usr/lib/python3.12/site-packages/astroid/builder.py:176: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

data = '_\ndef create_module(self, spec):', type_comments = True

    def _parse_string(
        data: str, type_comments: bool = True
    ) -> tuple[ast.Module, ParserModule]:
        parser_module = get_parser_module(type_comments=type_comments)
        try:
>           parsed = parser_module.parse(data + "\n", type_comments=type_comments)

/usr/lib/python3.12/site-packages/astroid/builder.py:482: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = ParserModule(unary_op_classes={<class 'ast.UAdd'>: '+', <class 'ast.USub'>: '-', <class 'ast.Not'>: 'not', <class 'ast...<class 'ast.Store'>: <Context.Store: 2>, <class 'ast.Del'>: <Context.Del: 3>, <class 'ast.Param'>: <Context.Store: 2>})
string = '_\ndef create_module(self, spec):\n', type_comments = True

    def parse(self, string: str, type_comments: bool = True) -> ast.Module:
>       return ast.parse(string, type_comments=type_comments)

/usr/lib/python3.12/site-packages/astroid/_ast.py:26: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

source = '_\ndef create_module(self, spec):\n', filename = '<unknown>'
mode = 'exec'

    def parse(source, filename='<unknown>', mode='exec', *,
              type_comments=False, feature_version=None):
        """
        Parse the source into an AST node.
        Equivalent to compile(source, filename, mode, PyCF_ONLY_AST).
        Pass type_comments=True to get back type comments where the syntax allows.
        """
        flags = PyCF_ONLY_AST
        if type_comments:
            flags |= PyCF_TYPE_COMMENTS
        if feature_version is None:
            feature_version = -1
        elif isinstance(feature_version, tuple):
            major, minor = feature_version  # Should be a 2-tuple.
            if major != 3:
                raise ValueError(f"Unsupported major version: {major}")
            feature_version = minor
        # Else it should be an int giving the minor version for 3.x.
>       return compile(source, filename, mode, flags,
                       _feature_version=feature_version)
E         File "<unknown>", line 2
E           def create_module(self, spec):
E                                         ^
E       IndentationError: expected an indented block after function definition on line 2

/usr/lib64/python3.12/ast.py:52: IndentationError

The above exception was the direct cause of the following exception:

self = <tests.test_astroid.TestAstroid testMethod=test_sys_modules>

    def test_sys_modules(self):
      """
      Verify all nodes on source files obtained from sys.modules.
      This can take a long time as there are many modules,
      so it only tests all modules if the environment variable
      ASTTOKENS_SLOW_TESTS has been set.
      """
      modules = list(sys.modules.values())
      if not os.environ.get('ASTTOKENS_SLOW_TESTS'):
        modules = modules[:20]
    
      start = time()
      for module in modules:
        # Don't let this test (which runs twice) take longer than 13 minutes
        # to avoid the travis build time limit of 30 minutes
        if time() - start > 13 * 60:
          break
    
        try:
          filename = inspect.getsourcefile(module)
        except TypeError:
          continue
    
        if not filename:
          continue
    
        filename = os.path.abspath(filename)
        print(filename)
        try:
          with io.open(filename) as f:
            source = f.read()
        except OSError:
          continue
    
        if self.is_astroid_test and (
            # Astroid fails with a syntax error if a type comment is on its own line
            re.search(r'^\s*# type: ', source, re.MULTILINE)
            # Astroid can fail on this file, specifically raising an exception at this line of code:
            #     lambda node: node.name == "NamedTuple" and node.parent.name == "typing"
            # with the error:
            #     AttributeError: 'If' object has no attribute 'name'
            # See https://github.com/gristlabs/asttokens/runs/7602147792
            # I think the code that causes the problem is:
            #     if sys.version_info >= (3, 11):
            #         NamedTuple = typing.NamedTuple
            or filename.endswith("typing_extensions.py")
        ):
          print('Skipping', filename)
          continue
    
>       self.create_mark_checker(source)

tests/test_mark_tokens.py:673: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_mark_tokens.py:49: in create_mark_checker
    checker.verify_all_nodes(self)
tests/tools.py:107: in verify_all_nodes
    rebuilt_node = test_case.parse_snippet(text, node)
tests/test_mark_tokens.py:804: in parse_snippet
    return self.module.parse('_\n' + text).body[1]
/usr/lib/python3.12/site-packages/astroid/builder.py:305: in parse
    return builder.string_build(code, modname=module_name, path=path)
/usr/lib/python3.12/site-packages/astroid/builder.py:146: in string_build
    module, builder = self._data_build(data, modname, path)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <astroid.builder.AstroidBuilder object at 0x7f4ba81a0b90>
data = '_\ndef create_module(self, spec):', modname = '', path = None

    def _data_build(
        self, data: str, modname: str, path: str | None
    ) -> tuple[nodes.Module, rebuilder.TreeRebuilder]:
        """Build tree node from data and add some informations."""
        try:
            node, parser_module = _parse_string(data, type_comments=True)
        except (TypeError, ValueError, SyntaxError) as exc:
>           raise AstroidSyntaxError(
                "Parsing Python code failed:\n{error}",
                source=data,
                modname=modname,
                path=path,
                error=exc,
            ) from exc
E           astroid.exceptions.AstroidSyntaxError: Parsing Python code failed:
E           expected an indented block after function definition on line 2 (<unknown>, line 2)

/usr/lib/python3.12/site-packages/astroid/builder.py:178: AstroidSyntaxError
----------------------------- Captured stdout call -----------------------------
/usr/lib64/python3.12/importlib/_bootstrap.py
/usr/lib64/python3.12/importlib/_bootstrap_external.py
=========================== short test summary info ============================
FAILED tests/test_astroid.py::TestAstroid::test_fixture9 - astroid.exceptions...
FAILED tests/test_astroid.py::TestAstroid::test_sys_modules - astroid.excepti...

hroncok · 2023-07-01T11:28:14Z

Is it possible that this would in fact be a problem in astroid?

setup.cfg

aqeelat · 2023-07-02T07:48:08Z

@alexmojaki This is not a review of the fix you created but of the deprecation of Python 2.7.

Don't forget to remove the python 2 and 2.7 classifiers and the dependency on six.

Although, I think that should be done in a separate PR that gets merged before this one. I can give it a shot if you create an GH issue and assign it to me.

alexmojaki · 2023-07-02T11:56:01Z

Don't forget to remove the python 2 and 2.7 classifiers

Thanks, this is an important reminder.

and the dependency on six. Although, I think that should be done in a separate PR that gets merged before this one.

I don't really see why continuing to depend on six is a critical problem that should block this PR (although this PR isn't actually ready to merge yet), even without Python 2 support.

I can give it a shot if you create an GH issue and assign it to me.

I think we need to decide to actually drop Python 2 support, and that's up to @dsagal. But I've opened #111 for that discussion to continue there.

dsagal · 2023-08-31T03:52:15Z

asttokens/mark_tokens.py

+    if sys.version_info >= (3, 12):
+      last = first_token
+      while True:
+        if util.match_token(last, getattr(token, "FSTRING_START")):


Could you add brief comments here, or maybe an example? Also curious why the syntax getattr(token, "FSTRING_START") vs token.FSTRING_START? Is that mainly for editor/linting convenience?

Yes, mypy doesn't currently accept token.FSTRING_START: https://github.com/gristlabs/asttokens/actions/runs/6036177479/job/16377962926

Amazingly, it only complains about it in 3.12. My guess is that earlier versions respect the sys.version_info >= (3, 12). But they still complain about type: ignore if I use that, claiming it's unused: https://github.com/gristlabs/asttokens/actions/runs/6036208109/job/16378046034

asttokens/util.py

tests/test_util.py

…versions

dsagal

Thank you!

aqeelat reviewed Jul 2, 2023

View reviewed changes

setup.cfg Show resolved Hide resolved

alexmojaki mentioned this pull request Jul 2, 2023

Drop Python 2.7? #111

Open

alexmojaki added 9 commits August 29, 2023 19:57

Account for problems being fixed in Python 3.12

d51702e

Test 3.12 on GHA

0396af5

Fix GHA python versions

48b6158

mypy 3.12

7cb69d9

pypy3.7 is having problems in CI

d4f9681

allow unused type ignore

8b6b704

account for nested f-strings

f4a9b2a

test_nested_fstrings

8e4be6d

restore 2.7 on GHA, test newer 3.12

715d543

alexmojaki force-pushed the 3.12 branch from 0958b40 to 715d543 Compare August 29, 2023 18:00

alexmojaki added 6 commits August 29, 2023 20:04

try pypy-3.7 again

6fefb2c

try dealing with type errors without type:ignore

56d8e11

catch weird errors from inspect.getsourcefile

cca515d

fix test_nested_fstrings for when not fstring_positions_work()

4229ce9

skip a file which has astroid problems

9104f66

Handle weird astroid errors in a more robust way

1654b40

alexmojaki requested a review from dsagal August 30, 2023 12:10

dsagal reviewed Aug 31, 2023

View reviewed changes

alexmojaki added 5 commits August 31, 2023 12:30

try without getattr

bf98796

type ignore

9ef9bdd

back to getattr, add comments.

58dba97

comments and clarification

0b4650a

try another approach for selecting original tokens that works across …

b18cbae

…versions

dsagal approved these changes Aug 31, 2023

View reviewed changes

alexmojaki merged commit bfa029e into master Aug 31, 2023
16 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Account for problems being fixed in Python 3.12 #110

Account for problems being fixed in Python 3.12 #110

alexmojaki commented Jun 30, 2023

alexmojaki commented Jun 30, 2023

hroncok commented Jul 1, 2023

hroncok commented Jul 1, 2023

aqeelat commented Jul 2, 2023 •

edited

Loading

alexmojaki commented Jul 2, 2023

dsagal Aug 31, 2023

alexmojaki Aug 31, 2023 •

edited

Loading

alexmojaki Aug 31, 2023

dsagal left a comment

Account for problems being fixed in Python 3.12 #110

Account for problems being fixed in Python 3.12 #110

Conversation

alexmojaki commented Jun 30, 2023

alexmojaki commented Jun 30, 2023

hroncok commented Jul 1, 2023

hroncok commented Jul 1, 2023

aqeelat commented Jul 2, 2023 • edited Loading

alexmojaki commented Jul 2, 2023

dsagal Aug 31, 2023

Choose a reason for hiding this comment

alexmojaki Aug 31, 2023 • edited Loading

Choose a reason for hiding this comment

alexmojaki Aug 31, 2023

Choose a reason for hiding this comment

dsagal left a comment

Choose a reason for hiding this comment

aqeelat commented Jul 2, 2023 •

edited

Loading

alexmojaki Aug 31, 2023 •

edited

Loading