W606 in async function

[afq984]

async def foo():
    await [3]


def bar():
    await = [1, 2, 3, 4]
    await [3]

pycodestyle --version; python -V; pycodestyle aw.py 
2.7.0
Python 3.7.10
aw.py:2:5: W606 'async' and 'await' are reserved keywords starting with Python 3.7
aw.py:6:5: W606 'async' and 'await' are reserved keywords starting with Python 3.7
aw.py:7:5: W606 'async' and 'await' are reserved keywords starting with Python 3.7

pycodestyle --version; python -V; pycodestyle aw.py
2.7.0
Python 3.6.13
aw.py:6:5: W606 'async' and 'await' are reserved keywords starting with Python 3.7
aw.py:7:5: W606 'async' and 'await' are reserved keywords starting with Python 3.7
aw.py:7:10: E211 whitespace before '['

I think aw.py:2:5 should say: list object is not awaitable, considering that the code is already in an async function.
A user writing code in an async function might not understand which objects are awaitable, but I'd expect them to be aware of async/await keywords.

Maybe this semantic check can't be done in pycodestyle, and would be better lived in pyflakes.

W606 also pops out for cases like await 3, where it's still invalid syntax even if await is intended as an identifier. await [3] is tricky thus shown in the examples above, await is both valid as a keyword or as an identifier in await [3]. I believe using whether await sits in an async function to enable/disable W606 is good enough.

[sigmavirus24]

await isn't a valid identifier in Python 3.7+ which is what the check is telling you. Using it as such will break for you regardless of whether it's inside an async function or not

[afq984]

Adding a little background here. I have CI was set up to use flake8. A contributor wrote something like the following in a PR:

async def get_courses(client, urls):
    pages = await [page async for client.fetch(url) for url in urls]
    ...

It is obvious to experienced programmers that you cannot await on a list. But it took us a while to figure out why there was a warning about W606 'async' and 'await' are reserved keywords starting with Python 3.7. Here await is intended as a keyword but the contributor believed that there's some imaginary "async list" type. So I think list object is not awaitible would be more helpful here.

[afq984]

Using it as such will break for you regardless of whether it's inside an async function or not

Agreed.

I was just thinking that it'd be less likely for people to accidentally use await as an identifier in async functions, than in regular functions. People working with async stuff should probably already know the keywords and people working only with regular functions might be completely unaware of those.

The example above #985 (comment) shows that await an un-awaitable gets interpreted as using await as identifier.

[asottile]

this appears to be working as intended

[afq984]

@asottile would you mind elaborating how this is working as intended?

Let me lay down my observations:

0. I would interpret 'async' and 'await' are reserved keywords starting with Python 3.7 as don't use async or await as identifiers.
1. In Python 3.7+, all cases of using await as identifiers are reported as SyntaxError by CPython.
2. In Python 3.5-3.6, using await as identifiers in async def functions are reported as SyntaxError by Cpython.
3. In Python 3.4-, async def is invalid.

In the case:

async def foo():
    await 1
import asyncio
asyncio.get_event_loop().run_until_complete(foo())

This code means the same thing across all python versions supporting async def: load the int object 1 and await the object. And the behavior:

• CPython>=3.5: TypeError: object int can't be used in 'await' expression.
• pycodestyle with CPython<3.7: Happy with the await line.
• pycodestyle with CPython>=3.7: W606 'async' and 'await' are reserved keywords starting with Python 3.7.

The problem I see here is:

1. In all CPython versions, when the await token appears in an async def body, it will be treated by the interpreter as a keyword. If it is used as an identifier, a SyntaxError will pop out. PEP492 also explicitly mentions async/await are specifically treated as keywords. So one can not actually use async/await as identifiers in async def bodies; SyntaxError is also super easy to catch. Given the above, I'd argue that W606 is not that useful in async def bodies, and it also caused problems in the past: W606 false warning on consecutive await #811 W606: false positive when selecting async function from list by index #936.
2. Not sure why the behavior is inconsistent between Python3.6- and Python3.7+. In Python3.7+ async/await can only be used as keywords (so SyntaxErrors show up) but the W606 warning is absent in Python3.6-. I'm assuming W606 is about preventing future Python upgrade incompatibilities.

Apologies if I got it all wrong from my observation#0.

[asottile]

your example is garbage:

$ python3 t.py
Traceback (most recent call last):
  File "t.py", line 4, in <module>
    asyncio.get_event_loop().run_until_complete(foo())
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "t.py", line 2, in foo
    await 1
TypeError: object int can't be used in 'await' expression

garbage in, garbage out (in this case, a lint error)

pycodestyle doesn't have access to the ast so it approximates "async / await not used properly" as "has an operator after it". this means it cannot know whether it is inside an async function or not

silencing the warning you're asking to be silenced would have hidden a bug in your code, I don't think you want that

it's ignored in versions prior to 3.7 because it was not an error then

[afq984]

garbage in, garbage out

asking to be silenced would have hidden a bug in your code

That's exactly the case and I think the check (for a TypeError, which is consistent across python3.5-3.9) belongs to a linter like pyflakes rather than a style checker. A linter would be more capable to report these kinds of errors accurately. But I'll respect your project goals here.

it cannot know whether it is inside an async function or not

I thought it is already using some heuristic for that:

root@f8a5365964b4:/# cat t.py 
async def foo():
    await 1
def bar():
    await 1
root@f8a5365964b4:/# pycodestyle t.py 
t.py:3:1: E302 expected 2 blank lines, found 0
t.py:4:5: W606 'async' and 'await' are reserved keywords starting with Python 3.7
root@f8a5365964b4:/# python -V
Python 3.6.13

[sigmavirus24]

There isn't because 1 isn't an operator like [1] is. You could probably replicate this in your example with the semantically equivalent

async def foo():
    await +1

async def foo():
    await -1

Because pycodestyle will interpret that as an expression.

[afq984]

it's ignored in versions prior to 3.7 because it was not an error then -- #985 (comment)

No, it's a TypeError, at least for the particular example

$ python3.6 -q
>>> async def foo():
...     await 1
... 
>>> import asyncio
>>> asyncio.get_event_loop().run_until_complete(foo())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/asyncio/base_events.py", line 488, in run_until_complete
    return future.result()
  File "<stdin>", line 2, in foo
TypeError: object int can't be used in 'await' expression

So in Python3.6:

• async = 1 gets W606 while being valid;
• async def foo(): await 1 passes while being a TypeError

I thought it is already using some heuristic for that: -- #985 (comment)

My bad, just looked into the code and there's no heuristic whether a await is within async def. Maybe that's an unintended side effect?
In Python 3.5-3.6 the foo() await is tokenize.AWAIT while the bar() await is tokenize.NAME.
Thus we only have W606 in bar() in Python 3.5-3.6. In Python3.7 they all became tokenize.NAME so the gap is closed.

async def foo():
     await 1
def bar():
     await 1

tokenize output

root@c8ad34da8166:/# cat t.py 
async def foo():
     await 1
def bar():
     await 1
root@c8ad34da8166:/# python3.6 -m tokenize t.py
0,0-0,0:            ENCODING       'utf-8'        
1,0-1,5:            ASYNC          'async'        
1,6-1,9:            NAME           'def'          
1,10-1,13:          NAME           'foo'          
1,13-1,14:          OP             '('            
1,14-1,15:          OP             ')'            
1,15-1,16:          OP             ':'            
1,16-1,17:          NEWLINE        '\n'           
2,0-2,5:            INDENT         '     '        
2,5-2,10:           AWAIT          'await'        
2,11-2,12:          NUMBER         '1'            
2,12-2,13:          NEWLINE        '\n'           
3,0-3,0:            DEDENT         ''             
3,0-3,3:            NAME           'def'          
3,4-3,7:            NAME           'bar'          
3,7-3,8:            OP             '('            
3,8-3,9:            OP             ')'            
3,9-3,10:           OP             ':'            
3,10-3,11:          NEWLINE        '\n'           
4,0-4,5:            INDENT         '     '        
4,5-4,10:           NAME           'await'        
4,11-4,12:          NUMBER         '1'            
4,12-4,13:          NEWLINE        '\n'           
5,0-5,0:            DEDENT         ''             
5,0-5,0:            ENDMARKER      ''             

edit: specifically, tokenize.ASYNC and tokenize.AWAIT didn't receive special care in:

pycodestyle/pycodestyle.py

Line 1675 in 76edc0a

def python_3000_async_await_keywords(logical_line, tokens):