Regression: KeyError in replacer

[The-Compiler]

With #445, I'm seeing many tracebacks like this in my tests:

______________________ test_inserting_text_into_a_text_field_at_specific_position _______________________

request = <FixtureRequest for <Function test_inserting_text_into_a_text_field_at_specific_position>>
_pytest_bdd_example = {}

    @pytest.mark.usefixtures(*args)
    def scenario_wrapper(request, _pytest_bdd_example):
>       scenario = templated_scenario.render(_pytest_bdd_example)

.tox/bleeding/lib/python3.9/site-packages/pytest_bdd/scenario.py:173: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
.tox/bleeding/lib/python3.9/site-packages/pytest_bdd/parser.py:249: in render
    steps = [
.tox/bleeding/lib/python3.9/site-packages/pytest_bdd/parser.py:251: in <listcomp>
    name=templated_step.render(context),
.tox/bleeding/lib/python3.9/site-packages/pytest_bdd/parser.py:364: in render
    return STEP_PARAM_RE.sub(replacer, self.name)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

m = <re.Match object; span=(18, 24), match='<Home>'>

    def replacer(m: typing.Match):
        varname = m.group(1)
>       return str(context[varname])
E       KeyError: 'Home'

.tox/bleeding/lib/python3.9/site-packages/pytest_bdd/parser.py:362: KeyError

There are many other such failures, but this one is from this scenario:

    Scenario: Inserting text into a text field at specific position
        When I open data/paste_primary.html
        And I insert "one two three four" into the text field
        And I run :click-element id qute-textarea
        And I wait for "Entering mode KeyMode.insert (reason: clicking input)" in the log
        # Move to the beginning and two characters to the right
        And I press the keys "<Home>"
        And I press the key "<Right>"
        And I press the key "<Right>"
        And I run :insert-text Hello world
        # Compare
        Then the javascript message "textarea contents: onHello worlde two three four" should be logged

specifically, the And I press the keys "<Home>" line there.

The underlying Python code is simple:

@bdd.when(bdd.parsers.re('I press the keys? "(?P<keys>[^"]*)"'))
def press_keys(quteproc, keys):
    """Send the given fake keys to qutebrowser."""
    quteproc.press_keys(keys)

A slightly simpler example:

    Scenario: :selection-follow with link tabbing (without JS)
        When I set content.javascript.enabled to false
        And I run :mode-leave
        And I run :jseval document.activeElement.blur();
        And I run :fake-key <tab>
        And I run :selection-follow
        Then data/hello.txt should be loaded

with this code:

@bdd.when(bdd.parsers.parse("I run {command}"))
def run_command(quteproc, server, tmpdir, command):
    # ...

results in a KeyError: tab. In other words, it seems like anything in <...> in a scenario now seems to be parsed in some special way.

---

I've tried to write a reproducer:

bug.feature:

Feature: Reproducer
    Scenario: Pressing keys
        When I run :fake-key <Ctrl+c>
        And I run :fake-key <tab>

test_bug.py:

import pytest_bdd as bdd

bdd.scenarios('bug.feature')

@bdd.when(bdd.parsers.parse("I run {command}"))
def run_command(command):
    pass

but unfortunately, I can not reproduce the issue there. Any ideas what could be going wrong there?

[elchupanebrej]

<...> is treated as a step parameter and is rendered during Scenario execution using scenario examples. There is no way to escape < symbol for now

[The-Compiler]

The changelog for 5.0.0 doesn't seem to indicate this anywhere - it links to a "Migration from 4.x.x" document which doesn't seem to exist.

Also, even if that's really the culprit, it seems to me like the error message should be a bit clearer than just an unhandled KeyError somewhere deep in pytest-bdd...

[dcendents]

I have the same error, I need to validate the string value or a variable.
In one of the tests the string contains ".....", this is a business value that needs to be tested, not some test convention that I can modify.

[dcendents]

as a workaround until this is fixed, if we use an examples table to feed the value it will work

Given the following step definition:

@given(parsers.parse("variable {var} is set to '{value}'"))
def set_var(context, var, value):
    context[var] = value

This will fail:

Scenario: Test fail string with <value>
    Given variable v is set to 'some value with <value> in it'

This will pass:

Scenario: Test pass with example
    Given variable v is set to '<value>'

    Examples: Vertical
        | value | some value with <value> in it |

[bbatliner]

FWIW the parameterization example in the README appears broken on 5.0.0 after its changes in #445. What I ran:

import pytest
from pytest_bdd import scenario, given, when, then, parsers

# Here we use pytest to parametrize the test with the parameters table
@pytest.mark.parametrize(
    ['start', 'eat', 'left'],
    [(12, 5, 7)])
@scenario(
    'parametrized.feature',
    'Parametrized given, when, thens',
)
# Note that we should take the same arguments in the test function that we use
# for the test parametrization either directly or indirectly (fixtures depend on them).
def test_parametrized(start, eat, left):
    """We don't need to do anything here, everything will be managed by the scenario decorator."""

@given(parsers.parse('there are {start:d} cucumbers'), target_fixture="start_cucumbers")
def start_cucumbers(start):
    return dict(start=start)

@when(parsers.parse('I eat {eat:d} cucumbers'))
def eat_cucumbers(start_cucumbers, start, eat):
    start_cucumbers['eat'] = eat

@then(parsers.parse('I should have {left:d} cucumbers'))
def should_have_left_cucumbers(start_cucumbers, start, eat, left):
    assert start - eat == left
    assert start_cucumbers['start'] == start
    assert start_cucumbers['eat'] == eat

Feature: parametrized

    Scenario: Parametrized given, when, thens
        Given there are <start> cucumbers
        When I eat <eat> cucumbers
        Then I should have <left> cucumbers

$ pytest test_parameter.py
===================================================================================== test session starts ======================================================================================
platform linux -- Python 3.6.8, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /home/bbatliner/package, configfile: pytest.ini
plugins: bdd-5.0.0
collected 1 item                                                                                                                                                                               

test_parameter.py F                                                                                                                                                                      [100%]

=========================================================================================== FAILURES ===========================================================================================
__________________________________________________________________________________ test_parametrized[12-5-7] ___________________________________________________________________________________

request = <FixtureRequest for <Function test_parametrized[12-5-7]>>, _pytest_bdd_example = {}

    @pytest.mark.usefixtures(*args)
    def scenario_wrapper(request, _pytest_bdd_example):
>       scenario = templated_scenario.render(_pytest_bdd_example)

../.cache/pypoetry/virtualenvs/package-dlKzdzJ9-py3.6/lib/python3.6/site-packages/pytest_bdd/scenario.py:173: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../.cache/pypoetry/virtualenvs/package-dlKzdzJ9-py3.6/lib/python3.6/site-packages/pytest_bdd/parser.py:257: in render
    for templated_step in self.steps
../.cache/pypoetry/virtualenvs/package-dlKzdzJ9-py3.6/lib/python3.6/site-packages/pytest_bdd/parser.py:257: in <listcomp>
    for templated_step in self.steps
../.cache/pypoetry/virtualenvs/package-dlKzdzJ9-py3.6/lib/python3.6/site-packages/pytest_bdd/parser.py:364: in render
    return STEP_PARAM_RE.sub(replacer, self.name)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

m = <_sre.SRE_Match object; span=(10, 17), match='<start>'>

    def replacer(m: typing.Match):
        varname = m.group(1)
>       return str(context[varname])
E       KeyError: 'start'

../.cache/pypoetry/virtualenvs/package-dlKzdzJ9-py3.6/lib/python3.6/site-packages/pytest_bdd/parser.py:362: KeyError
=================================================================================== short test summary info ====================================================================================
FAILED test_parameter.py::test_parametrized[12-5-7] - KeyError: 'start'

The <...> syntax in feature files raises KeyError, even if the the < is intended to be a literal (like in @The-Compiler's use cases) or is intended to be overridden by an existing pytest fixture.

[bbatliner]

My apologies, the above comment has an issue already: #448. Including a solution that is tested.

[jirikuncar]

I am having a same issue but my step definitions contain text values with <KEY> that is not part of the example.

Feature: print values
  Scenario: print HTML tag
    Given tag p
    When printed with text
    Then the document contains "<p>text</p>"

raises KeyError: 'p'

Replacer should only replace keys defined in examples of a scenario outline.

[youtux]

Sorry for the inconvenience, when I implemented this change (to behave more according to the Gherkin specs) I didn't foresee this edge case. It seems that Gherkin does not mention how to workaround this issue in https://cucumber.io/docs/gherkin/reference/#scenario-outline.

I see a couple of options here to solve the problem:

1. One option is to apply templating only in scenarios that use the Scenario Outline (or Scenario Template) keyword, but it's still limiting what you can put in your steps in that case.

2. Another option is to proceed as suggested in Ignore </> in step parameter values #465 and just silently ignore the fact that the render expected to render a parameter, and instead leave the original text. I'm not a big fan of this one though, as adding an Example table with h1 would have unexpected consequences on the test (it would be testing something very different).

@olegpidsadnyi, what's your opinion on this?

[jirikuncar]

adding an Example table with h1 would have unexpected consequences on the test (it would be testing something very different).

I find it a less of a problem as it can be solved by renaming example parameters.

[elchupanebrej]

2. Another option is to proceed as suggested in Ignore </> in step parameter values #465 and just silently ignore the fact that the render expected to render a parameter, and instead leave the original text. I'm not a big fan of this one though, as adding an Example table with h1 would have unexpected consequences on the test (it would be testing something very different).

3. We could raise a warning. In case if it really wanted behavior user could just filter it out

[elchupanebrej]

Approach from #439 also could be a solution

[TBBle]

Sorry for the inconvenience, when I implemented this change (to behave more according to the Gherkin specs) I didn't foresee this edge case. It seems that Gherkin does not mention how to workaround this issue in https://cucumber.io/docs/gherkin/reference/#scenario-outline.

For the record, the approach in Cucumber that hit the stage of "send a PR and we'll see" in cucumber/common#1004 (before the issue timed-out a couple of years ago) was to use hot-comments to change the example-delimiter markers for a feature file.

AFAIK pytest-bdd doesn't support the existing language and encoding hot-comments in Gherkin, but the idea of using a different delimiter setting for a feature file (or at a different granularity, e.g. @scenario/scenarios which are the common user-facing API that already carry an encoding parameter) might allow resolving this issue for pytest-bdd users.

[youtux]

I'm fixing this in #524 by parsing the params in angular brackets only for Scenario Outlines. Normal Scenarios will treat them verbatim.

[The-Compiler]

Wheeee, thanks for the fix! I just upgraded from 4.1.0 to 6.0.0 finally, and it seems to work great again! ✨

(Other than a bug in my own code the upgrade did uncover: tests: Fix broken BDD definition · qutebrowser/qutebrowser@a16aa95)

[elchupanebrej]

This fix could be ok for the current Gherkin dialect implemented in pytest_bdd, but in official gherkin language Scenario, Example and Scenario Outline are just aliases and all of them could be used with angular brackets

#439 (comment)

[youtux]

@elchupanebrej I don’t see that in the Gherkin reference. They say that Scenario and Example are aliases, that Scenario Outline and Scenario Template are aliases, but not that all four of these are aliases.

[elchupanebrej]

You are right! But internal implementation of the official Gherkin parser uses all of them as aliases. This could be checked by using parse directly https://github.com/cucumber/common/tree/main/gherkin