False positives on CyberDojo

[aslakhellesoy]

http://cyber-dojo.cucumber.io/review/show/AA5C4F80AD?avatar=squid&was_tag=17&now_tag=17

The image isn't recognising certain errors as errors, so green is reported for both errors and undefined steps sometimes.

Anything to add @JonJagger?

[JonJagger]

Looking at the Shouty workbook in the section headed "Snippet generation"
There are two places where it tells you to

1. put a pending step-definition in place
2. put a [raise "bad step"] step-definition in place
   And it says both cases will cause the build to fail.
   And it does fail in the sense of the stdout you see.
   But the cyber-dojo regex gives you a green...

[sebrose]

The logic in use is contained in https://github.com/cyber-dojo-languages/ruby-cucumber/blob/master/docker/red_amber_green.rb

lambda { |stdout,stderr,status|
  output = stdout + stderr

  return :amber if /SyntaxError/.match(output)
  return :amber if /NameError/.match(output)
  return :amber if /^[.-UF]*U[.-UF]*$/.match(output)
  return :red if /^[.-UF]*F[.-UF]*$/.match(output)
  return :green
}

It sounds like we need to add P to the logic. In JS I alsoo had to add A.