-
Notifications
You must be signed in to change notification settings - Fork 38
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Running tests without calling environment.InitFlags()
func leads to failed test being skipped instead of failure
#268
Comments
/kind bug |
* Adding `presubmit-tests.sh` script * Adding codecov * E2E tests beginning * Running integration tests from e2e runner * Eventshub starting, sort of. * E2E test is working * Resolver for kn-plugin-event binary location * Debugging the send error * Using proper image * Further dumping of events * Make sure to call `environment.InitFlags` func See also: knative-extensions/reconciler-test#268 * Upgrade knative.dev/reconciler-test * Clean up the e2e test * Test for sending to Knative service * Use library logger * Auto skip if short flag is used.
* Adding `presubmit-tests.sh` script * Adding codecov * E2E tests beginning * Running integration tests from e2e runner * Eventshub starting, sort of. * E2E test is working * Resolver for kn-plugin-event binary location * Debugging the send error * Using proper image * Further dumping of events * Make sure to call `environment.InitFlags` func See also: knative-extensions/reconciler-test#268 * Upgrade knative.dev/reconciler-test * Clean up the e2e test * Test for sending to Knative service * Use library logger * Auto skip if short flag is used.
* Adding `presubmit-tests.sh` script * Adding codecov * E2E tests beginning * Running integration tests from e2e runner * Eventshub starting, sort of. * E2E test is working * Resolver for kn-plugin-event binary location * Debugging the send error * Using proper image * Further dumping of events * Make sure to call `environment.InitFlags` func See also: knative-extensions/reconciler-test#268 * Upgrade knative.dev/reconciler-test * Clean up the e2e test * Test for sending to Knative service * Use library logger * Auto skip if short flag is used.
* 🎁 E2E tests (#126) * Adding `presubmit-tests.sh` script * Adding codecov * E2E tests beginning * Running integration tests from e2e runner * Eventshub starting, sort of. * E2E test is working * Resolver for kn-plugin-event binary location * Debugging the send error * Using proper image * Further dumping of events * Make sure to call `environment.InitFlags` func See also: knative-extensions/reconciler-test#268 * Upgrade knative.dev/reconciler-test * Clean up the e2e test * Test for sending to Knative service * Use library logger * Auto skip if short flag is used. * 🧹 Port parts of the eventshub readiness check (knative-extensions/reconciler-test#284) * 🐛 Always send to cluster local Ksvc (#146) * 🎁 Tests for Eventing (Broker & Channel) (#159) * Tests for Broker * Test for Channel * Enhance logging * Use the same version of golangci-lint as used on upstream.
This issue is stale because it has been open for 90 days with no |
/reopen |
@cardil: Reopened this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
/assign |
This issue is stale because it has been open for 90 days with no |
/remove-lifecycle stale |
The
environment.InitFlags()
is required to be executed. Without it, the flags are not set properly. Their values are outside of possible known values -s == 0
andl == 0
.https://github.com/knative-sandbox/reconciler-test/blob/816f2192fec9b6363c2b9ed79875aa54d472eede/pkg/environment/flags.go#L29-L30
This leads to nasty behavior - the
shouldFail(s *feature.Step) bool
func returns false, and instead of failures, the tests are skipped. This behavior is hard to understand.https://github.com/knative-sandbox/reconciler-test/blob/816f2192fec9b6363c2b9ed79875aa54d472eede/pkg/environment/magic.go#L306-L309
I think, we should amend the test framework to ensure flags have valid values. Either by forcing people to call
environment.InitFlags()
, or setting flag values with reasonable defaults.The text was updated successfully, but these errors were encountered: