-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Restructure Conformance Tests #6264
Comments
/assign @coryrc |
I intend to have something to discuss with serving WG next week. |
Fixes knative#6006 by making TestCustomResourceLimits to be a conformance test and moving e2e parts out. For knative#6226 fixes it for TestContainerErrorMsg and TestContainerExitingMsg, though could be other tests checking errors incorrectly. Addresses knative#6264 though continuing work is needed. Don't mix E2E and conformance: Strip E2E portions of conformance out into tests in the test/e2e directory. Commonality goes into test/scenarios or test/v1 depending on specificity of the common functionality. Rename to more align with runtime test naming (Must/Should).
Fixes knative#6006 by making TestCustomResourceLimits to be a conformance test and moving e2e parts out. For knative#6226 fixes it for TestContainerErrorMsg and TestContainerExitingMsg, though could be other tests checking errors incorrectly. Addresses knative#6264 though continuing work is needed. Don't mix E2E and conformance: Strip E2E portions of conformance out into tests in the test/e2e directory. Commonality goes into test/scenarios or test/v1 depending on specificity of the common functionality. Rename to more align with runtime test naming (Must/Should).
Issues go stale after 90 days of inactivity. Send feedback to Knative Productivity Slack channel or file an issue in knative/test-infra. /lifecycle stale |
Stale issues rot after 30 days of inactivity. Send feedback to Knative Productivity Slack channel or file an issue in knative/test-infra. /lifecycle rotten |
#6978 solves this |
This may also be relevant to the work @dprotaso has been looking into, but he's out for a few weeks. |
I'm back |
This issue is stale because it has been open for 90 days with no |
/remove-lifecycle stale Context dump: Serving's conformance requirements were outlined here: knative/pkg#1563 and a PoC test framework was written knative-sandbox/reconciler-test/pkg/test. Afterwards I was working towards having networking's conformance adopt this framework here: knative/networking#233 At this point the eventing test task force started to come up with their own requirements & frameworks. This started a consolidation around reconciler-test given their needs were a superset of servings. Next steps are to resolve gaps in the consolidate framework. Those issues are: |
Should this issue move to https://github.com/knative/specs now that there is a separate repository for conformance-related things (which can be versioned separately from Knative code itself)? /triage accepted |
Done. |
In what area(s)?
/area test-and-release
Describe the feature
Design doc
Summary:
Our API conformance tests were written before we had a pattern to split out tests and before the API specification identified ‘should’ and ‘must’ fields. This resulted in them covering a large swath of functionality in large tests, making it difficult to isolate conformance gaps. They need some help to become useful to people who wish to prove conformance and useful to people to make their implementation conformant.
The text was updated successfully, but these errors were encountered: