Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Validation tests don't match reality #1407

Open
ghaith opened this issue Feb 10, 2025 · 0 comments
Open

Validation tests don't match reality #1407

ghaith opened this issue Feb 10, 2025 · 0 comments
Labels
bug Something isn't working

Comments

@ghaith
Copy link
Collaborator

ghaith commented Feb 10, 2025

Describe the bug
Our validation tests don't use the build pipeline, and thus don't stop on parser critical paths.
Furthermore, they sometimes show errors that the user does not actually see because they are not reported correctly, other times, validations are duplicated.
This means we think a section in our code is being validated while the user does not see a relevant error for that.

Solution
Use the pipeline for validation tests, this ensures the validations are the same as what the user will see.
This means that all tests that are failing need to be re-checked and missing validations we were expecting has to be reported correctly.
Anything deleted from the insta snapshot means the user never saw this before. Any additional line to the insta snapshot means the user was seeing this error but we were not taking it into account.

Side effect
All validation tests will be integration tests since we need the full pipeline, but this should be fine.

@ghaith ghaith added the bug Something isn't working label Feb 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant