Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor and simplify e2e test traces loading and execution #2435

Closed
1 of 2 tasks
Tracked by #2406
MSalopek opened this issue Apr 24, 2023 · 0 comments
Closed
1 of 2 tasks
Tracked by #2406

Refactor and simplify e2e test traces loading and execution #2435

MSalopek opened this issue Apr 24, 2023 · 0 comments
Assignees
Labels
admin: epic An EPIC -- meta issue used to track a body of work other: decayed Stale issues that need follow up from commentators. Were closed for inactivity S: Productivity Productivity: Developer tooling, infrastructure improvements enabling future growth scope: testing Code review, testing, making sure the code is following the specification. type: feature-request New feature or request improvement type: tech-debt Slows down development in the long run

Comments

@MSalopek
Copy link
Contributor

MSalopek commented Apr 24, 2023

At present the gaia e2e testing uses hand-written tests to ensure normal operations are passing (happy path tests) and that no sources of non-determinisim exist.

The ICS e2e testing framework also uses hand-written test traces and executes them using the test runner.

All tests are performed in an isolated environment (docker container).

It would be beneficial to adopt a standardized trace format that comprises the following features:

  1. operation to perform
  2. system state before the operation
  3. system state after executing the operation

Adopting a standardized format would enable fast and reproducible test trace generation using MBT tooling (as opposed to writing tests traces manually), would help standardize the test output and enable meaningful and comprehensive test results.

In the first steps towards establishing e2e test trace generation it is important to maintain parity with the existing test frameworks. Existing test frameworks should be used as a benchmark for comparing all future work. The selected format should allow e2e testing framework refactoring with regards to ease of use and configuration.

Existing frameworks:

Existing trace formats:


Tasks

  1. S: Productivity scope: testing
    p-offtermatt
@MSalopek MSalopek assigned MSalopek and p-offtermatt and unassigned MSalopek Apr 24, 2023
@mpoke mpoke added the more-info-needed Further information is requested label Apr 24, 2023
@MSalopek MSalopek added type: feature-request New feature or request improvement scope: testing Code review, testing, making sure the code is following the specification. automation type: tech-debt Slows down development in the long run and removed more-info-needed Further information is requested labels Apr 24, 2023
@MSalopek MSalopek self-assigned this Apr 24, 2023
@github-project-automation github-project-automation bot moved this to 🩹 Triage in Cosmos Hub Apr 24, 2023
@MSalopek MSalopek moved this from 🩹 Triage to 🏗 In progress in Cosmos Hub Apr 24, 2023
@mpoke mpoke added the admin: epic An EPIC -- meta issue used to track a body of work label May 11, 2023
@mpoke mpoke added the S: Productivity Productivity: Developer tooling, infrastructure improvements enabling future growth label Sep 15, 2023
@mpoke mpoke moved this from 🏗 F3: InProgress to 🛑 F3: OnHold in Cosmos Hub Sep 19, 2023
@mpoke mpoke removed the automation label Mar 27, 2024
@mpoke mpoke closed this as completed Apr 4, 2024
@github-project-automation github-project-automation bot moved this from 🛑 F3: OnHold to 👍 F4: Assessment in Cosmos Hub Apr 4, 2024
@mpoke mpoke added the other: decayed Stale issues that need follow up from commentators. Were closed for inactivity label Apr 4, 2024
@mpoke mpoke moved this from 👍 F4: Assessment to ✅ Done in Cosmos Hub Apr 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
admin: epic An EPIC -- meta issue used to track a body of work other: decayed Stale issues that need follow up from commentators. Were closed for inactivity S: Productivity Productivity: Developer tooling, infrastructure improvements enabling future growth scope: testing Code review, testing, making sure the code is following the specification. type: feature-request New feature or request improvement type: tech-debt Slows down development in the long run
Projects
Status: ✅ Done
Development

No branches or pull requests

3 participants