Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simplify test results data #70

Merged
merged 7 commits into from
Sep 17, 2024
Merged

Conversation

ChrisC
Copy link
Contributor

@ChrisC ChrisC commented Sep 11, 2024

Closes #10 while also removing some additional IPC cruft in the way we were loading the test plans from the test files into the harness.

There was one request to in #10 to move the testId out of TestResult and into the TestPlanResult. Unfortunately, I didn't get to this, as it looks to me like it would require a bigger refactor of how we're loading the test plans via the CLI middleware. We don't actually get the test Id until later in the run process, when we JSON parse the file to run the tests. If you have a simple suggestion for making this change as requested, I'm all ears @jugglinmike ...

@ChrisC ChrisC marked this pull request as ready for review September 11, 2024 22:32
@ChrisC ChrisC marked this pull request as draft September 11, 2024 22:33
@jugglinmike
Copy link
Contributor

I just got all sorts of confused because we don't actually use the AriaATCIData.TestPlanResult type. It looks like we just emit a modified version of AriaATCIHost.TestPlan, instead. Maybe we could extend the modification logic because we have the testId at that moment. That doesn't seem like a huge refactoring to me, but maybe I've missed something important. In either case, I'm happy to punt that to another day if you don't mind creating an issue/Task for it.

@ChrisC
Copy link
Contributor Author

ChrisC commented Sep 12, 2024

@jugglinmike I got that testId property update in the last commits. Here's some sample TestPlanResult output as of c259f16:

{
    filepath: "test-06-navBackToSlider-voiceover_macos.collected.json",
    id: "navBackToSlider",
    log: [ ... ],
    results: [
      {
        capabilities: { ... },
        commands: [
          {
            command: "ctrl+opt+left ctrl+opt+left",
            response: "128\n            Red             slider\nRed\nYou are currently on a selectable text. ",
            assertions: [
              {
                expectation: "Role 'slider' is conveyed",
                verdict: null,
              },
              {
                expectation: "Name 'Red' is conveyed",
                verdict: null,
              },
              {
                expectation: "Value '128' is conveyed",
                verdict: null,
              },
              {
                expectation: "Orientation 'horizontal' is conveyed",
                verdict: null,
              },
              {
                expectation: "Minimum value '0' is conveyed",
                verdict: null,
              },
              {
                expectation: "Maximum value '255' is conveyed",
                verdict: null,
              },
              {
                expectation: "Screen reader switched from reading mode to interaction mode",
                verdict: null,
              },
            ],
          },
          { ... },
          { ... },
        ],
      },
    ],
  },

@ChrisC ChrisC marked this pull request as ready for review September 12, 2024 20:24
*/

/**
* Result from a single test in a test plan.
* @typedef AriaATCIData.TestResult
* @property {number} testId numeric id of a test in a test plan
* @property {string} testId id of a test in a test plan
* @property {number} presentationNumber numeric id of a test in a test plan
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that in the previous version we were inserting the string id as testId even though the type specifies that we were expecting a numeric id. I updated the types to reflect this. I'm assuming we want to stick with using the string version as the id and not the numeric presentationNumber...

@jugglinmike
Copy link
Contributor

The code looks good, and it functions as expected. Thanks for doing this, @ChrisC--these patches are really making the code base smaller and more coherent!

@jugglinmike jugglinmike merged commit 33b65e2 into w3c:main Sep 17, 2024
4 checks passed
@jugglinmike jugglinmike deleted the simplify-test-results-data branch September 17, 2024 20:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Test results data structure design
2 participants