Candidate Results for Disclosure Navigation Menu for macOS and VoiceOver stale as of macOS 15.1.1

[mmoss]

Agenda item request:

When re-running the through the V24.10.10 of Disclosure Navigation Menu Example Test Plan with macOS 15.1.1 and Safari 18.1.1 the results are now different from the previous reports created with macOS 14.7 and Safari 18.0.1.

With macOS 15.1.1 and Safari 18.1.1 the aria-expanded attribute state change is not reported to a user when interacting with the menu button using Control+Option+Space, Space, or Enter.

How does the group handle discrepancies when they arise? Since the current Candidate results seem to no longer provide an accurate portrayal of the current interoperability for macOS 15.1.1 and Safari 18.1.1 should we proceed to have the current Candidate results reviewed (currently they're Not Started by Vispero or Apple) or rerun with macOS 15.1.1 and Safari 18.1.1 prior to their start of the review?

[css-meeting-bot]

The ARIA-AT Community Group just discussed issue 1174 on disclosure nav testing.

The full IRC log of that discussion

<jugglinmike> Topic: issue 1174 on disclosure nav testing
<jugglinmike> github: https://github.com//issues/1174
<jugglinmike> mmoss: The results appear to be stale at this point
<jugglinmike> mmoss: With current macOS 15.1.1, we're no longer getting the expanded attribute state change when it happens
<jugglinmike> mmoss: What is the process? Would we let the current results move through the process and eventually make their way to the APG site and then do a new round of testing?
<jugglinmike> Matt_King: This is related to the next topic
<jugglinmike> Matt_King: The report that we have out there right now... What is the Candidate Review status of these? Has either company approved them?
<jugglinmike> Matt_King: Actually, we have a discussion with Vispero ongoing about this one
<jugglinmike> Matt_King: Vispero has recognized some bugs in their behavior, and they are planning on making some changes
<jugglinmike> Matt_King: However, the specific thing that you're talking about was a VoiceOver behavior related to ARIA-expanded
<jugglinmike> mmoss: Right, some of the core functionality--the "musts" in the test plan
<jugglinmike> mmoss: At the time I tested it, they were met, but now, they are not met
<jugglinmike> Matt_King: So VoiceOver has regressed
<jugglinmike> mmoss: That's correct
<jugglinmike> Matt_King: So we could re-run this test plan for VoiceOver and update that. I think that would be a good thing to do before having more discussion with Apple on this plan. I don't think we've even touched on this plan with Apple
<jugglinmike> Matt_King: I believe we can just re-add this to the test queue for just VoiceOver.
<jugglinmike> Matt_King: We could replace that report by running with a later version of VoiceOver
<jugglinmike> Matt_King: Ideally (related to the next agenda item), when a new version of VoiceOver comes out, we will automatically re-run all Candidate and Recommended test plans and re-generate those results. We shouldn't have that problem when we have that capability
<jugglinmike> mmoss: I'm happy to run through the test plan again
<jugglinmike> Matt_King: Okay. We can add this to the test queue right now
<jugglinmike> IsaDC: I'll re-add the "disclosure navigation menu" test plan to the test queue, just for VoiceOver
<jugglinmike> Matt_King: Then we will re-run it
<jugglinmike> Matt_King: That's a good observation, mmoss. It's a bummer that there's been a regression