pkg/_macros/test/executor/executor_test is failing

[a-siva]

The tests

pkg/_macros/test/executor/executor_test RuntimeError (expected Pass)

are failing on configurations

unittest-asserts-release-win-arm64
unittest-asserts-release-win-x64

[a-siva]

//cc @jakemac53

[dart-github-bot]

area-test, type-bug

The pkg/_macros/test/executor/executor_test tests are failing on Windows ARM64 and x64 configurations, resulting in a RuntimeError instead of the expected Pass.

[devoncarew]

around area-test might need to be made more explicit that it should prefer the affected functional area, or otherwise weighted down

I agree with that (and thanks for the feedback on the meta / triaging bot issue). Here, is it area-vm tests that are failing? We don't have a specific area for macros.; we're using area-language for non-implementation discussion issues.

We could improve the prompt for these 'test' cases. We're also not using the tuned model yet, just explicit directions in a prompt; switching to the tuned model should also improve accuracy here.

[jakemac53]

The area should be area-pkg afaik (the tests in question are in pkg/_macros).

[jakemac53]

@a-siva do you have a failure you can link me? I did recently land a fix for this test, and I am not sure how to validate if it was the same error or something different etc.

[a-siva]

Here is the first time the failure showed up logs
(I believe it was a Timeout initially and then turned into a RuntimeError later).

[devoncarew]

The area should be area-pkg afaik (the tests in question are in pkg/_macros).

Yes, but area-pkg is not a great area- itself. It's not triaged - there's no team that does secondary triage on it or ultimately has responsibility for items in that area.

[jakemac53]

Ok yeah looking at the log, that is the one that should be fixed. Is this still failing?

[a-siva]

It was failing on the windows bots until I created this issue and approved the failure

[jakemac53]

Moved this to the "no milestone/release" bucket. It isn't directly tied to any release and I haven't discovered any reasonable way to try and investigate the failure. I intend to just leave this as open tech debt for some time. There are no actual affected users, just a failing test in an environment that I can't reproduce.

[a-siva]

This failure was seen again here https://ci.chromium.org/ui/p/dart/builders/ci.sandbox/pkg-win-release/29263/overview

[aam]

and again https://dart-ci.appspot.com/log/pkg-win-release/unittest-asserts-release-win-x64/29419/pkg/_macros/test/executor/executor_test

[jakemac53]

My assumption was that this test can just be marked failing/flaky and not cause people friction. Is that not the case? If you chimed in here, is that because you were blocked or otherwise affected by the failure?

[aam]

Yes, the test failure causes friction as it turns the bot on go/darto dart waterfall red

If the test is doomed to flaky failures such that test deflaker can't handle it, it could be marked as Skip in the status file or even deleted altogether.

[jakemac53]

It is only doing this on windows (that I have seen) - we do want the coverage on other platforms. So possibly marking it as skipped on windows is best?

[aam]

It is only doing this on windows (that I have seen) - we do want the coverage on other platforms. So possibly marking it as skipped on windows is best?

Yeah, something like the following should do that

diff --git a/pkg/pkg.status b/pkg/pkg.status
index f129396f35c..daf26bf3345 100644
--- a/pkg/pkg.status
+++ b/pkg/pkg.status
@@ -243,6 +243,7 @@ vm_snapshot_analysis/test/*: SkipByDesign # Only meant to run on vm
 [ $system == windows ]
 front_end/test/fasta/bootstrap_test: Skip # Issue 31902
 front_end/test/incremental_dart2js_load_from_dill_test: Pass, Slow
+_macros/test/executor/executor_test: Skip # dartbug.com/56002
 vm_service/test/private_rpcs/dev_fs_http_put_test: Skip # Windows disallows "?" in paths
 vm_service/test/private_rpcs/dev_fs_http_put_weird_char_test: Skip # Windows disallows "\r" in paths
 vm_service/test/private_rpcs/dev_fs_weird_char_test: Skip # Windows disallows "\r" in paths

[jakemac53]

https://dart-review.googlesource.com/c/sdk/+/379240