skip tests run as "non-packaged" (non-rpm) validation

[Vishwanatha-HD]

Added changes to skip the dotnet regular tests which are not required to run during cross build SDK validation.

[Vishwanatha-HD]

@omajid. please review my changes and merge it, if the changes are fine.

[omajid]

Can you add a comment to this line (sort of like what is in the previous line) to indicate why we are skipping this? Same goes for other json files.

[Vishwanatha-HD]

@omajid, I have added the comment for all the files to specify the reason for skipping the test. Please review the same. thanks..

[tmds]

This test is checking if we're applying a certain patch for RHEL9.
I wonder why this test does not fail on hydra's vmr builds (since they don't include the patch) 🤔
We should take a closer look.

cc @nicrowe00

[tmds]

// skip during cross-bld-sdk/tarball validation

@Vishwanatha-HD this doesn't describe why the test fails. I've added some questions as I like to understand better how the test fail in the "non-packaged" environment?

[Vishwanatha-HD]

Hi @tmds..
Omair and me had a discussion about the failures and please find his explanation below on why the tests fail when we try to validate the cross-build SDKs.. We mean "non-packaged" when we try to validate just the tarball and not the complete RPMs..

The errors fall into a few buckets:

Failing tests >>>>
-> publish-aspnet-selfcontained
-> publish-dotnet-selfcontained
-> restore-with-rid
-> openssl-alpn
-> system-openssl:

• The above tests fail because they expect to be testing a source-built SDK and a cross-built SDK is different enough
  • For example, doesn't know about rhel.9-s390x RID
  • The cross-built SDK is portable, and doesn't link to some system libraries that the tests want to poke through

Failing tests >>>>
-> bash-completion
-> man-pages
-> tools-in-path

• The above tests fail because they expect to be testing a packaged SDK, which has features like bash-completion enabled, which isn't a part of the .NET SDK tarball.

Failing tests >>>>
-> file-permissions
-> rsa-pkcs-openssl
-> sha1-validation

• The above tests fail because they are testing Red Hat-specific changes, such as whether
  omajid/dotnet-runtime@076687f
  has been applied to the SDK.

Please let me know if you need more info and we will able to provide them.. Thanks..

[tmds]

why the tests fail when we try to validate the cross-build SDKs..

fyi, in hydra CI we don't run the regular tests against the cross-built SDKs. The cross-built SDK jobs provide SDKs for native builds. And those builds run the tests.

For example, doesn't know about rhel.9-s390x RID

The cross-built SDK for .NET 9 gets built with a rid of linux-390x and provides assets under that rid. Rather than skipping these tests, they can run if they are using the appropriate SDK rid.

The cross-built SDK is portable, and doesn't link to some system libraries that the tests want to poke through

Can you be explicit about this in the comments? I don't think non-packaged conveys these tests are skipped because the build is portable. Perhaps you could use something else, like skipWhen: portable.

Failing tests >>>>

For these other tests, which should be ones that already have vmr-ci but where we are now adding now for the same reason: non-packaged.

It would be nice if we can avoid this duplication by either using vmr-ci or non-packaged, but not add them both.

For the managed-symbols-available test you can either also extract the symbols tarball so the test would pass, or add a new dedicated trait like: skipWhen: no-symbols.

Note that the test runner can be given multiple traits to skip, so you can set it to skip test matching any of: vmr-ci/non-packaged, portable, and no-symbols.

[Vishwanatha-HD]

@tmds and @omajid,
I have addressed all the code review comments and made the corresponding changes.
Please take a look and merge the CL if its fine.. thanks

[omajid]

This is great! Thank for you this really nice cleanup!

[Vishwanatha-HD]

@omajid, I took a look at your changes to run the testcase with SDK runtime-ids rather than the RPM ones.. The changes are looking good.. Thanks for doing this for me.. !!