Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set LiveTestCtestRegex to match CtestRegex in ci.yml across the repo, in a consistent format, following the template project. #6224

Open
ahsonkhan opened this issue Nov 15, 2024 · 2 comments
Labels
EngSys This issue is impacting the engineering system. MQ This issue is part of a "milestone of quality" initiative.

Comments

@ahsonkhan
Copy link
Member

ahsonkhan commented Nov 15, 2024

Tables' ci.yml was the only one that was inconsistent. All other instances within the repo have the LiveTestCtestRegex value match CtestRegex, so we fixed it in #6223, for consistency.

That said, some use wild-card (*), while others have it wrapped in quotes (""), while the rest leave it as plain-text:
https://github.com/search?q=repo%3AAzure%2Fazure-sdk-for-cpp%20LiveTestCtestRegex&type=code

We should consider picking a viable format, and apply it consistently, to avoid confusion.

A strawman proposal is to have it be unquoted, without the wild card, matching what we have in our template and core project:
azure-template
azure-core
azure-security-keyvault
azure-storage

Open question:
Would it make a difference to the set of tests that get matched/run if the regex value ended with a period . or not?

CtestRegex: azure-template

CtestRegex: "azure-security-keyvault.*"
LiveTestCtestRegex: "azure-security-keyvault.*"

CtestRegex: azure-storage
LineCoverageTarget: 81
BranchCoverageTarget: 47
LiveTestCtestRegex: azure-storage

@ahsonkhan ahsonkhan added EngSys This issue is impacting the engineering system. MQ This issue is part of a "milestone of quality" initiative. labels Nov 15, 2024
@ahsonkhan
Copy link
Member Author

cc @danieljurek in case you have any thoughts on the proposed format we could pick, for consistency

@danieljurek
Copy link
Member

danieljurek commented Nov 16, 2024

  • * is a special character in YAML so if used in a string the string should be wrapped in quotes
  • We use --tests-regex which only requires a match of the name to the regex (not a match of the entire name)

Two things to consider:

  1. You can probably remove the .* from the regexes and not change the tests that run. Which means you can also remove the quotes.
  2. If tests that depend on live services don't require special regexes and filtering you can probably consolidate everything into a single CtestRegex instead of having both CtestRegex and LiveTestCtestRegex .. This means that live tests should be able to (if they aren't already) run properly in the absence of live services (e.g. playback of recordings) or know when to skip

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
EngSys This issue is impacting the engineering system. MQ This issue is part of a "milestone of quality" initiative.
Projects
None yet
Development

No branches or pull requests

2 participants