Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mutants Optimisations: big-packages, timeout multiplier, stacks-signer case #23

Merged
merged 8 commits into from
Feb 15, 2024

Conversation

ASuciuX
Copy link
Contributor

@ASuciuX ASuciuX commented Feb 14, 2024

Updates:

  1. split big packages into stackslib and stacks-node
  2. update cargo mutants to use specific version from github in order to include a timeout multiplier feature
  3. add a specific run case for stacks-signer

@wileyj
Copy link
Collaborator

wileyj commented Feb 14, 2024

can you share an example workflow using these changes?

@ASuciuX
Copy link
Contributor Author

ASuciuX commented Feb 14, 2024

@wileyj
Copy link
Collaborator

wileyj commented Feb 15, 2024

3. https://github.com/ASuciuX/stacks-core/actions/runs/7901429286?pr=57

if i understand you here, the link in item 3 is what is being proposed in this change?
can you explain if this specific error is expected? https://github.com/ASuciuX/stacks-core/actions/runs/7901429286/job/21571077307?pr=57#step:2:125

https://github.com/stacks-network/actions/blob/feat/mutation-testing/stacks-core/mutation-testing/output-pr-mutants/action.yml#L65

is being evaluated as:

  
  # Check and append the folders that should be uploaded to artifacts for the stacks-node package
  if  == "true"; then
    if  == "true"; then
      for i in {0..3}; do
        folders+=("mutants-shard-stacks-node-$i")
      done
    else
      folders+=("mutants-shard-stacks-node--1")
    fi
  fi

there may be an issue here: https://github.com/ASuciuX/stacks-core/blob/feat/test-stacks-signer/.github/workflows/pr-differences-mutants.yml#L207

edit: this result is quite strange. if you weren't sending an input, the composite workflow should fail because small_packages is required. but it's clear that something is being provided, and the action shows it's sending "true" (just like another input that is expanding the variable correctly). this is strange, i'm not sure what's going on here.

might be worth adding some echo "input: ${{ inputs.small_packages}} statements in the workflow bash script, or running the bash script in verbose mode to troubleshoot.

@wileyj
Copy link
Collaborator

wileyj commented Feb 15, 2024

Run stacks-network/actions/stacks-core/mutation-testing/output-pr-mutants@feat/mutation-testing
  with:
    stackslib_package: true
    shards_for_stackslib_package: false
    small_packages: true
    shards_for_small_packages: false
    stacks_signer: true
    ```
    

@ASuciuX
Copy link
Contributor Author

ASuciuX commented Feb 15, 2024

I've had false outputs set to those variables before. They added more if else statements, and I removed them so now there are only true outputs set, but I can change that if it would more appropriate to have it more verbose ( if there are no mutants on a specific package the output will be empty `` instead of false, which is still different than true ).

Also, the runs I've shared before were for specific testing cases from when I was developing them. This would be a final run:

PR ASuciuX/archived-stacks-core#58
Workflow run https://github.com/ASuciuX/stacks-core/actions/runs/7904568232
The workflow timed out as there were too many mutants on stackslib, after I finish fixing the code, I'll start adding skip-mutants headers for timeout functions so this should happen a lot less in the future ( there were a lot of mutants and most of them were timeout ).

@wileyj
Copy link
Collaborator

wileyj commented Feb 15, 2024

I've had false outputs set to those variables before. They added more if else statements, and I removed them so now there are only true outputs set, but I can change that if it would more appropriate to have it more verbose ( if there are no mutants on a specific package the output will be empty `` instead of false, which is still different than true ).

Also, the runs I've shared before were for specific testing cases from when I was developing them. This would be a final run:

PR ASuciuX/stacks-core#58 Workflow run https://github.com/ASuciuX/stacks-core/actions/runs/7904568232 The workflow timed out as there were too many mutants on stackslib, after I finish fixing the code, I'll start adding skip-mutants headers for timeout functions so this should happen a lot less in the future ( there were a lot of mutants and most of them were timeout ).

same issue here:
https://github.com/ASuciuX/stacks-core/actions/runs/7904568232/job/21587524440#step:2:77
https://github.com/ASuciuX/stacks-core/actions/runs/7904568232/job/21587524440#step:2:78

i'm comfortable merging this when there are specific unhandled script errors

@ASuciuX
Copy link
Contributor Author

ASuciuX commented Feb 15, 2024

same issue here: https://github.com/ASuciuX/stacks-core/actions/runs/7904568232/job/21587524440#step:2:77 https://github.com/ASuciuX/stacks-core/actions/runs/7904568232/job/21587524440#step:2:78

i'm comfortable merging this when there are specific unhandled script errors

I changed the action to send false if it's not true. Here's a run with it working: https://github.com/ASuciuX/stacks-core/actions/runs/7901429286/job/21625886921?pr=57#step:2:49

@wileyj wileyj merged commit 45654a9 into main Feb 15, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants