Feat: mutation testing integrated flows #3

ASuciuX · 2023-12-11T12:00:17Z

WIP in order to merge

add CI workflows
fill this PR info
update docs
upload initial sync mutants output
- upload most of the sync without a few which are having problems on mac os

Description

Add 2 main new workflows for the mutation testing: filter-pr and logger. They interact with the stacks blockchain repo and keep track and filter the mutations for the PR and push actions there. They are intended to be run on master, develop and next.

Logger uses 2 smaller workflows:

upload-initial-cache if no cache is found on stacks repository. It will upload using the local output for the specific branch.
update-cache if a cache exists. It will take the old cache, update it with the output from the cargo mutants, which are run on the functions that are highlighted in the git diff.

It also includes the scripts that are run and the last synced outputs for the master, develop and next branches of the stacks blockchain
repository.

Motivation for change
To have mutations testing for better coverage, keeping track of untested functions and testing them as much as possible afterwards.
What was changed
I've added the workflows.
How does this impact application developers
They will need to have tests for the functions they are committing in the PR. If there are missing/timeout/unviable tests the CI workflow will crash. In the case that a specific function shouldn't be tested, it can be highlighted so by specifying in its header #[mutants::skip] or other attributes containing mutants::skip (e.g. #[cfg_attr(test, mutants::skip)]).
Link to relevant issues and documentation
Mutation Testing ( splitted into 2 PRs ) stacks-core#4089
https://github.com/sourcefrog/cargo-mutants
https://mutants.rs/

Type of Change

New feature

Does this introduce a breaking change?

No

Are documentation updates required?

No. I've included readme files as well.

needs initial output pushed into this branch before merging

…output-branch_name

Missing: - develop and next: stacks-node and stackslib for - master: blockstack-core and stacks-node

jbencin · 2024-01-23T18:35:25Z

mutation-testing/filter-pr/action.yml

+      run: |
+        mv git.diff ./actions-repo/mutation-testing/shell-scripts/
+
+    - uses: Swatinem/rust-cache@v2


I believe we have been pinning external actions by hash to avoid possible breakage or supply chain attacks

jbencin · 2024-01-23T18:44:18Z

mutation-testing/filter-pr/action.yml

+      id: git_checkout_stacks_core
+      uses: actions/checkout@v3
+      with:
+        fetch-depth: 0


fetch-depth: 0 fetches entire history. Can we optimize by only fetching the branches (or commits) we are comparing?

jbencin · 2024-01-23T19:29:07Z

mutation-testing/shell-scripts/append-match-package.sh

+## search in that specific folder on all 4 files
+## if it is matchy, remove it from that file
+## based on the file it was taken from, append it to the same file in the STABLE folder
+


It's considered good practice to add set -euo pipefail to all bash scripts. This will cause cause failures on certain errors, rather continuing to run and producing bad results

jbencin · 2024-01-23T19:29:22Z

mutation-testing/shell-scripts/append-match-package.sh

+# the append-match-package.sh
+## goes through each line in the output and based on the package ( first element before /)
+### verifies the line with the other lines in that specific folder
+#### in our case folder_name == package_name


Why the extra #s?

jbencin · 2024-01-23T19:30:53Z

mutation-testing/shell-scripts/create-stable.sh

+
+# removes everything except .txt files
+
+#!/bin/bash


This needs to be the first line in the file

jbencin · 2024-01-23T19:31:41Z

mutation-testing/shell-scripts/create-stable.sh

+#!/bin/bash
+
+# moves to mutation-testing folder
+cd ../packages-output


Shellcheck warning:

cd ../packages-output ^-------------------^ SC2164 (warning): Use 'cd ... || exit' or 'cd ... || return' in case cd fails.

jbencin · 2024-01-23T19:33:11Z

mutation-testing/shell-scripts/modular-mutants-run.sh

@@ -0,0 +1,41 @@
+#!/bin/bash


This script needs some comments

jbencin · 2024-01-23T19:34:21Z

mutation-testing/shell-scripts/remove-deleted-functions-from-output.sh

+    impl_block="${impl_block%%\ {*}"
+    impl_block="<impl$impl_block>" # add 'impl' word, '<' and '>' characters
+    impl_block=$(echo "$impl_block" | sed -E 's/(^[^ ]*impl)[^ ]*/\1/') # remove everything after 'impl', until the next space
+    impl_block=$(echo "$impl_block" | sed 's/\(<[^<]*\)[^ ]* /\1 /') # remove everything starting with '<' and ending at the next space, from the second word


Shellcheck warning:

impl_block=$(echo "$impl_block" | sed 's/$<[^<]*$[^ ]* /\1 /') # remove everything starting with '<' and ending at the next space, from the second word ^-- SC2001 (style): See if you can use ${variable//search/replace} instead.

jbencin · 2024-01-23T19:39:14Z

mutation-testing/update-cache/action.yml

+      shell: bash
+      run: |
+        if [ -f ./packages-output/last_commit_hash.txt ]; then
+          cargo mutants --no-shuffle -j 2 -vV --in-diff ./scripts/git.diff --output temp/


Why is the diff in the ./scripts directory?

Also, the first half of this file looks the same as filter-pr. Why not use that?

jbencin · 2024-01-23T19:57:12Z

mutation-testing/filter-pr/action.yml

+
+    - name: Remove deleted file lines from git.diff file
+      shell: bash
+      run: ./remove-deleted-file-lines.sh


Is this necessary? Shouldn't cargo mutants be able to read a diff and know test deleted functions?

wileyj · 2024-01-25T19:40:02Z

I'm not a fan of keeping the raw text output in this repo in that initial-output folder - are there alternatives you've explored for keeping this data, and more importantly keeping this data up to date?

for example, some process should be periodically updating this output, correct? would a non-versioned storage location work better or is the versioning imperative for this data?

ASuciuX · 2024-01-25T23:10:52Z

Yes, I'll save it somewhere else and we could remove it from here afterwards. Also, as a non stable output i was thinking of the cache as it is used in the actions workflow process, but not sure what's the best stable place to keep it. ( updating a github repo, something automated )

wileyj · 2024-01-25T23:23:17Z

for now it's fine, we should ask others for alternatives though. it just feels wrong to keep that type of data in this specific repo one idea is the location of the data shouldn't matter... maybe we add an input with a download url so it's dependent on any specific service (like s3, github etc)

…

On Thu, Jan 25, 2024 at 15:11 ASuciuX ***@***.***> wrote: Yes, I'll save it somewhere else and we could remove it from here soon. Also, as a non stable output i was thinking of the cache as it is used in the actions workflow process, but not sure what's the best stable place to keep it. ( updating a github repo, something automated ) — Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAVXIHE75VOK6FCFRCES643YQLRANAVCNFSM6AAAAABAPUMPJGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJRGE2DENZVGQ> . You are receiving this because you commented.Message ID: ***@***.***>

ASuciuX added 2 commits December 11, 2023 01:14

feat: add ci workflows

227c667

needs initial output pushed into this branch before merging

feat: upload artifact to fetch data easily & rename cache to mutants-…

33a3cbb

…output-branch_name

ASuciuX mentioned this pull request Dec 11, 2023

Mutation Testing ( splitted into 2 PRs ) stacks-network/stacks-core#4089

Closed

6 tasks

ASuciuX added 4 commits December 11, 2023 23:27

feat: docs readme mutations

ed270a1

feat: add retry to cache/artifact save in case of failure

f47472d

fix: regex matching for mutants v12.3

9c2ab75

feat: most packages output for initial upload

0831a07

Missing: - develop and next: stacks-node and stackslib for - master: blockstack-core and stacks-node

This was referenced Dec 15, 2023

Mutation testing - pr differences stacks-network/stacks-core#4178

Merged

Mutation testing - logger tracking missing/timeout/unviable tests stacks-network/stacks-core#4179

Closed

jbencin reviewed Jan 23, 2024

View reviewed changes

mutation-testing/shell-scripts/modular-mutants-run.sh

@@ -0,0 +1,41 @@

#!/bin/bash

Copy link

Contributor

jbencin Jan 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This script needs some comments

jbencin reviewed Jan 23, 2024

View reviewed changes

ASuciuX marked this pull request as draft February 14, 2024 22:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat: mutation testing integrated flows #3

Feat: mutation testing integrated flows #3

ASuciuX commented Dec 11, 2023 •

edited

Loading

jbencin Jan 23, 2024

jbencin Jan 23, 2024

jbencin Jan 23, 2024

jbencin Jan 23, 2024

jbencin Jan 23, 2024

jbencin Jan 23, 2024

jbencin Jan 23, 2024

jbencin Jan 23, 2024

jbencin Jan 23, 2024

jbencin Jan 23, 2024

jbencin Jan 23, 2024

wileyj commented Jan 25, 2024

ASuciuX commented Jan 25, 2024 •

edited

Loading

wileyj commented Jan 25, 2024 via email

Feat: mutation testing integrated flows #3

Are you sure you want to change the base?

Feat: mutation testing integrated flows #3

Conversation

ASuciuX commented Dec 11, 2023 • edited Loading

Description

Type of Change

Does this introduce a breaking change?

Are documentation updates required?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wileyj commented Jan 25, 2024

ASuciuX commented Jan 25, 2024 • edited Loading

wileyj commented Jan 25, 2024 via email

ASuciuX commented Dec 11, 2023 •

edited

Loading

ASuciuX commented Jan 25, 2024 •

edited

Loading