Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ci(MM-63199): code coverage tracking #8642

Draft
wants to merge 77 commits into
base: main
Choose a base branch
from
Draft

ci(MM-63199): code coverage tracking #8642

wants to merge 77 commits into from
+244 −1

Conversation

rahimrahman
Copy link
Contributor

@rahimrahman rahimrahman commented Feb 28, 2025 •
edited
Loading

Summary

Introducing Code Coverage Report. Every time a PR is being merge to main, it will run test:coverage and save the code coverage summary as an artifact. At the same time, a github action cache will store the last run id. When a PR is created and anytime a new commit is pushed to the PR, the action will pull coverage summary from main (which is stored as an artifact), and then run the test-coverage on current code. Using a shell script, the two code coverage summaries will be compared in order to produce the report.

Currently, it has a 1% threshold, which can be modified in the compare-coverage.sh. So, if the current code changes went down by 1% (for either lines, statement, functions or branch), it will cause the action to post a warning along with the coverage report summary.

Currently, if the test fail, it will trigger a failure (this is the same as today). But if the coverage report dips below the threshold, then this should not fail the PR.

This can be improved in so many ways:

  • code coverage is showing code shifting even when doing non JS/TS code changes, but very small (0.0x)
  • getting coverage report summary every commit can be annoying?
    • solve by using a github action
  • block PR when you hit below 80% total.
  • block PR when hitting threshold, but allow to move forward with label

Ticket Link

https://mattermost.atlassian.net/browse/MM-63199

Checklist

  • Added or updated unit tests (required for all new features)
  • Has UI changes
  • Includes text changes and localization file updates
  • Have tested against the 5 core themes to ensure consistency between them.
  • Have run E2E tests by adding label E2E iOS tests for PR.

Device Information

This PR was tested on:

Screenshots

Release Note

@enahum
Copy link
Contributor

enahum commented Mar 10, 2025

Coverage Comparison Report

```+-----------------+------------+------------+-----------+| Metric | Current | Previous | Diff |+-----------------+------------+------------+-----------+| Lines | 85.25% | 85.25% | 0.00% || Statements | 85.13% | 85.13% | 0.00% || Branches | 70.81% | 70.83% | -0.02% || Functions | 83.22% | 83.22% | 0.00% |+-----------------+------------+------------+-----------+\n```

@enahum
Copy link
Contributor

enahum commented Mar 10, 2025

Coverage Comparison Report

+-----------------+------------+------------+-----------+
| Metric         | Current    | Previous   | Diff      |
+-----------------+------------+------------+-----------+
| Lines           |      85.25% |      85.25% |      0.00% |
| Statements      |      85.13% |      85.13% |      0.00% |
| Branches        |      70.81% |      70.81% |      0.00% |
| Functions       |      83.22% |      83.22% |      0.00% |
+-----------------+------------+------------+-----------+

@enahum
Copy link
Contributor

enahum commented Mar 10, 2025

Coverage Comparison Report

+-----------------+------------+------------+-----------+
| Metric         | Main       | This PR    | Diff      |
+-----------------+------------+------------+-----------+
| Lines           |      85.25% |      85.25% |      0.00% |
| Statements      |      85.13% |      85.13% |      0.00% |
| Branches        |      70.81% |      70.83% |      0.02% |
| Functions       |      83.22% |      83.22% |      0.00% |
+-----------------+------------+------------+-----------+

@enahum
Copy link
Contributor

enahum commented Mar 12, 2025

Coverage Comparison Report

+-----------------+------------+------------+-----------+
| Metric          | Main       | This PR    | Diff      |
+-----------------+------------+------------+-----------+
| Lines           |     85.22% |     85.25% |     0.03% |
| Statements      |     85.10% |     85.13% |     0.03% |
| Branches        |     70.72% |     70.76% |     0.04% |
| Functions       |     83.28% |     83.36% |     0.08% |
+-----------------+------------+------------+-----------+
| Total           |     81.08% |     81.12% |     0.04% |
+-----------------+------------+------------+-----------+

@github-actions GitHub Actions
Copy link

Hello world ! 👋

@github-actions GitHub Actions
Copy link

Coverage Comparison Report

+-----------------+------------+------------+-----------+
| Metric          | Main       | This PR    | Diff      |
+-----------------+------------+------------+-----------+
| Lines           |     85.22% |     85.25% |     0.03% |
| Statements      |     85.10% |     85.13% |     0.03% |
| Branches        |     70.72% |     70.76% |     0.04% |
| Functions       |     83.28% |     83.36% |     0.08% |
+-----------------+------------+------------+-----------+
| Total           |     81.08% |     81.12% |     0.04% |
+-----------------+------------+------------+-----------+

@github-actions GitHub Actions
Copy link

::error::Lines coverage has decreased by more than 1.0% (.03%)
::error::Statements coverage has decreased by more than 1.0% (.03%)
::error::Branches coverage has decreased by more than 1.0% (.03%)
::error::Functions coverage has decreased by more than 1.0% (.08%)

Coverage Comparison Report

+-----------------+------------+------------+-----------+
| Metric          | Main       | This PR    | Diff      |
+-----------------+------------+------------+-----------+
| Lines           |     85.22% |     85.25% |     0.03% |
| Statements      |     85.10% |     85.13% |     0.03% |
| Branches        |     70.72% |     70.75% |     0.03% |
| Functions       |     83.28% |     83.36% |     0.08% |
+-----------------+------------+------------+-----------+
| Total           |     81.08% |     81.12% |     0.04% |
+-----------------+------------+------------+-----------+

⚠️ Warning: One or more coverage metrics have decreased by more than 1.0%

@github-actions GitHub Actions
Copy link

Coverage Comparison Report

+-----------------+------------+------------+-----------+
| Metric          | Main       | This PR    | Diff      |
+-----------------+------------+------------+-----------+
| Lines           |     85.22% |     85.25% |     0.03% |
| Statements      |     85.10% |     85.13% |     0.03% |
| Branches        |     70.72% |     70.76% |     0.04% |
| Functions       |     83.28% |     83.36% |     0.08% |
+-----------------+------------+------------+-----------+
| Total           |     81.08% |     81.12% |     0.04% |
+-----------------+------------+------------+-----------+

⚠️ Warning: One or more coverage metrics have decreased by more than 1.0%

@github-actions GitHub Actions
Copy link

Coverage Comparison Report

+-----------------+------------+------------+-----------+
| Metric          | Main       | This PR    | Diff      |
+-----------------+------------+------------+-----------+
| Lines           |     85.22% |     85.25% |     0.03% |
| Statements      |     85.10% |     85.13% |     0.03% |
| Branches        |     70.72% |     70.76% |     0.04% |
| Functions       |     83.28% |     83.36% |     0.08% |
+-----------------+------------+------------+-----------+
| Total           |     81.08% |     81.12% |     0.04% |
+-----------------+------------+------------+-----------+

@enahum
Copy link
Contributor

enahum commented Mar 12, 2025 •
edited by github-actions bot
Loading

Coverage Comparison Report

Generated on March 13, 2025 at 16:26:41 UTC

+-----------------+------------+------------+-----------+
| Metric          | Main       | This PR    | Diff      |
+-----------------+------------+------------+-----------+
| Lines           |     85.25% |     85.20% |    -0.05% |
| Statements      |     85.13% |     85.08% |    -0.05% |
| Branches        |     70.76% |     70.70% |    -0.06% |
| Functions       |     83.36% |     83.25% |    -0.11% |
+-----------------+------------+------------+-----------+
| Total           |     81.12% |     81.05% |    -0.07% |
+-----------------+------------+------------+-----------+

@rahimrahman rahimrahman requested a review from larkox March 13, 2025 16:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Willyfrog Willyfrog Willyfrog approved these changes

@larkox larkox Awaiting requested review from larkox

Assignees
No one assigned
Labels
release-note
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@rahimrahman @enahum @Willyfrog @mm-cloud-bot