Codec summary report #566

JenySadadia · 2024-08-28T07:34:55Z

Generate codec summary report and email notifications for the same.

Created on top of #565 to use common.j2.

JenySadadia · 2024-08-28T10:54:48Z

Hello @denisyuji @helen-fornazier
This would generate an email notification like the below:

Subject: KernelCI report for fluster tests: [email protected]

REVISION

    Commit
        name: v6.11-rc4
        hash: 3d5f968a177d468cd13568ef901c5be84d83d32b
     Checked out from
        https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master


FLUSTER TESTS


      sc7180-trogdor-kingoftown:
      -v4l2-decoder-conformance-av1:                  passed: 0, failed: 0
      -v4l2-decoder-conformance-av1-chromium-10bit:   passed: 0, failed: 0
      -v4l2-decoder-conformance-h264:                 passed: 0, failed: 0
      -v4l2-decoder-conformance-h264-frext:           passed: 0, failed: 0
      -v4l2-decoder-conformance-h265:                 passed: 1, failed: 1
      -v4l2-decoder-conformance-vp8:                  passed: 0, failed: 0
      -v4l2-decoder-conformance-vp9:                  passed: 0, failed: 0

      sc7180-trogdor-lazor-limozeen:
      -v4l2-decoder-conformance-av1:                  passed: 0, failed: 0
      -v4l2-decoder-conformance-av1-chromium-10bit:   passed: 0, failed: 0
      -v4l2-decoder-conformance-h264:                 passed: 0, failed: 0
      -v4l2-decoder-conformance-h264-frext:           passed: 0, failed: 0
      -v4l2-decoder-conformance-h265:                 passed: 0, failed: 0
      -v4l2-decoder-conformance-vp8:                  passed: 0, failed: 0
      -v4l2-decoder-conformance-vp9:                  passed: 0, failed: 2

See complete and up-to-date report at:

    https://kcidb.kernelci.org/d/revision/revision?orgId=1&var-git_commit_hash=3d5f968a177d468cd13568ef901c5be84d83d32b&var-patchset_hash=


Tested-by: kernelci.org bot <[email protected]>

Thanks,
KernelCI team

Please let me know your feedback on this.
Note: I have generated report on dummy data so you won't be able to see these results on KCIDB dashboard.

helen-fornazier · 2024-08-28T15:33:03Z

@JenySadadia thanks for this work.

I was just wondering if it wouldn't be better to point to this instead:

https://fresh2-kcidb-grafana-m6io3uhhiq-uc.a.run.app/d/bdviluqy6gv7kc/codecs?orgId=1&var-platform=All&%20var-datasource=default&var-git_commit_hash=3ec3f5fc4a91e389ea56b111a73d97ffc94f19c6

I also noticed that it takes some time for the tests to be run after the checkout. For instance, by the time of this post we still don't have tests for the last checkout, so I was wondering how you handle this.

JenySadadia · 2024-08-29T05:22:38Z

@JenySadadia thanks for this work.

I was just wondering if it wouldn't be better to point to this instead:

https://fresh2-kcidb-grafana-m6io3uhhiq-uc.a.run.app/d/bdviluqy6gv7kc/codecs?orgId=1&var-platform=All&%20var-datasource=default&var-git_commit_hash=3ec3f5fc4a91e389ea56b111a73d97ffc94f19c6

Sure, I'll update the report to point it to the new dashboard.

I also noticed that it takes some time for the tests to be run after the checkout. For instance, by the time of this post we still don't have tests for the last checkout, so I was wondering how you handle this.

We are sending out notifications after 3 hours when a revision create/update event is received. Tests will be executed in that waiting period.

JenySadadia · 2024-08-29T05:46:35Z

https://fresh2-kcidb-grafana-m6io3uhhiq-uc.a.run.app/d/bdviluqy6gv7kc/codecs?orgId=1&var-platform=All&%20var-datasource=default&var-git_commit_hash=3ec3f5fc4a91e389ea56b111a73d97ffc94f19c6

Do you consider MISS/ERROR/SKIP tests in total count?

JenySadadia · 2024-08-29T09:49:50Z

Updated templates to display compatible[0] instead of platform.

helen-fornazier · 2024-08-29T11:02:12Z

https://fresh2-kcidb-grafana-m6io3uhhiq-uc.a.run.app/d/bdviluqy6gv7kc/codecs?orgId=1&var-platform=All&%20var-datasource=default&var-git_commit_hash=3ec3f5fc4a91e389ea56b111a73d97ffc94f19c6

Do you consider MISS/ERROR/SKIP tests in total count?

Yes

JenySadadia · 2024-08-30T10:41:12Z

Report v2:

use compatible string instead of platform
display total count
use link to the new grafana dashboard

REVISION

    Commit
        name: v6.11-rc4
        hash: 3d5f968a177d468cd13568ef901c5be84d83d32b
     Checked out from
        https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master


FLUSTER TESTS

      google,kingoftown:
      -v4l2-decoder-conformance-av1:                  passed: 0, failed: 0, total: 0
      -v4l2-decoder-conformance-av1-chromium-10bit:   passed: 0, failed: 0, total: 0
      -v4l2-decoder-conformance-h264:                 passed: 0, failed: 0, total: 0
      -v4l2-decoder-conformance-h264-frext:           passed: 0, failed: 0, total: 0
      -v4l2-decoder-conformance-h265:                 passed: 1, failed: 1, total: 2
      -v4l2-decoder-conformance-vp8:                  passed: 0, failed: 0, total: 0
      -v4l2-decoder-conformance-vp9:                  passed: 0, failed: 0, total: 0

      google,lazor-rev5-sku5:
      -v4l2-decoder-conformance-av1:                  passed: 0, failed: 0, total: 0
      -v4l2-decoder-conformance-av1-chromium-10bit:   passed: 0, failed: 0, total: 0
      -v4l2-decoder-conformance-h264:                 passed: 0, failed: 0, total: 0
      -v4l2-decoder-conformance-h264-frext:           passed: 0, failed: 0, total: 0
      -v4l2-decoder-conformance-h265:                 passed: 0, failed: 0, total: 0
      -v4l2-decoder-conformance-vp8:                  passed: 0, failed: 0, total: 0
      -v4l2-decoder-conformance-vp9:                  passed: 0, failed: 2, total: 3


See complete and up-to-date report at:

    https://fresh2-kcidb-grafana-m6io3uhhiq-uc.a.run.app/d/bdviluqy6gv7kc/codecs?orgId=1&var-platform=All&var-datasource=default&var-git_commit_hash=3d5f968a177d468cd13568ef901c5be84d83d32b


Tested-by: kernelci.org bot <[email protected]>

Thanks,
KernelCI team

JenySadadia · 2024-08-30T12:14:10Z

@helen-fornazier @denisyuji
Once the report looks good to you, I can work on improving formatting e.g. table-like view for stats if required.

denisyuji · 2024-08-30T16:58:52Z

@JenySadadia , this looks good, good work!
It would be great if we can also add some information about the regression. Like it is produced by the summary script:

$ git clone https://github.com/kernelci/kernelci-pipeline.git && cd kernelci-pipeline
$ touch .env
$ sed -i 's/api_config = "docker-host"/api_config = "staging"/' config/kernelci.toml
$ docker compose run result_summary --preset=summary-v4l2-decoder-conformance-regressions --last-updated-from=$(date --date='yesterday' --rfc-3339=date)

The result is displayed at data/output/v4l2-decoder-conformance-regressions.html

JenySadadia · 2024-09-02T05:26:28Z

@JenySadadia , this looks good, good work! It would be great if we can also add some information about the regression. Like it is produced by the summary script:
$ git clone https://github.com/kernelci/kernelci-pipeline.git && cd kernelci-pipeline
$ touch .env
$ sed -i 's/api_config = "docker-host"/api_config = "staging"/' config/kernelci.toml
$ docker compose run result_summary --preset=summary-v4l2-decoder-conformance-regressions --last-updated-from=$(date --date='yesterday' --rfc-3339=date)
The result is displayed at data/output/v4l2-decoder-conformance-regressions.html

At the moment, maestro is not sending regressions to KCIDB. Hence, I don't think that is possible along with this report. However, we can send this information directly from maestro pipeline.
@r-c-n Could you please point me out if there is another way that I'm not aware of?

spbnick

Thank you, Jeny!

I added some inline comments, a draft amendment, and also a change to raise the limit on notification body size. I forgot we had that, and that was where the ✂️ came from 🙈

kcidb/oo/__init__.py

kcidb/templates/fluster_test.j2

kcidb/oo/__init__.py

spbnick · 2024-09-10T09:46:18Z

Please move your logic for fluster tests from Python to Jinja template code, either the way I did it, or whatever way you prefer. Do not keep my DRAFT: commit, but make sure to keep the max body size amendment, as a separate commit.

Thank you!

Also, I think we need to make the notification generation a bit more flexible, so you could prepare data to be rendered inside the subscription, and then pass it to Jinja.

JenySadadia · 2024-09-10T13:10:44Z

Please move your logic for fluster tests from Python to Jinja template code, either the way I did it, or whatever way you prefer. Do not keep my DRAFT: commit, but make sure to keep the max body size amendment, as a separate commit.

As you have contributed to the PR by converting python code to Jinja2, I would like to keep your name in commits with Co-authored-by tag. Thanks again for all the help.

Templates for generating fluster tests report. Signed-off-by: Jeny Sadadia <[email protected]> Co-authored-by: Nikolai.Kondrashov <[email protected]>

Script for generating email notifications for codec summary report. Signed-off-by: Jeny Sadadia <[email protected]>

JenySadadia · 2024-09-11T10:52:42Z

Report v3:

Subject: KernelCI report for fluster tests: [email protected]
REVISION

    Commit
        name: v6.11-rc4
        hash: 3d5f968a177d468cd13568ef901c5be84d83d32b
     Checked out from
        https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master


FLUSTER TESTS

      google,kingoftown:
       av1:                  ✅ 0  ❌ 0  Σ 0
       av1-chromium-10bit:   ✅ 0  ❌ 0  Σ 0
       h264:                 ✅ 0  ❌ 0  Σ 0
       h264-frext:           ✅ 0  ❌ 0  Σ 0
       h265:                 ✅ 1  ❌ 1  Σ 2
       vp8:                  ✅ 0  ❌ 0  Σ 0
       vp9:                  ✅ 0  ❌ 0  Σ 0

      google,lazor-rev5-sku5:
       av1:                  ✅ 0  ❌ 0  Σ 0
       av1-chromium-10bit:   ✅ 0  ❌ 0  Σ 0
       h264:                 ✅ 0  ❌ 0  Σ 0
       h264-frext:           ✅ 0  ❌ 0  Σ 0
       h265:                 ✅ 0  ❌ 0  Σ 0
       vp8:                  ✅ 0  ❌ 0  Σ 0
       vp9:                  ✅ 0  ❌ 2  Σ 3

See complete and up-to-date report at:

    https://fresh2-kcidb-grafana-m6io3uhhiq-uc.a.run.app/d/bdviluqy6gv7kc/codecs?orgId=1&var-platform=All&var-datasource=default&var-git_commit_hash=3d5f968a177d468cd13568ef901c5be84d83d32b


Tested-by: kernelci.org bot <[email protected]>

Thanks,
KernelCI team

Update test to increate body text size according to increased maximum allowed size for notification body to keep it over limit length. Signed-off-by: Jeny Sadadia <[email protected]>

spbnick

LGTM!

I left one non-blocking comment inline, for later.

Thank you, Jeny!

spbnick · 2024-09-12T11:12:41Z

kcidb/templates/fluster_test.j2

+        "gstreamer_h264_frext":    "h264-frext",
+        "gstreamer_h265":          "h265",
+        "gstreamer_vp8":           "vp8",
+        "gstreamer_vp9":           "vp9"


Looks like we don't need this map, we can have a list of current keys instead, and just cut off gstreamer_ from the names to get the current values. But we can do that separately next time. Just spotted this.

JenySadadia force-pushed the codec-summary-report branch from 2e040bd to 18e5158 Compare August 28, 2024 10:17

JenySadadia had a problem deploying to staging August 28, 2024 10:32 — with GitHub Actions Failure

JenySadadia requested a review from spbnick August 28, 2024 10:53

JenySadadia marked this pull request as ready for review August 28, 2024 10:53

JenySadadia requested review from denisyuji and helen-fornazier August 28, 2024 10:58

JenySadadia force-pushed the codec-summary-report branch from 18e5158 to dbe2f2e Compare August 28, 2024 11:41

JenySadadia had a problem deploying to staging August 28, 2024 11:56 — with GitHub Actions Failure

JenySadadia force-pushed the codec-summary-report branch from dbe2f2e to 7d6308f Compare August 29, 2024 05:32

JenySadadia had a problem deploying to staging August 29, 2024 05:47 — with GitHub Actions Failure

JenySadadia force-pushed the codec-summary-report branch 2 times, most recently from 5f9cd1a to 304fc48 Compare August 29, 2024 09:49

JenySadadia had a problem deploying to staging August 29, 2024 10:04 — with GitHub Actions Failure

JenySadadia force-pushed the codec-summary-report branch 2 times, most recently from 14f7de3 to cf7657e Compare August 30, 2024 10:50

JenySadadia had a problem deploying to staging August 30, 2024 11:05 — with GitHub Actions Failure

JenySadadia had a problem deploying to staging August 30, 2024 11:54 — with GitHub Actions Failure

JenySadadia mentioned this pull request Sep 5, 2024

Codec summary report #568

Open

JenySadadia force-pushed the codec-summary-report branch from cf7657e to 4045777 Compare September 6, 2024 07:26

JenySadadia had a problem deploying to staging September 6, 2024 07:41 — with GitHub Actions Failure

denisyuji approved these changes Sep 9, 2024

View reviewed changes

spbnick requested changes Sep 10, 2024

View reviewed changes

kcidb/oo/__init__.py Outdated Show resolved Hide resolved

kcidb/templates/fluster_test.j2 Outdated Show resolved Hide resolved

kcidb/templates/fluster_test.j2 Outdated Show resolved Hide resolved

kcidb/oo/__init__.py Outdated Show resolved Hide resolved

Jeny Sadadia and others added 2 commits September 11, 2024 16:20

kcidb/templates: add templates for creating codec report

1edc5ca

Templates for generating fluster tests report. Signed-off-by: Jeny Sadadia <[email protected]> Co-authored-by: Nikolai.Kondrashov <[email protected]>

Codec summary subscription script

3e67c88

Script for generating email notifications for codec summary report. Signed-off-by: Jeny Sadadia <[email protected]>

JenySadadia force-pushed the codec-summary-report branch from 6cd2216 to 5cbbadc Compare September 11, 2024 10:50

spbnick and others added 2 commits September 11, 2024 17:19

monitor: Increase body length limit to 64KB

6244c43

kcidb/monitor/test_output: update test with over limit body

b930d10

Update test to increate body text size according to increased maximum allowed size for notification body to keep it over limit length. Signed-off-by: Jeny Sadadia <[email protected]>

JenySadadia force-pushed the codec-summary-report branch from 5cbbadc to b930d10 Compare September 11, 2024 11:52

JenySadadia had a problem deploying to staging September 11, 2024 12:07 — with GitHub Actions Failure

spbnick approved these changes Sep 12, 2024

View reviewed changes

spbnick merged commit ed1b8b9 into kernelci:main Sep 12, 2024
5 of 6 checks passed

JenySadadia deleted the codec-summary-report branch September 13, 2024 05:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Codec summary report #566

Codec summary report #566

JenySadadia commented Aug 28, 2024 •

edited

Loading

JenySadadia commented Aug 28, 2024 •

edited

Loading

helen-fornazier commented Aug 28, 2024

JenySadadia commented Aug 29, 2024 •

edited

Loading

JenySadadia commented Aug 29, 2024

JenySadadia commented Aug 29, 2024

helen-fornazier commented Aug 29, 2024

JenySadadia commented Aug 30, 2024 •

edited

Loading

JenySadadia commented Aug 30, 2024

denisyuji commented Aug 30, 2024

JenySadadia commented Sep 2, 2024

spbnick left a comment

spbnick commented Sep 10, 2024

JenySadadia commented Sep 10, 2024

JenySadadia commented Sep 11, 2024

spbnick left a comment

spbnick Sep 12, 2024

Codec summary report #566

Codec summary report #566

Conversation

JenySadadia commented Aug 28, 2024 • edited Loading

JenySadadia commented Aug 28, 2024 • edited Loading

helen-fornazier commented Aug 28, 2024

JenySadadia commented Aug 29, 2024 • edited Loading

JenySadadia commented Aug 29, 2024

JenySadadia commented Aug 29, 2024

helen-fornazier commented Aug 29, 2024

JenySadadia commented Aug 30, 2024 • edited Loading

JenySadadia commented Aug 30, 2024

denisyuji commented Aug 30, 2024

JenySadadia commented Sep 2, 2024

spbnick left a comment

Choose a reason for hiding this comment

spbnick commented Sep 10, 2024

JenySadadia commented Sep 10, 2024

JenySadadia commented Sep 11, 2024

spbnick left a comment

Choose a reason for hiding this comment

spbnick Sep 12, 2024

Choose a reason for hiding this comment

JenySadadia commented Aug 28, 2024 •

edited

Loading

JenySadadia commented Aug 28, 2024 •

edited

Loading

JenySadadia commented Aug 29, 2024 •

edited

Loading

JenySadadia commented Aug 30, 2024 •

edited

Loading