Missing report with manually triggered job

[ctron]

I have a setup where I trigger a job manually, and that job actually checks out another repository that is runs for testing.

I still would like to see the report attached to the outcome of the github actions job. However, in the logs I can see:

2021-08-19 09:34:59 +0000 - publish-unit-test-results -  INFO - reading results.xml
2021-08-19 09:34:59 +0000 - publish -  INFO - publishing failure results for commit 31c89cd698a58ecda2c2ed82e27177d17413a7dd
2021-08-19 09:34:59 +0000 - publish -  INFO - creating check
2021-08-19 09:35:00 +0000 - publish -  INFO - there is no pull request for commit 31c89cd698a58ecda2c2ed82e27177d17413a7dd

And in most cases I don't see any results. Interestingly, sometimes I do.

[EnricoMi]

Can you run the action with:

with:
  log_level: DEBUG
  root_log_level: DEBUG

and point me to a run that works and one that fails (if in a public repo)?

[ralfstuckert]

Same problem here, I've set up a simple example with debugging enabled.
The run on push has a report, the manually started one hasn't.
Hope this helps

[EnricoMi]

@ralfstuckert it looks like the manually started run published its report to the push run and overwrote that one. Move your mouse to the 3 days ago to see the actual time:

The push run started 7:52 CEST and finished before 7:56 CEST, where its report was added (overwritten) 5:59 CEST.

The manual run started 5:58 CEST and finished 5:59 CEST, which fits with the report's publish time.

These check results (reports) are added via a GitHub API call, which does not know which workflow run does the call, so GitHub decides where to put those check results (probably the oldest one). See #12.

[ralfstuckert]

Showing the log's timestamps, I see the following:
Push:
Wed, 01 Sep 2021 05:52:28 GMT Current runner version: '2.280.3'

Manually:
Wed, 01 Sep 2021 05:58:36 GMT Current runner version: '2.280.3'

This fits the order in the history. Where did you see 7:52 CEST?

[EnricoMi]

The time is localized by the Web UI, where it says "3 days ago". Hover the mouse over that string and the tool tip says the time in local timezone, for me CEST (MESZ):

CEST is +2h from GMT, so those timestamps fit.

[MarcelWilson]

And in most cases I don't see any results. Interestingly, sometimes I do.

I have a very similar setup as @ralfstuckert; I noticed that if any new commits occur in the repo where the tests run then a new result will be created. But if no new commits exist in the repo, any subsequent test results overwrite the results from the one run (as @EnricoMi pointed out)

            -   name: Publish Unit Test Results
                if: always()
                uses: EnricoMi/[email protected]
                with:
                    files: ${{env.OUTPUT_DIR}}*.xml
                    report_individual_runs: "true"
                    hide_comments: off
                    check_run_annotations: all tests
                    comment_mode: create new

I interpreted this series of events (perhaps mistakenly) as; this publishes the results to the last run where a commit occurred. Is that what's happening here?

[MarcelWilson]

I also noticed that when the publish overwrites a previous run's results (instead of the proper run where the results were generated), there doesn't seem to be a similar problem occurring when artifacts are uploaded to that same run. (It's possible I just haven't seen it happen yet?) Granted, I know nothing of how the github API works but I wonder if there is a work-around there?

[EnricoMi]

Very interesting observation, I will try o reproduce this. If that is the case, then the work-around would be to add an artifact to your workflow.

[MarcelWilson]

I'm sorry, I was not clear. There isn't any sort of interaction between the artifacts and the results. I only noticed that artifacts don't appear to suffer from the same issue that results do. Which is a little puzzling to me; they both attach info to a run.

I have two runs. The first run happens just after a commit. The second run occurs after the first run is finished (no new commits). Both have artifacts but only the first run will have results. As you've noted the first run will contain the second run's results which overwrite the first's results. This pattern will continue to occur (new runs overwriting the first run's results) until a new commit happens, at which point the run immediately after the new commit will now hold a new set of results. And yet, every one of those runs will contain their own artifacts.

My curiosity is about what allows the artifacts to post to the proper run; whereas results can't? Does https://github.com/actions/upload-artifact not use the same API, perhaps?

[EnricoMi]

Hey @MarcelWilson, that is an excellent question.

I'll have a look into the artifact upload action to find out which API it uses and how it resolves the workflow run. I suspect that the artifact upload API is workflow run specific and the check run API is commit-centered, which internally picks a workflow run (the first per commit).

Thanks for sharing the observation.

[MarcelWilson]

I started to read up on check-run/check-suite and I suspect you're right.

Which has me wondering, could a workflow run add a commit prior to when the publish action step is triggered? Would this even work from an order of events standpoint?

While it might be slightly annoying to have essentially empty commits, it wouldn't be the end of the world in my case. (at least until github updates the API)

[MarcelWilson]

Automatically generating a commit at the start of the workflow run sorta works but there are some timing caveats. It's by no means a perfect work-around. Executing two runs concurrently can still end up with the results overwriting a different run depending on the timing. As long as runs are triggered in serial it will work.

[MarcelWilson]

As I read more about check-runs I'm starting to wonder if result handling actions (like this one, along with a few others) that utilize the 'Checks' api calls are even appropriate for workflow_dispatch situations such as mine and @ctron? As you alluded, this action is commit-centric whereas we're looking for run-centric handling. Am I mistaken in this thought?

Note: The Checks API only looks for pushes in the repository where the check suite or check run were created. Pushes to a branch in a forked repository are not detected and return an empty pull_requests array.

Secondly,
I'm a bit confused about the relationship between check-suite and check-run. I tried the workaround I mention above by adding a 'dummy' commit every time the workflow runs, which when run one-at-a-time appears to work fine. As an added precaution I even added concurrency: automated_regression to all of the workflows in the repo as a means of avoiding commit confusion. But when I went to test it out by clicking Run Workflow twice in succession, the 2nd run didn't contain the results even though it didn't execute until after 1st run completed.

I noticed the GITHUB_SHA value is being set when the workflow is queued not when it actually runs (since it is delayed by concurrency). Which means both run #1 and #2 start with the same sha but by the time #1 finishes it has created a new commit. So I figure I needed to get the latest sha at the start of #2 and pass that into into the action.

concurrency: automated_regression
jobs:
    use_our_action:
        runs-on: ubuntu-latest
        steps:
            -   name: Checkout
                uses: actions/[email protected]
                with:
                    ref: 'main'
            -   run: echo "GITHUB_SHA_LATEST=$(/usr/bin/git log -1 --format='%H')" >> $GITHUB_ENV
            # "dummy" commit occurs here
            # tests run here
            -   uses: EnricoMi/[email protected]
                with:
                    files: ${{inputs.OUTPUT_DIR}}*.xml
                    report_individual_runs: "true"
                    hide_comments: off
                    check_run_annotations: all tests
                    comment_mode: create new
                    commit: ${{env.GITHUB_SHA_LATEST}}

This doesn't appear to fix the issue and I don't really understand why.

When check-runs is called it uses the sha passed in via commit, right? According to the create check-suite and Update repository preferences for check suites documentation it appears a check-suite can be created automatically by two different events...

By default, check suites are automatically created when you create a check run.
By default, a check suite is automatically created each time code is pushed to a repository.

From what I can piece together from the above and jwgmeligmeyling's explanation, check-suite is our problem but I don't seem to understand how exactly. I figured the commit occurring in run #2 would create a new check-suite at the proper time thus allowing for results to work. There is some nuance to all of this that I must be missing.

[MarcelWilson]

Oh! there are multiple check-suites on the same SHA!! One gets created for SHA#1 by run #1, then another gets created for SHA#1 by run #2 right when I triggered it - regardless of the fact concurrency is being used.

When Run #1 goes to publish it creates a new check-run but since a check-suite was already created by that workflow it gets added to the existing suite (?). However, when run #2 gets to the publish part it's got a new SHA (since I told it to use the latest commit) which has the effect of creating a new check-suite.

So the problem with my attempt at cleverness appears to be foiled by the fact that when a workflow is triggered it sets the SHA and creates a check-suite too early for what I need. If only run #2 would set the the GITHUB_SHA when it actually runs rather than when it was triggered...

[EnricoMi]

I consider the original question as answered and the follow up question a duplicate of #12. Please reopen if that is not the case. Thanks for your thoughts.