Publishes to random workflow

[EnricoMi]

When there are multiple GitHub workflows for one commit, the action publishes the commit status to a random workflow. It should be the workflow that contains the action. This is due to GitHub's API not allowing to specify which workflow to add the check run to.

Both REST and GraphQL APIs do not allow to specify the workflow:

• https://developer.github.com/v3/checks/runs/#create-a-check-run
• https://developer.github.com/v4/mutation/createchecksuite/

The github.run_id workflow variable provides the check run id where the action runs.

Update 2022-05-14:
With GitHub introducing "job summary", the action now additionally publishes the results at the summary page of the workflow that runs the publish action. From there, a link to the check annotations is provided if failures exists: https://github.com/EnricoMi/publish-unit-test-result-action/tree/v1.35#github-actions-job-summary. However, this is not a proper fix for this issue.

Only Github can fix this issue. Please see this discussion: https://github.com/orgs/community/discussions/24616

Checks can be disabled with check_run: false.

[airadier]

This seems to be related with this issue we are suffering for scheduled checks.

We are now using your latest 1.5 release, where I see support for all kind of events was added. However, it is not working as expected for "schedule" event type... as you mentioned, the check runs are published with the commit ID:

publish-unit-test-result-action/githubext/Repository.py

Line 24 in 075b72a

def create_check_run(self: Repository,

and unfortunately that check gets added to our latest action run for the "push" event

but not for the scheduled event we run hourly:

This is the output from the step:

Run EnricoMi/[email protected]
/usr/bin/docker run --name e8eba344359fc6689913976734_0d4603 --label 179394 --workdir /github/workspace --rm 
-e pythonLocation -e LD_LIBRARY_PATH -e INPUT_GITHUB_TOKEN -e INPUT_CHECK_NAME -e INPUT_FILES 
-e INPUT_REPORT_INDIVIDUAL_RUNS -e INPUT_DEDUPLICATE_CLASSES_BY_FILE_NAME -e INPUT_HIDE_COMMENTS 
-e INPUT_COMMENT_ON_PR -e INPUT_LOG_LEVEL -e HOME -e GITHUB_JOB -e GITHUB_REF -e GITHUB_SHA 
-e GITHUB_REPOSITORY -e GITHUB_REPOSITORY_OWNER -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER 
-e GITHUB_RETENTION_DAYS -e GITHUB_ACTOR -e GITHUB_WORKFLOW -e GITHUB_HEAD_REF 
-e GITHUB_BASE_REF -e GITHUB_EVENT_NAME -e GITHUB_SERVER_URL -e GITHUB_API_URL 
-e GITHUB_GRAPHQL_URL -e GITHUB_WORKSPACE -e GITHUB_ACTION -e GITHUB_EVENT_PATH 
-e GITHUB_ACTION_REPOSITORY -e GITHUB_ACTION_REF -e GITHUB_PATH -e GITHUB_ENV -e RUNNER_OS 
-e RUNNER_TOOL_CACHE -e RUNNER_TEMP -e RUNNER_WORKSPACE -e ACTIONS_RUNTIME_URL 
-e ACTIONS_RUNTIME_TOKEN -e ACTIONS_CACHE_URL -e GITHUB_ACTIONS=true -e CI=true 
-v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" 
-v "/home/runner/work/_temp/_github_workflow":"/github/workflow" 
-v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" 
-v "/home/runner/work/demoenv-checker/demoenv-checker":"/github/workspace" 179394:777995e8eba344359fc6689913976734
2020-11-17 15:05:08 +0000 - publish-unit-test-results -  INFO - reading junit.xml: ['junit.xml']
2020-11-17 15:05:09 +0000 - publish.publisher -  INFO - publishing results for commit 7e5be34347badfe8b51f94522303a9c471e05e75
2020-11-17 15:05:09 +0000 - publish.publisher -  INFO - creating check
2020-11-17 15:05:11 +0000 - publish.publisher -  INFO - creating comment
2020-11-17 15:05:12 +0000 - publish.publisher -  INFO - hiding unit test result comment MDEyOklzc3VlQ29tbWVudDcyODk1MDQ2NA==

But as you mentioned there is not way to provide the run_id when creating the check...

[EnricoMi]

Thanks for the screenshots and feedback, nice to see the action running in other situations then I envisioned. I'll keep trying to find a solution for this though.

[EnricoMi]

This is a missing feature in the Actions API and Github developers are aware: https://github.community/t/specify-check-suite-when-creating-a-checkrun/118380. 👀

[EnricoMi]

Branch branch-multi-workflows-random-check-suite reproduces this issue. The first workflow runs the unit test action, the second workflow does something else. The check run with unit test results from the first workflow appear on the second workflow:

[EnricoMi]

My hypothesis is that the result is posted to the workflow that started last. One idea to test this and tackle this issue then is to have the current unit test result workflow sleep a bit and then trigger another workflow that actually runs the unit test result action. That way we increase chances the action is started last and its result is then posted to the same workflow.

In #64 @RyanSchw posted a similar setup where the push event triggers another workflow that then runs the action.

Worth giving it a try.

[cicdguy]

The way that Github's Super-Linter does it is by creating a new status check for each language that it lints:
https://github.com/github/super-linter/blob/master/lib/linter.sh#L598-L611

Would it make sense to post a separate status check for this altogether? That way the status check would be decoupled from the workflow and prevent assignment to a random workflow.

[EnricoMi]

An interesting observation is that artifact uploads are workflow run specific, they end up at the right workflow: #165 (comment). I'll look into how that action uses the API to achieve this.

[djcsdy]

FYI for artifacts the workflowRunId is part of the URL to which the artifact is posted:
https://github.com/actions/upload-artifact/blob/11e311c8b504ea40cbed20583a64d75b4735cff3/dist/index.js#L8309

I'm guessing that doesn't help us here, but thought I'd point it out JIC.

[bobleujr]

It'd be nice if you guys could go give a +1 in the communty too so github gives more attention to this issue. https://github.community/t/associate-a-check-run-with-a-workflow/122540/4

[TWiStErRob]

@bobleujr see also #12 (comment)
Can we link them in community?

[bobleujr]

@TWiStErRob just cross-commented. Thanks for bringing it to my attention!

[chriskinsman]

Hitting this also.

[TWiStErRob]

FYI: might be related slightly community/community#13480

[EnricoMi]

With GitHub introducing "job summary", the action now additionally publishes the results at the summary page of the workflow that runs the publish action. From there, a link to the check annotations is provided if failures exists: https://github.com/EnricoMi/publish-unit-test-result-action/tree/v1.35#github-actions-job-summary

This make it less urgent for GitHub to fix this randow-workflow issue for checks.

[titimoby]

I still have "Unit Test Results" under "Prettify" workflow but it is from ">Quick Tests" workflow.
Did I miss something to improve visibility of checks under the proper workflow?

[EnricoMi]

The "random workflow" issue is not fixed by the new "job summary" feature, but it helps to mitigate the impact.

The workflow that runs the publish job now contains the "job summary" with test results. In case test failures exist, the "job summary" contains a link to the test result annotations in the "random workflow".

So the location of the test result annotations does not matter as much any more.

[oadamczyk]

Ping! - still present for my multi workflows pipeline, you guys have any solution? 🙏

[EnricoMi]

Only Github can fix this: https://github.com/orgs/community/discussions/24616

[steve-todorov]

On other actions I've seen people commenting the only workaround to this epic issue is to publish only the summary result without the check suite section. Are there any existing configuration flags that allow this behavior in EnricoMi/publish-unit-test-result-action ?

[lt-mayonesa]

I did some tests and it doesn't seem to be completely random, from what I can tell it publishes to the first created workflow for the commit. So what I ended up doing was renaming my workflows to something like:

• 01_ci-cd
• 02_guidelines
• 03_DevEx

not very pretty, I know

So far they are all landing on ci-cd. You can check it out here lt-mayonesa/hexagon

This is the case when all workflows run successfully, If one of them fails early I think they start landing on the failed one (but I still have to validate this scenario).

[steve-todorov]

@lt-mayonesa I could save you the time as I've seen the issue multiple times during a migration to GH Actions. Long story short it is not random and you can't fix it on your end.

The best way to explain it would be with a mutability field. Assume you have var workflow = workflow-1 that is defined by the workflow that executes the build (i.e. workflow-1). Immediately afterwards, a second workflow-2 sets var workflow = workflow-2. Somehow this field is shared between these workflows internally and is mutable. As a result, once workflow-1 completes the results will be published at workflow-2 (because the var workflow = workflow-2 instead of var workflow = workflow-1)

Same goes if workflow-2 started first and next was workflow-1 -- the results of workflow-2 will end up workflow-1. If you had 3 or 4 workflows the same would be true -- it only matters which workflow started executing last. That would be the workflow containing the results from all other workflows.

It's either this or internally there's some metadata that's missing and the runner picks the last running workflow (i.e. sorted by build number)

Anyway... in both cases there isn't much that can be done from the user end point. Feel free to complain here:

https://github.com/orgs/community/discussions/24616

Maybe if enough pressure is put the issue will be resolved.

[lt-mayonesa]

@steve-todorov , thanks for the info, tho I think we ment the same thing. So far for me, by using the "naming convention" I managed to get that ~90% of the runs the checks get created were they should (01_ci-cd). So I'm quite happy with it.

[dev-hanz-ops]

You can force the check-run of this action to always use a separate check-suite. To do so, you

• create a GitHub App (permissions e.g. Read and write access to actions, checks, code, discussions, and pull requests - possibly too broad for your case)
• create a private key for it and store it in your GH Actions secrets
• install the app to your org / repo

•       - id: app_token
          uses: tibdex/github-app-token@b62528385c34dbc9f38e5f4225ac829252d1ea92
          if: always()
          env:
            OPENSSL_CONF: /dev/null  # https://github.com/tibdex/github-app-token/issues/54
          with:
            app_id: your-app-id
            private_key: ${{ secrets.PRIVATE_KEY_YOUR_APP }}
  
        - name: Publish Test Results
          uses: EnricoMi/publish-unit-test-result-action@v2
          if: always()
          with:
            # when using the regular GITHUB_TOKEN, the check-run created by this step will be assigned to a
            # random workflow in the GH UI. to prevent this, we can force the check-run to be created in a separate
            # check-suite, which is created automatically if we use the credentials of a GitHub App
            github_token: ${{ steps.app_token.outputs.token }}
  

result

It comes at the cost of not having the published message as part of the GH Actions run - this would otherwise be randomly assigned.

Answer in discussion: https://github.com/orgs/community/discussions/24616#discussioncomment-5607870

[dagansandler]

@EnricoMi In order to avoid this issue, will it be possible to add a flag that disables the check run reporting and only keep the comments and/or job summary? I rather have that than see the check appear at a random location

[rwader-swi]

From what I observed, the workflow reports are attached to the check run referring to the commit that is latest on the branch-from-which-workflow-has-run (in my case I'm using workflow_dispatch). I found that the env var GITHUB_SHA which is defined by GHA has the default value of the latest commit SHA on the triggering branch - https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_dispatch, so if I change this env variable will the check runs & summary go to the intended place. Also I'm changing this env variable only for Enrico test report step, so are you guys using it anywhere in a way that would create a clashing?

[rahulshrivastava-eaton]

@EnricoMi I am getting error "Error: Container action is only supported on Linux" when running this on the Self-Hosted Windows runners any suggestions ?

[rahulshrivastava-eaton]

@TWiStErRob i tried that it is giving the error
shell: C:\Windows\system32\bash.EXE --noprofile --norc -e -o pipefail {0}
The file cannot be accessed by the system.
Error: Process completed with exit code 1.

[EnricoMi]

@rahulshrivastava-eaton can you please discuss this in a separate issue, this is completely unrelated.

[EnricoMi]

@EnricoMi In order to avoid this issue, will it be possible to add a flag that disables the check run reporting and only keep the comments and/or job summary? I rather have that than see the check appear at a random location

This feature has been released 19th, Dec. 2023. See README.md for details:

Check runs can be disabled with check_run: false.

[robrich]

I just hit this issue too. The monorepo has 2 projects and 2 builds. When both builds kick off at once, the build summary is on the correct build, but for both the annotations and test results tab, all results are on the first build that finishes.