quality: Round 2 of SDK test workflow migrations

[typotter]

🎟️ Fixes FF-3088 towards FF-3085

👯‍♂️ Related PRs

• Round 1 sdk-test-data#57
• java-server-sdk#75
• js-client-sdk#96
• python-sdk#65

Motivation

Changes are often made to the sdk-test-data repository to capture new behaviours, bugs and edge cases. When these changes are pushed to main, the SDKs are cloned locally (locally to the github action running) and their respective tests are run. These tests are set up and run by copies of the SDK test workflows - see sdk-test-data workflow. There are a number of limitations to this setup:

• Test steps are copied from the SDK’s respective workflows; changes to the SDK test workflows need to be replicated in sdk-test-data and this is not obvious to devs
• The SDK tests are not able to run against in-flight changes to sdk-test-data
• When new test-data is committed that breaks an SDK, that breakage is not surfaced in the SDK’s repository until some action triggers its testing workflow (on demand, on pull-request, etc.).

Description of Changes

This change
🚀 - Each SDK's testing workflow is enhanced into a reusable workflow, exposing parameters for SDK branch and the sdk-test-data branch to use in testing.

• The test workflow runs using the main branch of sdk-test-data on all main pushes and Pull Requests using the PR's branch

External to this Change
sdk-test-data get two testing workflows.

1. ♻️ "Local Testing"- For all pull request changes, the "Local Testing" workflow calls the reusable SDK workflows, test results are recorded only in the sdk-test-data action. This is run using the main SDK branch and the "current" branch of workflow, i.e. the pull request branch. (SDK repo does not see/is not notified of test failures during PR lifecycle, only on push to main)
2. ♻️ 🧑‍💻 "Remote Testing" - On all pushes to the main branch of sdk-test-data, the SDK testing workflows are triggered to run within their respective repositories, alerting all subscribers of failed runs (repo is red/green).

[greghuels]

Looks like this line was duplicated by accident

[typotter]

Thanks for the catch; late night merges...

[aarsilv]

Approving as-is, but I'd consider using a matrix to simplify/consolidate things

[aarsilv]

temporary I'm guessing?

What about running these shared tests on all pull requests, like we do for the individual SDKs?

[typotter]

test-sdks.yml is for in-flight changes; we don't want to make the SDKs in their respective repos until the test cases are checked in.

Temporary indeed; removed.

[aarsilv]

lots of duplicated code, have you considered using a matrix?

Unchecked example from ChatGPT:

test-sdks:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        sdk:
          - { repo: "java-server-sdk", workflow: "lint-test-sdk.yml", ref: "tp/workflows/remote" }
          - { repo: "php-sdk", workflow: "run-tests.yml", ref: "main" }
          - { repo: "js-client-sdk", workflow: "lint-test-sdk.yml", ref: "tp/workflows/remote" }
          - { repo: "python-sdk", workflow: "lint-test-sdk.yml", ref: "tp/workflows/remote" }
    steps:
    - name: Display workflow details
      shell: bash
      run: |
        echo "Testing eppo-exp/${{ matrix.sdk.repo }}"
    - name: Run remote workflow
      uses: convictional/[email protected]
      with:
        owner: Eppo-exp
        repo: ${{ matrix.sdk.repo }}
        workflow_file_name: ${{ matrix.sdk.workflow }}
        ref: ${{ matrix.sdk.ref }}
        github_token: ${{ secrets.AUTH_TOKEN }}
        wait_interval: 10
        propagate_failure: true
        trigger_workflow: true
        wait_workflow: true

[typotter]

I hadn't, in fact. 🧑‍🍳

[aarsilv]

If you merge these first I imagine you won't need this

[typotter]

that's right; will remove these refs after SDK PRs are merged

[aarsilv]

Is the eventual plan to remove this file and just use test-sdks-remote?

[typotter]

the remote workflow is to trigger the tests to run in the respective SDK repos, effectively turning them red when the test data causes a test failure. The test-sdks.yml runs the SDK tests but in the sdk-test-data repo, so it doesn't cause spurious notifications from the repos.