[CI/CD] Keep testing our main branch downstream every new patch merged

[dougsland]

CI/CD upstream don't have all arches we support downstream, we must be faster and smarter in the approach of dealing with it.
This issue is about to create a gitlab ci/cd job to monitor all patches merged in QM upstream, as soon it get merged, test the patch(es) in the arches downstream CI/CD, if it do not work, automatically create an issue reporting the output from the test for further investigation mentioning also the patches in question and the arches were tested.

gitlab monitoring script for github repo and Why create a github token for this task

Why do we need a github token? There is a rate limits for unauthenticated requests. Without a token, you can still use the GitHub API, but the number of allowed requests is much lower (60 requests per hour for unauthenticated users vs. 5,000 per hour for authenticated users).

To create a GitHub Personal Access Token (PAT), follow these steps:

Step-by-Step Guide to Create a GitHub Token:

1. Sign in to GitHub:

   • Go to github.com and log in to your GitHub account.

2. Access the Token Creation Page:

   • In the upper-right corner of any page, click on your profile picture, then select Settings.

3. Go to Developer Settings:

   • In the left sidebar, scroll down and click on Developer settings.

4. Personal Access Tokens:

   • Under Developer settings, click on Personal access tokens.
   • Click Tokens (classic) (you can also explore fine-grained tokens if needed, but classic tokens are most commonly used for API access).

5. Generate New Token:

   • Click Generate new token (for classic tokens).
   • GitHub will ask you to confirm your password if you haven't done so recently.

6. Configure Token Settings:

   • Note: Name your token in the Note field to remind you of its purpose (e.g., “GitLab CI Integration”).
   • Expiration: Choose an expiration date for the token. For security reasons, it's recommended to set an expiration (e.g., 30 or 60 days).
   • Scopes: Select the required scopes based on your needs:
     • repo: Full control over private repositories (required if you are working with private repositories).
     • read:org: Read access to organization membership.
     • workflow: Necessary if you need to trigger workflows.
     • write:repo_hook: If you need to create or modify repository hooks.
     • admin:repo_hook: Full control of repository hooks if needed.

7. Generate Token:

   • After selecting the scopes, click Generate token at the bottom.

8. Save the Token:

   • Important: After the token is generated, copy the token. This is the only time GitHub will show the token.
   • Store it in a safe place (e.g., a password manager) as you will not be able to view it again.

Example of How to Use the Token in Your Script:

Once you have your token, you can use it in your GitLab CI/CD job:

variables:
  GITHUB_API_TOKEN: "<your_github_token>"  # Add your token here or store it as a GitLab environment variable

monitor_patches:
  stage: monitor
  script:
    - echo "Monitoring GitHub for new merged pull requests..."
    - curl -s -H "Authorization: token ${GITHUB_API_TOKEN}" \
      "https://api.github.com/repos/${GITHUB_REPO}/pulls?state=closed&per_page=5" > prs.json

Storing Token in GitLab:

• Environment Variable: It's a good practice to store your GitHub token as a GitLab CI/CD environment variable. In your GitLab project:
  1. Go to Settings -> CI/CD -> Variables.
  2. Add a new variable named GITHUB_API_TOKEN with the value as your token. Make sure to mask it for security.

This way, your token won't be hardcoded in your .gitlab-ci.yml file.

INITIAL AND NON TESTED gitlab CI/CD script to monitor new patches merged upstream in QM and execute internal builds in gitlab and in case of failures (due different architectures for downstream) open an issue in github with the information so developers upstream are aware.

stages:
  - monitor
  - test
  - report_issue

variables:
  GITHUB_REPO: "containers/qm"  # owner/repo format for GitHub API, not the full URL
  INTERNAL_REPO: "git@internal-git-server:internal/repo.git"
  GITHUB_API_TOKEN: "<github token>"  # Store in GitLab CI/CD variables

# Step 1: Monitor for merged patches in upstream GitHub
monitor_patches:
  stage: monitor
  script:
    - echo "Monitoring GitHub for new merged pull requests..."
    - curl -s -H "Authorization: token $GITHUB_API_TOKEN" \
        "https://api.github.com/repos/${GITHUB_REPO}/pulls?state=closed&per_page=5" > prs.json
    - cat prs.json | jq '.[] | select(.merged_at != null) | .patch_url' > patch_urls.txt
    - if [ -s patch_urls.txt ]; then
        echo "New patches found";
        echo "patches_exist=true" > patch_status.env;
      else
        echo "No new patches";
        echo "patches_exist=false" > patch_status.env;
      fi
    - cat patch_status.env
  artifacts:
    reports:
      dotenv: patch_status.env  # Exporting the patch status to the next job

# Step 2: Apply patches and trigger internal tests if new patches were merged
test_patches:
  stage: test
  needs:
    - job: monitor_patches
  script:
    - if [ "$patches_exist" = "true" ]; then
        echo "Applying patches to internal repo and running tests...";
        for url in $(cat patch_urls.txt); do
          curl -s $url > patch.diff;
          git clone ${INTERNAL_REPO} internal_repo;
          cd internal_repo;
          git apply ../patch.diff;
          git commit -am "Applied patch from upstream GitHub";
          git push origin master;
        done;
        echo "Running tests on internal repo...";
        ./run_tests.sh || echo "tests_failed=true" >> patch_status.env;
      else
        echo "No new patches, skipping tests.";
      fi
  artifacts:
    reports:
      dotenv: patch_status.env  # Exporting the test status to the next job
  rules:
    - if: $patches_exist == "true"

# Step 3: Report failure by opening a GitHub issue with architecture and patches used
open_github_issue:
  stage: report_issue
  needs:
    - job: test_patches
  script:
    - if [ "$tests_failed" = "true" ]; then
        echo "Tests failed, reporting issue to upstream GitHub...";
        ARCHITECTURE=$(uname -m)  # Get the system architecture
        PATCHES_USED=$(cat patch_urls.txt)
        ISSUE_TITLE="CI/CD Test Failure on ${ARCHITECTURE}"
        ISSUE_BODY="CI/CD tests failed on ${ARCHITECTURE}.\nPatches applied:\n${PATCHES_USED}"
        curl -X POST \
          -H "Authorization: token $GITHUB_API_TOKEN" \
          -H "Accept: application/vnd.github.v3+json" \
          -d "{\"title\": \"${ISSUE_TITLE}\", \"body\": \"${ISSUE_BODY}\"}" \
          "https://api.github.com/repos/${GITHUB_REPO}/issues";
      else
        echo "Tests passed, no issue to report.";
      fi
  rules:
    - if: $tests_failed == "true"