Skip to content

Add CoreML CI jobs for optimum-executorch models #12870

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 15 commits into
base: main
Choose a base branch
from

Conversation

metascroy
Copy link
Contributor

@metascroy metascroy commented Jul 25, 2025

This PR:

  • Bumps the optimum-executorch pin in our CI
  • Adds a new script .ci/scripts/test_huggingface_optimum_model.py to test optimum-executorch models
  • Adds a new trunk job to run CoreML models

Currently the following models are added:

  • qwen3 (quantized)
  • smollm (quantized)
  • llama3 (quantized)
  • olmo (quantized)

The following models can run for me locally with the script .ci/scripts/test_huggingface_optimum_model.py, but are not running in CI (requires investigation):

  • bert (quantized)
  • roberta (quantized)
  • distilbert (quantized)
  • vit

This CI job complements the other CI script .ci/scripts/test_model.sh that is based on the ExecuTorch examples repo.

Copy link

pytorch-bot bot commented Jul 25, 2025

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/12870

Note: Links to docs will display an error until the docs builds have been completed.

❌ 4 New Failures, 82 Pending, 1 Unrelated Failure

As of commit 9ec0ca7 with merge base 094b3c4 (image):

NEW FAILURES - The following jobs have failed:

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 25, 2025
Copy link

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@metascroy metascroy temporarily deployed to upload-benchmark-results July 25, 2025 23:34 — with GitHub Actions Inactive
@metascroy metascroy temporarily deployed to upload-benchmark-results July 26, 2025 00:30 — with GitHub Actions Inactive
@metascroy metascroy changed the title Update optimum-et pin Add CoreML optimum-et jobs Jul 28, 2025
@metascroy metascroy temporarily deployed to upload-benchmark-results July 28, 2025 17:20 — with GitHub Actions Inactive
@metascroy metascroy temporarily deployed to upload-benchmark-results July 28, 2025 17:50 — with GitHub Actions Inactive
@metascroy metascroy temporarily deployed to upload-benchmark-results July 31, 2025 22:43 — with GitHub Actions Inactive
@metascroy metascroy temporarily deployed to upload-benchmark-results July 31, 2025 23:33 — with GitHub Actions Inactive
@metascroy metascroy temporarily deployed to upload-benchmark-results July 31, 2025 23:46 — with GitHub Actions Inactive
@metascroy metascroy temporarily deployed to upload-benchmark-results August 1, 2025 02:58 — with GitHub Actions Inactive
@metascroy metascroy temporarily deployed to upload-benchmark-results August 1, 2025 06:07 — with GitHub Actions Inactive
@metascroy metascroy force-pushed the add-coreml-hf-tests branch 2 times, most recently from 28d6c5d to 80d6bbb Compare August 1, 2025 22:26
@metascroy metascroy changed the title Add CoreML optimum-et jobs Add CoreML CI jobs for optimum-executorch models Aug 1, 2025
@metascroy metascroy force-pushed the add-coreml-hf-tests branch from 7bc520d to 7ba2a60 Compare August 1, 2025 23:09
@metascroy metascroy marked this pull request as ready for review August 1, 2025 23:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ciflow/trunk CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants