Skip to content

metal : initial Metal4 tensor API support #16634

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 9 commits into
base: master
Choose a base branch
from
Open

metal : initial Metal4 tensor API support #16634

wants to merge 9 commits into from
+410 −131

Conversation

@ggerganov
Copy link
Member

@ggerganov ggerganov commented Oct 17, 2025 •
edited
Loading

  • Rework matrix-matrix multiplication
  • Use Tensor API when available

TODOs

  • Update mul_mm_id kernel
  • Test on M5 (looking for volunteers to test as I won't have hardware anytime soon)

@github-actions github-actions bot added testing Everything test related ggml changes relating to the ggml tensor library for machine learning Apple Metal https://en.wikipedia.org/wiki/Metal_(API) labels Oct 17, 2025
@jeffbolznv
Copy link
Collaborator

Any early performance data?

@ggerganov ggerganov force-pushed the gg/metal-mul-mm-rework branch from 6271c44 to 6726e53 Compare October 18, 2025 14:24
@ggerganov
Copy link
Member Author

@jeffbolznv I think the performance using the tensor API is the same as the old simdgroup-based implementation, but I haven't done detailed analysis yet. I don't have hardware yet to test the actual Neural Accelerators that exist in the new chips and if they would be utilized with these changes.

@ggerganov ggerganov force-pushed the gg/metal-mul-mm-rework branch from 6726e53 to 57fa815 Compare October 20, 2025 13:01
@ggerganov ggerganov marked this pull request as ready for review October 20, 2025 16:30
@ggerganov ggerganov requested a review from slaren as a code owner October 20, 2025 16:30
@ggerganov
Copy link
Member Author

Looking for volunteers with iPhone 17 or MacBook M5 for testing

@ddh0
Copy link
Contributor

ddh0 commented Oct 23, 2025

Looking for volunteers with iPhone 17 or MacBook M5 for testing

I have an iPhone 17, how can I help?

@ggerganov
Copy link
Member Author

@ddh0 Run the benchmark from #4508 on master and on this branch and report the results.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@slaren slaren Awaiting requested review from slaren

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

Apple Metal https://en.wikipedia.org/wiki/Metal_(API) ggml changes relating to the ggml tensor library for machine learning testing Everything test related

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ggerganov @jeffbolznv @ddh0