CoreML support, for Apple Silicon Macs?

[angelkaragiozov]

I'm using Vlad UI on my M2 MBP and it's working great with the Torch 2.0 support, but I wondered if there was any way to use the CoreML models that Apple released (e.g. https://huggingface.co/apple/coreml-stable-diffusion-v1-4/tree/main/original/compiled)? They're not in .ckpt format so I assume I can't just drop them in the /models folder.

Any plans to support Core ML models in the near future?

via:
https://github.com/apple/ml-stable-diffusion

P.S. Mochi Diffusion and Diffusion Bee are fantastic apps, and they work very fast natively on M1 / M2 , but they are very limited , it would be great to have it all in a comprehensive UI

[vladmandic]

coreml is veeery different, it would be a large efford. i'd welcome it if someone took it on themselves - i'm not a mac user and cant do much about it.

[angelkaragiozov]

They made it easier now, soon it will be an automated process and just a matter of convention. Check this: https://developer.apple.com/videos/play/tech-talks/10154/
Perhaps you can get support from these guys, for sure they will be interested in the task ;)

[vladmandic]

Not really easier at all. Converting model is easy part - then what? All of the code is PyTorch.
Unless they make torch-coreml, this remains a massive and not likely efford.

[angelkaragiozov]

They invented a programming language Swift it is a joke ;)

I am not an expert, just asking...

I am not sure but as I understand you can convert not only models but also scripts, perhaps it will be something like every app has an installer for each OS

Convert Models to ML Programs <---- https://coremltools.readme.io/docs/ml-programs
You can convert a TensorFlow or PyTorch model, or a model created directly in the Model Intermediate Language (MIL), to a Core ML model that is either an ML program or a neural network. The Unified Conversion API can produce either type of model with the convert() method.

(GPT says) CoreML is compatible with PyTorch scripts through the use of conversion tools and libraries. CoreML is a framework developed by Apple for machine learning model deployment on iOS, macOS, watchOS, and tvOS devices. PyTorch, on the other hand, is a popular deep learning framework commonly used for model development and training.

However, it is important to note that not all PyTorch operators are supported by Core ML. If you are using an operator that is not supported, you will need to modify your model or use a different method to convert it to Core ML.

Overall, CoreML is a good choice for deploying PyTorch models on Apple devices. It is easy to use and supports a wide range of features. However, it is important to be aware of the limitations of CoreML before converting your models.

[vladmandic]

safetensors is a model. script they are talking about are scripts inside a model. not applications that actually use a model.
what about entire sdnext source code where its using torch? who's going to convert that. that's what i'm talking about.

[angelkaragiozov]

Unfortunately ..... I hope they will find a solution, something like Rosetta, but for PyTorch, so we Mac users can take advantage of the neural engine in Apple Silicon chips.

tnx and good luck :)

[raisindetre]

Apologies if this is off-topic but I'm wondering what the status/viability is of wiring Apple's tensorflow-metal plugin into the venv setup for sd.next. Looking at the code, it looks like nothing needs to be done other than installing the plug-in:

https://developer.apple.com/metal/tensorflow-plugin/

I'd be happy to test any support (Macbook Pro M2, 16GB). If I wanted to give it a go locally, would it just be a case of navigating into venv and running the plugin installer?

[vladmandic]

sure, but to what purpose? that's tensorflow plugin and majority of work here is done with torch - and apple did not release torch plugin.