Add CPU device-id support #150
Conversation
| @@ -91,11 +91,13 @@ | |||
| def main(): | |||
| args = parser.parse_args() | |||
|
|
|||
| dtype = torch.float32 if args.device_id == "cpu" else torch.float16 | |||
There was a problem hiding this comment.
Added this because otherwise fails on i5 CPU:
RuntimeError: Input type (torch.FloatTensor) and weight type (torch.HalfTensor) should be the same or input should be a MKLDNN tensor and weight is a dense tensor
|
Hi @Tradunsky - Sorry for the delayed update. I've been quite sick and a lil busy with work in the past few weeks. This is a lovely addition however, this would be incredibly slow for any audio above a minute. A more fool-proof way would be to use something like: ONNX pipeline (https://huggingface.co/Intel/whisper-large-v2-onnx-int4). This would require a bit of rework but should still be doable, would you like to update your PR to add support for it? |
|
Hi @Vaibhavs10, That's a great suggestion! I was also thinking of using neural compressor to speed up this on intel CPU 🤔 One potential drawback is users would have to switch between their local debugging and deployable models, risking to deal with model performance differences. Anyway, I would love to contribute when I have time 🙏🏻 |
Thank you for an easy to use CLI ❤️
Currently, if the library runs on CPU-only machine it fails with the following error:
It makes sense to have the library optimised for GPU 👍🏻 However, do you think CPU option can also be valuable for debugging locally?