Questions: new upcoming models (replacing v0 ones) + details

[jchwenger]

Hi there,

Since @ziyuan-linn said I could do that, I'm opening this more as a question for the future of the new API rather than a bug report:

• I saw that you have a llama branch: are you planning on supporting language models in some form at some point? How is this going so far? Of course I could imagine a bright future with streamdiffusion replacing pix2pix, and other things for style transfer, but that will all take quite a bit of work!
• The sound classifier still works with Teachable Machine, yes? I just tried it using v1 with models I had trained last year, but the new docs don't mention this, is it correct? (Just wondering if it's a matter of updating the docs, or if the Teachable Machine compatibility with sound is going to go away.)
• The v0 face mesh had a scaled version, that is gone, yes? I'm guessing one can just reproduce the scaled version manually, but just checking!

Thanks a lot!

[ziyuan-linn]

Hi @jchwenger, thank you for the questions!

I saw that you have a llama branch: are you planning on supporting language models in some form at some point? How is this going so far? Of course I could imagine a bright future with streamdiffusion replacing pix2pix, and other things for style transfer, but that will all take quite a bit of work!

The llama branch was an experiment to run a small language model in-browser. We are also looking into incorporating Transformers.js into the library. There are things we have to consider like API for larger cloud models and increased package size. Things are mostly experimental at the moment, but we are excited about diffusion models and transformers. Suggestions and contributions are always welcome!

The sound classifier still works with Teachable Machine, yes? I just tried it using v1 with models I had trained last year, but the new docs don't mention this, is it correct? (Just wondering if it's a matter of updating the docs, or if the Teachable Machine compatibility with sound is going to go away.)

The SoundClassifer still works Teachable Machine! We mentioned it on soundClassfier's reference page. Though it would be nice to have a dedicated Sound + Teachable Machine page similar to the Image + Teachable Machine page we have. Tagging @alanvww about this.

The v0 face mesh had a scaled version, that is gone, yes? I'm guessing one can just reproduce the scaled version manually, but just checking!

I just checked the v0 face mesh and there is a scaled version that no longer exists in v1. I am not too sure what the scaled version is since I never worked on the v0 models. Perhaps others can provide more context about it. If it is something that people find helpful, we could add it back!

[jchwenger]

Hi @ziyuan-linn,

And thanks a lot for your answers!

Integrating the flexibility of Huggingface for models would be quite amazing! I would be interested in helping out if I can, although that might have to wait until January next year, it's going to be tricky for me before that.

Props to the Sound + Teachable Machine docs!

As for the scaling, it returned normalised keypoints, which would allow you to have a facemesh following your movements, but say only in one corner of the canvas. I think this can still be achieved by normalizing by the canvas size manually (I haven't tried, but that seems straightforward). Probably best not to have it, so that the object returned by the model is slightly simpler and easier to read!