New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for MLC #59

Open

trufae opened this issue Sep 24, 2024 · 4 comments

Contributor

trufae commented Sep 24, 2024

https://llm.mlc.ai/

Collaborator

dnakov commented Sep 27, 2024

MLC looks great! We'll need some kind of installation script tho, to figure out whether to install the CUDA or CPU versions

Contributor

nitanmarcel commented Oct 7, 2024

You can serve it and use the openapi option we have in -m to use it if I'm not wrong. They seem to use a similar endpoint

https://llm.mlc.ai/docs/deploy/rest.html#deploy-rest-api

Contributor Author

trufae commented Oct 8, 2024

Maybe we should start by having a working installation radareorg/radare2-pm@b17240c do you want to give it a try?

Collaborator

dnakov commented Oct 28, 2024

simonw's implementation for his llm tool seems nice: https://github.com/simonw/llm-mlc/blob/main/llm_mlc.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment