-
Notifications
You must be signed in to change notification settings - Fork 103
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add voice command integration with whisper #66
Comments
We could make this a potentially powerful tool for people who have disabilities! If we let people voice type code, and it’s transcribe, we can have GPT evaluate it, and suggest edits (assuming there are going to be transcription imperfections). So Whisper hears “assign a random sample from a normal with mean 2 too variable C” then GPT writes code and its pasted, and if it’s evaluated not to be valid R code 2/3 alternates are offered? |
Do you know of an R package which allows to record audio? |
Id say a pipeline thats loosely: ` transcript <- whisper_api_call(audio) # we write based on this API call: POST https://api.openai.com/v1/audio/transcriptio explained here: https://platform.openai.com/docs/api-reference/audio/create chat_response <- gpt_chat(..., instruction = transcript) ` |
You can indeed of course make an API call and send the voice with a http request to an openai endpoint. |
I played around with this a good bit over the weekend. It's tough to get it working consistently. Milage was quite different with different browsers and even versions of RStudio. The issue is mostly related to collecting the recording. Whisper API is straightforward. |
Maybe a javascript library exists which does cross-platform audio recording in the browser and an R interface can be created on top of it wrapped alongside htmltools? |
This looks promising https://github.com/coolbutuseless/carelesswhisper |
Use whisper to incorporate voice commands.
h/t @jwijffels
See: https://twitter.com/jwijffels1/status/1638650905636618241?s=20
The text was updated successfully, but these errors were encountered: