FirstAI-Voice is an end-to-end voice model. FirstAI-Voice can directly understand and generate Indonesia and English speech, engage in real-time voice conversations, and change attributes such as emotion, intonation, speech rate, and dialect based on user instructions.
We provide the three components of FirstAI-Voice:
- FirstAI-Voice-Tokenizer: Trained by adding vector quantization to the encoder part of Whisper, converting continuous speech input into discrete tokens. Each second of audio is converted into 12.5 discrete tokens.
- FirstAI-Voice-9B: Pre-trained and aligned on speech modality based on FirstAI-9B, enabling understanding and generation of discretized speech.
- FirstAI-Voice-Decoder: A speech decoder supporting streaming inference, retrained based on CosyVoice, converting discrete speech tokens into continuous speech output. Generation can start with as few as 10 audio tokens, reducing conversation latency.
A more detailed technical report will be published later.
| Model | Type | Download |
|---|---|---|
| FirstAI-Voice-Tokenizer | Speech Tokenizer | 🤗 Huggingface |
| FirstAI-Voice-9B | Chat Model | 🤗 Huggingface |
| FirstAI-Voice-Decoder | Speech Decoder | 🤗 Huggingface |
We provide a Web Demo that can be launched directly. Users can input speech or text, and the model will respond with both speech and text.
First, download the repository
git clone --recurse-submodules https://github.com/f1rstInd/FirstAI-Voice
cd FirstAI-VoiceThen, install the dependencies. You can also use our pre-built docker image zhipuai/FirstAI-voice:0.1 to skip the step.
pip install -r requirements.txtSince the Decoder model does not support initialization via transformers, the checkpoint needs to be downloaded separately.
# Git model download, please ensure git-lfs is installed
git clone https://huggingface.co/f1rstInd/FirstAI-Voice- Start the model server
python model_server.py --host localhost --model-path FirstAI-Voice-9b --port 10000 --dtype bfloat16 --device cuda:0If you need to launch with Int4 precision, run
python model_server.py --host localhost --model-path FirstAI-Voice-9b --port 10000 --dtype int4 --device cuda:0This command will automatically download FirstAI-Voice-9b. If network conditions are poor, you can manually download it and specify the local path using --model-path.
- Start the web service
python web_demo.py --tokenizer-path f1rstind/FirstAI-Voice-tokenizer --model-path f1rstind/FirstAI-Voice-9b --flow-path ./FirstAI-voice-decoderYou can access the web demo at http://127.0.0.1:8888.
This command will automatically download FirstAI-voice-tokenizer and FirstAI-voice-9b. Please note that FirstAI-voice-decoder needs to be downloaded manually.
If the network connection is poor, you can manually download these three models and specify the local paths using --tokenizer-path, --flow-path, and --model-path.
- Gradio’s streaming audio playback can be unstable. The audio quality will be higher when clicking on the audio in the dialogue box after generation is complete.
Some code in this project is from:
-
The use of FirstAI model weights must follow the Model License Agreement.
-
The code in this open-source repository is licensed under the Apache 2.0 License.