M2 Pro Mac MLX installs, model loads, command line works, web version does not.

[abhinandanshahdev]

Due diligence

• I have done my due diligence in trying to find the answer myself.

Topic

The MLX implementation

Question

It shows microphone bars jumping up and down, no command line errors, just no audio out.

[Info] listening to http://localhost:8998
[Info] opening browser at http://localhost:8998
[Info] accepted connection
[Info] connection closed
[Info] done with connection
[Info] accepted connection
[Info] connection closed

[kunci115]

same experience in macbook air m3

[Gitterman69]

Those steps worked for me on M1 Pro (16gb)

Prerequisites

• macOS operating system
• Python 3.12 installed
• Git installed
• Basic familiarity with terminal commands

Step 1: Set Up the Environment

1. Open Terminal on your Mac.

2. Create a new directory for the project:

   mkdir ~/moshi_mmx
   cd ~/moshi_mmx

3. Create a Python virtual environment:

   python3.12 -m venv .venv

4. Activate the virtual environment:

   source .venv/bin/activate

Step 2: Clone the Repository

1. Clone the Moshi repository:

   git clone https://github.com/kyutai-labs/moshi.git

2. Navigate to the moshi_mlx directory:

   cd moshi/moshi_mlx

Step 3: Install the Package

Install the package in editable mode:

pip install -e .

Step 4: Create the AppleScript Launcher

1. Open "Script Editor" on your Mac (Applications > Utilities > Script Editor).

2. Copy and paste the following AppleScript code:

   on run
       set modelChoice to button returned of (display dialog "Choose a model:" buttons {"8-bit Model", "4-bit Model", "BF16 Model"} default button 1)
       
       set modelParams to ""
       if modelChoice is "8-bit Model" then
           set modelParams to "-q 8 --hf-repo kyutai/moshika-mlx-q8"
       else if modelChoice is "4-bit Model" then
           set modelParams to "-q 4 --hf-repo kyutai/moshika-mlx-q4"
       else
           set modelParams to "--hf-repo kyutai/moshiko-mlx-bf16"
       end if
       
       tell application "Terminal"
           activate
           set currentTab to do script "cd ~/moshi_mmx && source .venv/bin/activate && python -m moshi_mlx.local_web " & modelParams
       end tell
       
       -- Wait for the server to start (adjust the delay if needed)
       delay 5
       
       -- Open Safari and navigate to the local web interface
       tell application "Safari"
           activate
           open location "http://localhost:8998"
       end tell
   end run

3. Save the script:

   • Click on File > Save in the menu bar.
   • Choose a name (e.g., "Launch Moshi MLX")
   • Select a location (e.g., your Desktop for easy access)
   • For "File Format", choose "Application"
   • Check "Stay open after run handler"
   • Click "Save"

Step 5: Running the Application

1. Double-click the AppleScript application you created.
2. Choose the model you want to use from the dialog box.
3. The script will:
   • Open Terminal
   • Activate the virtual environment
   • Start the Moshi MLX web interface
   • Open Safari and navigate to the application

The web interface will be available at http://localhost:8998.

Troubleshooting

• If you encounter any permission issues, you may need to grant Terminal permission to control your computer. Go to System Preferences > Security & Privacy > Privacy > Automation and ensure Terminal is checked.
• If the web interface doesn't open automatically, wait a few more seconds and then manually navigate to http://localhost:8998 in your web browser.
• If you encounter any Python-related errors, ensure you're using Python 3.12 and that all dependencies are correctly installed.

Closing the Application

To stop the application, close the Terminal window that was opened by the script.

Enjoy using Moshi MLX!

[Gitterman69]

Quick mention: bf16 doesnt work on my 16gb ram machine... i guess its just not enough ram...

[LaurentMazare]

Right, moshi-mlx with a 16GB machine would be a bit tight as the weights alone are 15.4GB. Hopefully the q8 version should have an almost similar quality.

[Gitterman69]

Mosshi feels unhinged and i love it haha

[abhinandanshahdev]

have you gotten APIs to work with it? say hosting a conversational AI server locally on a M2 / M1 mac? Talking to the model is fun, but Id love to do a serious use case. Thanks for the help!

[Gitterman69]

actually im facing the problem that after about 2 minutes i reach this point and the server/inference webui crashes in console...

.....
ror in encoder thread narrow invalid args start + len > dim_len: [4096, 32], dim: 0, start: 4096, len:2
error in encoder thread narrow invalid args start + len > dim_len: [4096, 32], dim: 0, start: 4096, len:2
error in encoder thread narrow invalid args start + len > dim_len: [4096, 32], dim: 0, start: 4096, len:2
error in encoder thread narrow invalid args start + len > dim_len: [4096, 32], dim: 0, start: 4096, len:2
error in encoder thread narrow invalid args start + len > dim_len: [4096, 32], dim: 0, start: 4096, len:2
error in encoder thread narrow invalid args start + len > dim_len: [4096, 32], dim: 0, start: 4096, len:2
error in encoder thread narrow invalid args start + len > dim_len: [4096, 32], dim: 0, start: 4096, len:2
error in encoder thread narrow invalid args start + len > dim_len: [4096, 32], dim: 0, start: 4096, len:2
error in encoder thread narrow invalid args start + len > dim_len: [4096, 32], dim: 0, start: 4096, len:2
error in encoder thread narrow invalid args start + len > dim_len: [4096, 32], dim: 0, start: 4096, len:2
error in encoder thread narrow invalid args start + len > dim_len: [4096, 32], dim: 0, start: 4096, len:2
[Info] connection closed
[Info] done with connection

[LaurentMazare]

This last issue is likely reaching out the maximum conversation time, see #51 .

[LaurentMazare]

Closing as it's hopefully all good now, feel free to re-open/create a new one if you still encounter some issues.