Unable to reproduce the performance on the llamaQuestions dataset as reported in the paper.

[UltraEval]

Due diligence

• I have done my due diligence in trying to find the answer myself.

Topic

The paper

Question

Unable to Reproduce Moshi's Performance on llamaQuestions Dataset

I attempted to replicate Moshi's performance on the llamaQuestions dataset as reported in the paper, but achieved only 13%.

Testing Methodology:

I used the inference method detailed at:
https://github.com/kyutai-labs/moshi/blob/main/moshi/README.md#api

Request for Clarification:

Could you please release the test script for LlamaQuestions?

Observation:

If the script used is https://github.com/kyutai-labs/moshi/blob/main/scripts/moshi_benchmark.py, it does improve performance to 55%. However, it seems unusual to perform inference 10 times for same question.

Attached is my testing process file.

moshi_llamaQ.zip

[alkeryn]

afaik they didn't release the paper's model but only a demo, less powerful version