Multilingual support

[ArpitaisAn0maly]

Multilingual Support.

• Added code which detects language from user questions and translates response in same language in order to get more deterministic language translation.

• Language translation using prompt engineering was not giving concrete results. We also have query language which is default index/prompt language so we dont want to complicate prompt with many language parameters.

• Used deep_translator python library for translation and used langdetect for language detection for user question and response translation.deep_translator allows to perform language detection, but it does not work out of the box. you have to register and get the api key so that is why I used langdetect.

• with these approach it only sometimes misses translation but otherwise definitely an improvement over existing implementation. Translation is not 100% with LLM so this is acceptable.

• SS below Albanian and French question/response while in bg prompt and index is in english

[ArpitaisAn0maly]

PR Updated.

• Language detection and translation with Azure Translator API
• Search query translation using Azure translator API. Removal of query term language from prompt for translation.
• Removal of Query Language and speller from semantic reranking.
• Kept some language in prompt for translation of default responses.
• Kept query term language in prompt to help with robust retrieval.