Feature Request: Support for Multiple LLM AI API Endpoints for Self-Hosting and Model Selection

[2good4hisowngood]

Title: Support for Multiple LLM AI API Endpoints for Self-Hosting and Model Selection

Feature Request

Description:
We would like to propose the addition of a new feature to ChatDev that enables users to configure and utilize multiple Language Model (LLM) AI API endpoints for self-hosting and experimentation with different models. This feature would enhance the flexibility and versatility of ChatDev for developers and researchers working with LLMs.

Feature Details:

1. Endpoint Configuration:

   • Allow users to store API keys and configure multiple endpoints.
   • Implement support for an environment (env) file to securely store sensitive information locally, facilitating scripted pass-through of values. Add to .gitignore

2. Custom Endpoint Names:

   • Provide the ability to assign user-friendly names to each configured endpoint. This helps users easily identify and differentiate between endpoints. It would also allow multiple models to be leveraged on the same endpoint by having different configurations for each. A check could occur to validate that the endpoint has the model expected, and if not, do a quick unload/load of the desired model.

3. Chat Parameters:

   • Integrate settings for chat parameters, such as temperature and other relevant options, that can be adjusted per endpoint. This allows users to fine-tune model behavior.

4. Model Selection (if applicable):

   • If applicable to the specific LLM, enable users to preset a model for each endpoint. This feature can be especially useful when working with multiple LLMs simultaneously.

5. API Key Management (if applicable):

   • For LLM services like OpenAI that require API keys, provide a dedicated parameter in each endpoint for users to input and manage their API keys for each endpoint.

6. Endpoint Address:

   • Allow users to specify the endpoint address (URL) to which API requests should be sent. This flexibility is crucial for self-hosted instances or when working with different LLM providers.

7. Optional - Endpoint Tagging:

• Allowing us to add tags like #code #logic, or #budget could let us give key indicators of where a model's strengths are in, and select from a pool of models with a particular benefit, allowing more diverse outcomes. It could also allow for side-by-side comparisons. This could allow future result tracking/scoring to better identify which models are best at particular features, by having multiple #code models, and testing each's results you can identify and retrain or replace under-performing models to build an optimum workflow.

Expected Benefits:
This feature will benefit developers, researchers, and users who work with LLMs by offering a centralized and user-friendly interface for managing multiple AI API endpoints. It enhances the ability to experiment with various models, configurations, and providers while maintaining security and simplicity. This could allow different characters to leverage specific fine-tuned models rather than the same model for each. This could also allow self-hosted users to experiment with expand the number of repeated looped calls without drastically increasing the bill.

Additional Notes:
Consider implementing an intuitive user interface for configuring and managing these endpoints within the GitHub platform, making it accessible to both novice and experienced users.

References:
Include any relevant resources or references that support the need for this feature, such as the growing popularity of LLMs in various fields and the demand for flexible API management solutions.

Related Issues/Pull Requests:
#27
#33
Azure OpenAI #55

Assignees:
If you permit this ticket to remain open, I will assemble some links and resources, as well as opening another ticket to handle TextGenWebUI with relevant links there to implementing it. I can try implementing and doing a PR if someone else doesn't get to it first.

Thank you for considering this feature request. I believe that this enhancement will greatly benefit the ChatDev community and its users working with Language Model AI API endpoints.

[qianc62]

Thank you for your advice. At present, we are carrying out continuous repair and maintenance for some potential problems, and currently lack enough manpower to complete this matter. If possible, we'd love for you to come up with your ideas and send a pull request.
Thank you for your question and hope you can enjoy ChatDev! 😀

[Egalitaristen]

Hey, I use LM Studio to run local models and I just discovered a way to run multiple local models at the same time with it. I currently have 4 different local models running on my desktop at the same time right now.

And I've also found a way to use LM Studio Server with ChatDev. The output is a bit funky right now but I am running ChatDev locally and once you know how it's done the entire setup for everything takes less than 5 minutes.

I've "documented" it on the LM Studio discord server.

[Alphamasterliu]

Hello, regarding the use of other GPT models or local models, you can refer to the discussion on our GitHub page: #27. Some of these models have corresponding configurations in this Pull Request: #53. You may consider forking the project and giving them a try. While our team currently lacks the time to test every model, it's worth noting that they have received positive feedback and reviews. If you have any other questions, please don't hesitate to ask. We truly appreciate your support and suggestions. We are continuously working to improve more significant features, so please stay tuned.😊

[jwvanderbeck]

Hey, I use LM Studio to run local models and I just discovered a way to run multiple local models at the same time with it. I currently have 4 different local models running on my desktop at the same time right now.

And I've also found a way to use LM Studio Server with ChatDev. The output is a bit funky right now but I am running ChatDev locally and once you know how it's done the entire setup for everything takes less than 5 minutes.

I've "documented" it on the LM Studio discord server.

Could you share the process of making multiple model endpoints in LM Studio?

[Jonny-P]

Hey, I use LM Studio to run local models and I just discovered a way to run multiple local models at the same time with it. I currently have 4 different local models running on my desktop at the same time right now.
And I've also found a way to use LM Studio Server with ChatDev. The output is a bit funky right now but I am running ChatDev locally and once you know how it's done the entire setup for everything takes less than 5 minutes.
I've "documented" it on the LM Studio discord server.

Could you share the process of making multiple model endpoints in LM Studio?

Just open another instance/window of LM Studio:
1 - Load the other model that you want.
2 - Change the server port (Must be different port from all the other LM Studio instances)
3 - Start Server

Now you can query different models in different ports running at the same time on your local machine.