[REQUEST] Make Ollama URL customizable

[ferpizza]

Reference Issues

No response

Summary

Currently, Ollama URL is hardcoded to http://localhost:11434/api

This becomes an issue when deploying Kotaemon in more advanced scenarios, like running different containers for Kotaemon and Ollama.

Basic Example

Kotaemon docker image doesn't include Ollama, so one approach to tackle this is to run a 2nd docker image with Ollama in it.

The challenge is that on this scenario, Ollama is no longer available on http://localhost:11434/api, but on some other URL like http://ollama:11434/api.

As Ollama URL is not a setting we can manage through EnvVars, the above deployment is not possible without having to tweak the code.

Drawbacks

We are unable to run Kotaemon and Ollama on different VMs/Containers, and this is a common pattern when working with contaiers (eg. kubernetes, docker-compose)

Additional information

No response

[taprosoft]

@ferpizza User can always edit any parameters to the models, including base_url like this

Can you try this method?

[ferpizza]

In my case, the issue with Ollama was happening at the initial Setup screen, where we have no chance to edit these settings ... and it was failing to download models and move forward (and I didn't wanted to go into advance mode to get to the screen you are sharing).

I workaround it by adding a sed command to the official docker entrypoint, and was able to download models and get a working connection with Ollama running on an independent container.

But I think that having this on an EnvVar is better and cleaner. (also, my workaround is not bulletproof)

    entrypoint:
      - /bin/sh
      - -c
      - |
        sed -i 's/localhost:11434/ollama:11434/' libs/ktem/ktem/pages/setup.py &&
        python app.py

[ferpizza]

Also, on the screen you share, I still see localhost:11434 so I guess it's not showing the actual config used on the instance, but some default setting that are sourced from somewhere else? ... maybe from flowsettings.py (that I havent changed) ?

[taprosoft]

@ferpizza yes we did not consider alternate Ollama endpoint in first setup screen. User are expected to use Skip setup and Set the model in the UI manually later.

An env var specifically for Ollama might probably help, we will consider this feature.