9GB download for every docker compose up

[bilogic]

Hi,

There is this 9GB download each time I run docker compose up which I assume is a LLM?

Why not move this into the docker-compose.yaml examples on https://hub.docker.com/r/3x3cut0r/privategpt#docker-compose

Assuming it is a LLM, this file should persist on the host file system, and will be downloaded only if it does not exists.

Better right?

[3x3cut0r]

If i create this stack i do not have do download the LLM again. Could you provide your docker-compose.yml that i can have a look on it?

[bilogic]

Hi,

I tested it a bit deeper, the redownload only happens if I run docker system prune --volumes -f -a when the container is not running. I think the bigger question is, is this 9GB really all code or should we store it in the data folder?

[3x3cut0r]

its not code ... its the default LLM and its stored at '/home/worker/app/models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf'
if you want to put it into a volume than feel free to do so. no need to be defined by me or do you get some permission problems?

[bilogic]

1. But Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf is only 4.9GB in my data folder
2. When I run docker system prune --volumes -f -a, Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf still exists on my data folder, so why up still need to download again?
3. See below, my up has a 9.063GB download 64142b05a920

[+] Running 9/17
 ⠙ privategpt [⣿⣿⣿⣿⣿⣿⣿⣦⣿⣿⠀⠀⠀⠀⠀⠀] Pulling                                                                                                                                                15.2s
   ✔ a2318d6c47ec Already exists                                                                                                                                                         0.0s
   ✔ 40d734479f14 Pull complete                                                                                                                                                          1.1s
   ✔ 0b61b7b259eb Pull complete                                                                                                                                                          2.2s
   ✔ 081a3493c0e7 Pull complete                                                                                                                                                          2.2s
   ✔ 9ba21e5fdc5c Pull complete                                                                                                                                                          2.5s
   ✔ dd58e8c66e60 Pull complete                                                                                                                                                          2.5s
   ✔ b94476f1e614 Pull complete                                                                                                                                                         10.9s
   ⠼ 9c4fc54ecc6b Downloading [================================>                  ]  144.8MB/222.9MB                                                                                    11.5s
   ✔ 89e5a37135f1 Download complete                                                                                                                                                      3.1s
   ✔ 96bb6f2e6429 Download complete                                                                                                                                                      4.3s
   ⠼ 64142b05a920 Downloading [>                                                  ]  40.77MB/9.063GB                                                                                    11.5s
   ⠼ f0d70c17e50e Downloading [===>                                               ]  5.387MB/71MB                                                                                       11.5s
   ⠼ 14f2aa8c4cd9 Waiting                                                                                                                                                               11.5s
   ⠼ b92d1cd2ce25 Waiting                                                                                                                                                               11.5s
   ⠼ 991b3da41cb6 Waiting                                                                                                                                                               11.5s
   ⠼ ef7ee13a3b0b Waiting                                                                                                                                                               11.5s

[3x3cut0r]

So i released a new version of this docker container. I removed the setup inside the Dockerfile so no preinstalled llm is present. Image size i now reduced to 3,25gb.
I tested it out and after restarting (docker compose up) the container it wont redownload the model.

I know its been a while sorry for that.
Could you have a look on it if its still a thing.

[bilogic]

ok sure, give me some time thanks :)

[bilogic]

Hi, I confirmed it is no longer downloading the 9GB model.

1. May I ask what is this 3.096GB file? Curious as most if not all my other images come in under 1GB.
2. The image is now unable to start

privategpt  | 16:54:02.084 [INFO    ] privategpt version: 0.6.2
privategpt  | 16:54:02.269 [INFO    ] running scripts/setup
privategpt  | 16:54:06.970 [INFO    ] private_gpt.settings.settings_loader - Starting application with profiles=['default']
privategpt  | Traceback (most recent call last):
privategpt  |   File "/home/worker/app/scripts/setup", line 8, in <module>
privategpt  |     from private_gpt.paths import models_path, models_cache_path
privategpt  |   File "/home/worker/app/private_gpt/paths.py", line 4, in <module>
privategpt  |     from private_gpt.settings.settings import settings
privategpt  |   File "/home/worker/app/private_gpt/settings/settings.py", line 625, in <module>
privategpt  |     unsafe_typed_settings = Settings(**unsafe_settings)
privategpt  |                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
privategpt  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/pydantic/main.py", line 212, in __init__
privategpt  |     validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
privategpt  |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
privategpt  | pydantic_core._pydantic_core.ValidationError: 1 validation error for Settings
privategpt  | huggingface.access_token
privategpt  |   Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]
privategpt  |     For further information visit https://errors.pydantic.dev/2.9/v/string_type
privategpt  | 16:54:07.722 [INFO    ] privategpt version: 0.6.2
privategpt  | 16:54:07.907 [INFO    ] running scripts/setup
privategpt  | 16:54:10.630 [INFO    ] private_gpt.settings.settings_loader - Starting application with profiles=['default']
privategpt  | Traceback (most recent call last):
privategpt  |   File "/home/worker/app/scripts/setup", line 8, in <module>
privategpt  |     from private_gpt.paths import models_path, models_cache_path
privategpt  |   File "/home/worker/app/private_gpt/paths.py", line 4, in <module>
privategpt  |     from private_gpt.settings.settings import settings
privategpt  |   File "/home/worker/app/private_gpt/settings/settings.py", line 625, in <module>
privategpt  |     unsafe_typed_settings = Settings(**unsafe_settings)
privategpt  |                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
privategpt  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/pydantic/main.py", line 212, in __init__
privategpt  |     validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
privategpt  |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
privategpt  | pydantic_core._pydantic_core.ValidationError: 1 validation error for Settings
privategpt  | huggingface.access_token
privategpt  |   Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]
privategpt  |     For further information visit https://errors.pydantic.dev/2.9/v/string_type
privategpt  | 16:54:11.484 [INFO    ] privategpt version: 0.6.2
privategpt  | 16:54:11.670 [INFO    ] running scripts/setup
privategpt  | 16:54:14.357 [INFO    ] private_gpt.settings.settings_loader - Starting application with profiles=['default']
privategpt  | Traceback (most recent call last):
privategpt  |   File "/home/worker/app/scripts/setup", line 8, in <module>
privategpt  |     from private_gpt.paths import models_path, models_cache_path
privategpt  |   File "/home/worker/app/private_gpt/paths.py", line 4, in <module>
privategpt  |     from private_gpt.settings.settings import settings
privategpt  |   File "/home/worker/app/private_gpt/settings/settings.py", line 625, in <module>
privategpt  |     unsafe_typed_settings = Settings(**unsafe_settings)
privategpt  |                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
privategpt  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/pydantic/main.py", line 212, in __init__
privategpt  |     validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
privategpt  |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
privategpt  | pydantic_core._pydantic_core.ValidationError: 1 validation error for Settings
privategpt  | huggingface.access_token
privategpt  |   Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]
privategpt  |     For further information visit https://errors.pydantic.dev/2.9/v/string_type
privategpt  | 16:54:15.368 [INFO    ] privategpt version: 0.6.2
privategpt  | 16:54:15.556 [INFO    ] running scripts/setup
privategpt  | 16:54:18.367 [INFO    ] private_gpt.settings.settings_loader - Starting application with profiles=['default']
privategpt  | Traceback (most recent call last):
privategpt  |   File "/home/worker/app/scripts/setup", line 8, in <module>
privategpt  |     from private_gpt.paths import models_path, models_cache_path
privategpt  |   File "/home/worker/app/private_gpt/paths.py", line 4, in <module>
privategpt  |     from private_gpt.settings.settings import settings
privategpt  |   File "/home/worker/app/private_gpt/settings/settings.py", line 625, in <module>
privategpt  |     unsafe_typed_settings = Settings(**unsafe_settings)
privategpt  |                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
privategpt  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/pydantic/main.py", line 212, in __init__
privategpt  |     validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
privategpt  |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
privategpt  | pydantic_core._pydantic_core.ValidationError: 1 validation error for Settings
privategpt  | huggingface.access_token
privategpt  |   Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]
privategpt  |     For further information visit https://errors.pydantic.dev/2.9/v/string_type

[3x3cut0r]

Hi, I confirmed it is no longer downloading the 9GB model.

1. May I ask what is this 3.096GB file? Curious as most if not all my other images come in under 1GB.

Its llama-cpp-python i guess but i dont know.

2. The image is now unable to start

oh deer, was my fault ... shoud be fixed