Installing OpenLLM in Docker

[MathewT]

Hello! First time using OpenLLM. I am attempting to create a Docker image with OpenLLM in my local dev environment. I receive the following error message when attempting to install OpenLLM:

#0 92.76 ERROR: Cannot install transformers[accelerate,onnx,onnxruntime,tokenizers,torch]==4.29.0, transformers[accelerate,onnx,onnxruntime,tokenizers,torch]==4.29.1, transformers[accelerate,onnx,onnxruntime,tokenizers,torch]==4.29.2, transformers[accelerate,onnx,onnxruntime,tokenizers,torch]==4.30.0, transformers[accelerate,onnx,onnxruntime,tokenizers,torch]==4.30.1 and transformers[accelerate,onnx,onnxruntime,tokenizers,torch]==4.30.2 because these package versions have conflicting dependencies. #0 92.76 #0 92.76 The conflict is caused by: #0 92.76 transformers[accelerate,onnx,onnxruntime,tokenizers,torch] 4.30.2 depends on onnxruntime>=1.4.0; extra == "onnx" #0 92.76 transformers[accelerate,onnx,onnxruntime,tokenizers,torch] 4.30.1 depends on onnxruntime>=1.4.0; extra == "onnx" #0 92.76 transformers[accelerate,onnx,onnxruntime,tokenizers,torch] 4.30.0 depends on onnxruntime>=1.4.0; extra == "onnx" #0 92.76 transformers[accelerate,onnx,onnxruntime,tokenizers,torch] 4.29.2 depends on onnxruntime>=1.4.0; extra == "onnx" #0 92.76 transformers[accelerate,onnx,onnxruntime,tokenizers,torch] 4.29.1 depends on onnxruntime>=1.4.0; extra == "onnx" #0 92.76 transformers[accelerate,onnx,onnxruntime,tokenizers,torch] 4.29.0 depends on onnxruntime>=1.4.0; extra == "onnx" #0 92.76 #0 92.76 To fix this you could try to: #0 92.76 1. loosen the range of package versions you've specified #0 92.76 2. remove package versions to allow pip attempt to solve the dependency conflict #0 92.76 #0 92.76 ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

My Dockerfile is:

FROM python:3.11.3-alpine3.18
LABEL maintainer="[email protected]"
RUN apk add --update --no-cache bash bash-completion
libffi-dev tzdata git postgresql-client &&
apk add --update --no-cache --virtual .tmp-build-deps
build-base postgresql-dev

ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1

WORKDIR /app

RUN echo "alias ll='ls -alF'" >> $HOME/.bashrc
echo "alias la='ls -A'" >> $HOME/.bashrc
echo "alias l='ls -CF'" >> $HOME/.bashrc
echo "alias q='exit'" >> $HOME/.bashrc
echo "alias c='clear'" >> $HOME/.bashrc

COPY ./requirements.txt /tmp/requirements.txt
COPY ./requirements.dev.txt /tmp/requirements.dev.txt
COPY ./app /app/

ARG DEV=false
ENV VIRTUAL_ENV=/py
ENV PATH="${VIRTUAL_ENV}/bin:$PATH"

RUN python -m venv ${VIRTUAL_ENV} &&
${VIRTUAL_ENV}/bin/pip install --upgrade pip &&
${VIRTUAL_ENV}/bin/pip install -r /tmp/requirements.txt &&
if [ $DEV = "true" ];
then ${VIRTUAL_ENV}/bin/pip install -r /tmp/requirements.dev.txt ;
fi &&
rm -rf /tmp &&
apk del .tmp-build-deps &&
adduser
-D
-H
flask-user

USER flask-user
CMD [ "/bin/bash" ]

Any ideas will be appreciated. Thanks.

Mathew

[aarnphm]

You can try the following workflow

openllm build <model_name>

Then it should show you how you can containerize the LLM. I understand the notion of using alpine, but for the use case of LLM, it doesn't make sense because you will end up having to rebuild a lot of kernel from source, which is time consuming and not usually recommended.

you can use bentoml containerize to create a container for this given LLM.

[MathewT]

My apologies for not understanding. My attempt to build the Docker image is failing when it attempts to pip install openllm. It is failing with the error I shared in this discussion. If it is failing because of Alpine base, is there a different base Docker image you would recommend? Also, I am new to LLMs and I am still learning. Thank you for your help.

[aarnphm]

Usually, it makes sense to use ubuntu base, as they have all of the required kernel for all deep learning library to build on top of.

Just out of curious behind the decision for building your own dockerfile. OpenLLM should already help you with this 🤔

[coolrazor007]

I'm working on this using:
FROM python:3.11-slim-buster
So far it works. Once I get a docker-compose.yml going to I'll make a PR.

[Bennoo]

I don't get the philosophy here. If I want to build a container able to be run on a private K8s for example, do I have to use openllm build command?