Dockerfile for llm and service in docker compose

[Kleczyk]

To run llm you need to download the model from hugingface:

wget -P ./llm/models https://huggingface.co/TheBloke/Llama-2-7B-GGUF/resolve/main/llama-2-7b.Q3_K_L.gguf

smaller model does not work

Endpoint to send prompt

POST 127.0.0.1:9000/v1/completions
Request body

{
  "prompt": "\n\n### Instructions:\nWhat is the capital of France?\n\n### Response:\n",
  "stop": [
    "\n",
    "###"
  ]
}

Successful Response

{
  "id": "string",
  "object": "text_completion",
  "created": 0,
  "model": "string",
  "choices": [
    {
      "text": "string",
      "index": 0,
      "logprobs": {
        "text_offset": [
          0
        ],
        "token_logprobs": [
          0,
          null
        ],
        "tokens": [
          "string"
        ],
        "top_logprobs": [
          {
            "additionalProp1": 0,
            "additionalProp2": 0,
            "additionalProp3": 0
          },
          null
        ]
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 0,
    "completion_tokens": 0,
    "total_tokens": 0
  }
}

Validation Error

{
  "detail": [
    {
      "loc": [
        "string",
        0
      ],
      "msg": "string",
      "type": "string"
    }
  ]
}

[pgronkievitz]

suggestion: I'd add curl instructions as well (pro tip is -o flag) - there are systems with no curl and without wget, so this way you won't force anyone to modify the command or to install new package

[pgronkievitz]

nitpick: this doesn't seem to be proper pytest test and it seems like kinda duplicate of test_llm file

[Kleczyk]

this is a quick check of what llm is responding to us sometimes it can be useful

[pgronkievitz]

does it have any advantage over running regular test doing exactly the same?

[pgronkievitz]

in general lgtm

[pgronkievitz]

suggestion (non-blocking): please add both instructions with curl and wget