introduce ollama cpp optimized for Xeon CPU

[abdulari]

Description

This component introduce ollama cpp compiled with additional flags that suitable for serving llm on Xeon CPU. The additional flags are as follows: DGGML_AVX=on DGGML_AVX2=on DGGML_F16C=on DGGML_FMA=on DGGML_AVX512=on GGML_AVX512_VNNI=on DGGML_AVX512_VBMI=on

Issues

n/a

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds new functionality)
• Breaking change (fix or feature that would break existing design and interface)
• Others (enhancement, documentation, validation, etc.)

Dependencies

• ollama cpp

Tests

• build docker image
• deploy in Xeon k8s cluster
• gather performance

curl http://localhost:11434/api/chat -d '{  "model": "phi3",  "format":"json","stream":false, "messages": [    { "role": "user", "content": "why the sky is blue?" }  ]}' | jq .

{
  "model": "phi3",
  "created_at": "2024-11-04T03:10:46.469048957Z",
  "message": {
    "role": "assistant",
    "content": "{\n   \"type\": \"Answer\",\n   \"detail\": {\n      \"title\": \"Why Is The Sky Blue? - Explanran Answer\",\n      \"description\": \"The color of our sky comes down to a few atmospheric and physics factors. Here's how the process works: \\n\\n- When sunlight enters Earth’s atmosphere, it is composed of different colors that together appear white.\\n- As light travels through space, shorter wavelength blue and violet frequencies are scattered in many directions by oxygen and nitrogen molecules. This scattering effect for shorter wavelengths is known as Rayleigh Scattering.\\n\\nHowever, the sky does not appear violet because our eyes are more sensitive to blue light and less so at violet end of spectrum. Moreover, sunlight contains relatively little violet light to begin with.\",\n      \"source\": \"\"\n   }\n}"
  },
  "done_reason": "stop",
  "done": true,
  "total_duration": 10284471444,
  "load_duration": 4209713070,
  "prompt_eval_count": 15,
  "prompt_eval_duration": 137833000,
  "eval_count": 199,
  "eval_duration": 5887508000
}