deepseek R1 distill Qwen 1.5B INT4-NPU or FP16

[susantowijaya]

Hello,

I am using openvino notebooks https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/deepseek-r1/deepseek-r1.ipynb on NPU Lunar Lake, core ultra 7 with 32GB of Memory in Windows 11

The prompt is click provided by gradio, after the 5th prompt is stop responding... can help try or test it out?, use the latest one, basically just input any prompt and then the 5th prompt will stop responding... if possible can help fix it due I will use openvino as a demo. Thanks

[brmarkus]

On my Laptop (Intel Core Ultra 7 155H, 64GB system memory, Python v3.12.4, MS-Win11-Pro 24H2), using a new (deleted) Python virtual environment, sync'ed this Notebook repository.
Selected "NPU", "DeepSeek-R1-Distill-Qwen-1.5B", kept "English" language, kept "INT4-NPU".
Downloading the model, conversion, compression took several minutes.
Then running the cell in section "Instantiate pipeline with OpenVINO Generate API" with the given prompt "What is OpenVINO?".
Watching NPU load in TaskManager to see whether the NPU is really used - yes, it is:

Input text: What is OpenVINO?
I need to find information about it.

I need to find information about OpenVino, specifically the details of the framework, its purpose, and the technologies it uses. I also need to know the release date and the version of the framework. Additionally, I should find out the technologies and tools that are used in its implementation. I also need to find the technologies

Then changing the prompt several times and check if a plausible answer is returned:

• "What is DL-Streamer?"
  Getting strange result:

Input text: What is DL-Streamer?
two体 bet{ the attendantb4...... refactor1 the2... fibonacci... the � coli: On the... cons{ fo4... (( were)()(0 ...(...) :: (:(...: was was :*(. among... the �: the �:(ymbol0600 ... (em... (wh * ... * ( ( ( ): ( (::: *::otope:::::bi ... *::;... ......

6 ... *
::: * ... ...: ... *
.:.....

• "What can a Jupyter Notebook be used for?"
  Also getting a strange result:

Input text: What can a Jupyter Notebook be used for?
00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

• Testing the original prompt "What is OpenVINO?"
  Getting a strange result:

Input text: What is OpenVINO?
in in / / / / in in / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / in / / / / / / / / / / / / / / / / / in / / / / / / / / / / / / / / / / / / / / / / / /

Probably it's not a good idea to just run the same cell again multiple times, but to reset the pipeline instead of re-creating it whenever the cell gets executed again?

[susantowijaya]

you are correct, I press clear everytime for new questions, no issue... it's weird, but solved the issue