Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AI: Unable to chat. And error occured while building the embedding model. Failed to load PyTorch native library. #12169

Closed
ThiloteE opened this issue Nov 10, 2024 · 1 comment
Closed

AI: Unable to chat. And error occured while building the embedding model. Failed to load PyTorch native library. #12169

ThiloteE opened this issue Nov 10, 2024 · 1 comment
Labels
component: ai Related to AI Chat/Summarization [outdated] type: bug Confirmed bugs or reports that are very likely to be bugs status: has-workaround Bugs where a workaround exist

Comments

@ThiloteE
Copy link
Member

ThiloteE commented Nov 10, 2024 •
edited
Loading

JabRef commit ed0e90e

I am on Windows 10 with an Nvidia Geforce GTX 1060 3GB and an AMD Ryzen 5 5600 that supports AVX and AVX2.
image

Error message:

C:\Prog\Development\JabRef_ThiloteE-JabRef\ThiloteE-JabRef\jabref>gradle run

> Configure project :
Project : => 'org.jabref' Java module

> Task :run
WARNING: Using incubator modules: jdk.incubator.vector
Messages are not initialized before accessing key: Display help on command line options
2024-09-18 02:31:21 [JavaFX Application Thread] org.jabref.gui.theme.ThemeManager.addStylesheetToWatchlist()
INFO: Watching css C:\Prog\Development\JabRef_ThiloteE-JabRef\ThiloteE-JabRef\jabref\build\resources\main\org\jabref\gui\Base.css for live updates
2024-09-18 02:31:21 [JavaFX Application Thread] org.jabref.gui.theme.ThemeManager.updateThemeSettings()
INFO: Not updating theme because it hasn't changed
2024-09-18 02:31:21 [JavaFX Application Thread] org.jabref.gui.theme.ThemeManager.updateThemeSettings()
INFO: Theme set to Theme{type=DEFAULT, name=''} with base css StyleSheet{file:/C:/Prog/Development/JabRef_ThiloteE-JabRef/ThiloteE-JabRef/jabref/build/resources/main/org/jabref/gui/Base.css}
2024-09-18 02:31:21 [pool-2-thread-1] org.jabref.logic.ai.ingestion.model.UpdateEmbeddingModelTask.call()
INFO: Downloading embedding model...
2024-09-18 02:31:22 [JavaFX Application Thread] org.jabref.logic.push.CitationCommandString.from()
INFO: Wrong indexes -1 -1 for completeCiteCommand \parencite. Using default delimiter and suffix.
2024-09-18 02:31:23 [JavaFX Application Thread] org.jabref.gui.StateManager.setActiveDatabase()
INFO: No open database detected
2024-09-18 02:31:25 [pool-2-thread-2] org.jabref.gui.JabRefDialogService.notify()
INFO: Opening: 'C:\server-t-150\Jabref & LatexTests\OneMillionEntries.bib'
2024-09-18 02:31:25 [pool-2-thread-2] org.jabref.gui.JabRefDialogService.notify()
INFO: Opening: 'C:\server-t-150\Jabref & LatexTests\test-Files.bib'
2024-09-18 02:31:25 [pool-2-thread-3] org.jabref.gui.JabRefDialogService.notify()
INFO: Opening: 'C:\server-t-150\Jabref & LatexTests\Chocolate.bib'
2024-09-18 02:31:26 [JavaFX Application Thread] sun.util.logging.internal.LoggingProviderImpl$JULWrapper.log()
WARN: Resource "" not found.
2024-09-18 02:31:27 [pool-2-thread-1] ai.djl.pytorch.jni.LibUtils.downloadPyTorch()
INFO: Downloading https://publish.djl.ai/pytorch/2.4.0/cu124/win-x86_64/native/lib/asmjit.dll.gz ...
2024-09-18 02:31:27 [pool-2-thread-1] ai.djl.pytorch.jni.LibUtils.downloadPyTorch()
INFO: Downloading https://publish.djl.ai/pytorch/2.4.0/cu124/win-x86_64/native/lib/c10_cuda.dll.gz ...
2024-09-18 02:31:28 [pool-2-thread-1] ai.djl.pytorch.jni.LibUtils.downloadPyTorch()
INFO: Downloading https://publish.djl.ai/pytorch/2.4.0/cu124/win-x86_64/native/lib/c10.dll.gz ...
2024-09-18 02:31:28 [JavaFX Application Thread] sun.util.logging.internal.LoggingProviderImpl$JULWrapper.log()
WARN: Resource "" not found.
2024-09-18 02:31:28 [pool-2-thread-1] ai.djl.pytorch.jni.LibUtils.downloadPyTorch()
INFO: Downloading https://publish.djl.ai/pytorch/2.4.0/cu124/win-x86_64/native/lib/caffe2_nvrtc.dll.gz ...
2024-09-18 02:31:28 [JavaFX Application Thread] org.apache.lucene.internal.vectorization.PanamaVectorizationProvider.<init>()
INFO: Java vector incubator API enabled; uses preferredBitSize=256; FMA enabled
2024-09-18 02:31:29 [JavaFX Application Thread] org.apache.lucene.store.MemorySegmentIndexInputProvider.<init>()
INFO: Using MemorySegmentIndexInput with Java 21 or later; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
2024-09-18 02:31:29 [pool-2-thread-1] ai.djl.pytorch.jni.LibUtils.downloadPyTorch()
INFO: Downloading https://publish.djl.ai/pytorch/2.4.0/cu124/win-x86_64/native/lib/cublas64_12.dll.gz ...
2024-09-18 02:31:35 [pool-2-thread-1] ai.djl.pytorch.jni.LibUtils.downloadPyTorch()
INFO: Downloading https://publish.djl.ai/pytorch/2.4.0/cu124/win-x86_64/native/lib/cublasLt64_12.dll.gz ...
2024-09-18 02:32:02 [pool-2-thread-1] ai.djl.pytorch.jni.LibUtils.downloadPyTorch()
INFO: Downloading https://publish.djl.ai/pytorch/2.4.0/cu124/win-x86_64/native/lib/cudart64_12.dll.gz ...
2024-09-18 02:32:03 [pool-2-thread-1] ai.djl.pytorch.jni.LibUtils.downloadPyTorch()
INFO: Downloading https://publish.djl.ai/pytorch/2.4.0/cu124/win-x86_64/native/lib/cudnn_adv64_9.dll.gz ...
2024-09-18 02:32:17 [pool-2-thread-1] ai.djl.pytorch.jni.LibUtils.downloadPyTorch()
INFO: Downloading https://publish.djl.ai/pytorch/2.4.0/cu124/win-x86_64/native/lib/cudnn_cnn64_9.dll.gz ...
2024-09-18 02:32:19 [pool-2-thread-1] ai.djl.pytorch.jni.LibUtils.downloadPyTorch()
INFO: Downloading https://publish.djl.ai/pytorch/2.4.0/cu124/win-x86_64/native/lib/cudnn_engines_precompiled64_9.dll.gz ...
2024-09-18 02:32:51 [pool-2-thread-1] ai.djl.pytorch.jni.LibUtils.downloadPyTorch()
INFO: Downloading https://publish.djl.ai/pytorch/2.4.0/cu124/win-x86_64/native/lib/cudnn_engines_runtime_compiled64_9.dll.gz ...
2024-09-18 02:32:52 [pool-2-thread-1] ai.djl.pytorch.jni.LibUtils.downloadPyTorch()
INFO: Downloading https://publish.djl.ai/pytorch/2.4.0/cu124/win-x86_64/native/lib/cudnn_graph64_9.dll.gz ...
2024-09-18 02:32:53 [pool-2-thread-1] ai.djl.pytorch.jni.LibUtils.downloadPyTorch()
INFO: Downloading https://publish.djl.ai/pytorch/2.4.0/cu124/win-x86_64/native/lib/cudnn_heuristic64_9.dll.gz ...
2024-09-18 02:32:59 [pool-2-thread-1] ai.djl.pytorch.jni.LibUtils.downloadPyTorch()
INFO: Downloading https://publish.djl.ai/pytorch/2.4.0/cu124/win-x86_64/native/lib/cudnn_ops64_9.dll.gz ...
2024-09-18 02:33:06 [pool-2-thread-1] ai.djl.pytorch.jni.LibUtils.downloadPyTorch()
INFO: Downloading https://publish.djl.ai/pytorch/2.4.0/cu124/win-x86_64/native/lib/cudnn64_9.dll.gz ...
2024-09-18 02:33:06 [pool-2-thread-1] ai.djl.pytorch.jni.LibUtils.downloadPyTorch()
INFO: Downloading https://publish.djl.ai/pytorch/2.4.0/cu124/win-x86_64/native/lib/cufft64_11.dll.gz ...
2024-09-18 02:33:24 [pool-2-thread-1] ai.djl.pytorch.jni.LibUtils.downloadPyTorch()
INFO: Downloading https://publish.djl.ai/pytorch/2.4.0/cu124/win-x86_64/native/lib/cufftw64_11.dll.gz ...
2024-09-18 02:33:24 [pool-2-thread-1] ai.djl.pytorch.jni.LibUtils.downloadPyTorch()
INFO: Downloading https://publish.djl.ai/pytorch/2.4.0/cu124/win-x86_64/native/lib/cupti64_2024.1.0.dll.gz ...
2024-09-18 02:33:25 [pool-2-thread-1] ai.djl.pytorch.jni.LibUtils.downloadPyTorch()
INFO: Downloading https://publish.djl.ai/pytorch/2.4.0/cu124/win-x86_64/native/lib/curand64_10.dll.gz ...
2024-09-18 02:33:31 [pool-2-thread-1] ai.djl.pytorch.jni.LibUtils.downloadPyTorch()
INFO: Downloading https://publish.djl.ai/pytorch/2.4.0/cu124/win-x86_64/native/lib/cusolver64_11.dll.gz ...
2024-09-18 02:33:37 [pool-2-thread-1] ai.djl.pytorch.jni.LibUtils.downloadPyTorch()
INFO: Downloading https://publish.djl.ai/pytorch/2.4.0/cu124/win-x86_64/native/lib/cusolverMg64_11.dll.gz ...
2024-09-18 02:33:42 [pool-2-thread-1] ai.djl.pytorch.jni.LibUtils.downloadPyTorch()
INFO: Downloading https://publish.djl.ai/pytorch/2.4.0/cu124/win-x86_64/native/lib/cusparse64_12.dll.gz ...
2024-09-18 02:33:59 [pool-2-thread-1] ai.djl.pytorch.jni.LibUtils.downloadPyTorch()
INFO: Downloading https://publish.djl.ai/pytorch/2.4.0/cu124/win-x86_64/native/lib/fbgemm.dll.gz ...
2024-09-18 02:34:00 [pool-2-thread-1] ai.djl.pytorch.jni.LibUtils.downloadPyTorch()
INFO: Downloading https://publish.djl.ai/pytorch/2.4.0/cu124/win-x86_64/native/lib/libiomp5md.dll.gz ...
2024-09-18 02:34:01 [pool-2-thread-1] ai.djl.pytorch.jni.LibUtils.downloadPyTorch()
INFO: Downloading https://publish.djl.ai/pytorch/2.4.0/cu124/win-x86_64/native/lib/libiompstubs5md.dll.gz ...
2024-09-18 02:34:01 [pool-2-thread-1] ai.djl.pytorch.jni.LibUtils.downloadPyTorch()
INFO: Downloading https://publish.djl.ai/pytorch/2.4.0/cu124/win-x86_64/native/lib/nvJitLink_120_0.dll.gz ...
2024-09-18 02:34:04 [pool-2-thread-1] ai.djl.pytorch.jni.LibUtils.downloadPyTorch()
INFO: Downloading https://publish.djl.ai/pytorch/2.4.0/cu124/win-x86_64/native/lib/nvrtc-builtins64_124.dll.gz ...
2024-09-18 02:34:04 [pool-2-thread-1] ai.djl.pytorch.jni.LibUtils.downloadPyTorch()
INFO: Downloading https://publish.djl.ai/pytorch/2.4.0/cu124/win-x86_64/native/lib/nvrtc64_120_0.dll.gz ...
2024-09-18 02:34:07 [pool-2-thread-1] ai.djl.pytorch.jni.LibUtils.downloadPyTorch()
INFO: Downloading https://publish.djl.ai/pytorch/2.4.0/cu124/win-x86_64/native/lib/nvToolsExt64_1.dll.gz ...
2024-09-18 02:34:07 [pool-2-thread-1] ai.djl.pytorch.jni.LibUtils.downloadPyTorch()
INFO: Downloading https://publish.djl.ai/pytorch/2.4.0/cu124/win-x86_64/native/lib/torch_cpu.dll.gz ...
2024-09-18 02:34:13 [pool-2-thread-1] ai.djl.pytorch.jni.LibUtils.downloadPyTorch()
INFO: Downloading https://publish.djl.ai/pytorch/2.4.0/cu124/win-x86_64/native/lib/torch_cuda.dll.gz ...
2024-09-18 02:35:02 [pool-2-thread-1] ai.djl.pytorch.jni.LibUtils.downloadPyTorch()
INFO: Downloading https://publish.djl.ai/pytorch/2.4.0/cu124/win-x86_64/native/lib/torch.dll.gz ...
2024-09-18 02:35:03 [pool-2-thread-1] ai.djl.pytorch.jni.LibUtils.downloadPyTorch()
INFO: Downloading https://publish.djl.ai/pytorch/2.4.0/cu124/win-x86_64/native/lib/uv.dll.gz ...
2024-09-18 02:35:03 [pool-2-thread-1] ai.djl.pytorch.jni.LibUtils.downloadPyTorch()
INFO: Downloading https://publish.djl.ai/pytorch/2.4.0/cu124/win-x86_64/native/lib/zlibwapi.dll.gz ...
2024-09-18 02:35:06 [JavaFX Application Thread] org.jabref.logic.ai.ingestion.model.JabRefEmbeddingModel.lambda$startRebuildingTask$1()
ERROR: An error occurred while building the embedding model: ai.djl.engine.EngineException: Failed to load PyTorch native library
        at [email protected]/ai.djl.pytorch.engine.PtEngine.newInstance(PtEngine.java:90)
        at [email protected]/ai.djl.pytorch.engine.PtEngineProvider.getEngine(PtEngineProvider.java:41)
        at [email protected]/ai.djl.engine.Engine.getEngine(Engine.java:190)
        at [email protected]/ai.djl.Model.newInstance(Model.java:99)
        at [email protected]/ai.djl.repository.zoo.BaseModelLoader.createModel(BaseModelLoader.java:224)
        at [email protected]/ai.djl.repository.zoo.BaseModelLoader.loadModel(BaseModelLoader.java:165)
2024-09-18 02:35:06 [JavaFX Application Thread] org.jabref.gui.JabRefDialogService.notify()
INFO: An error occurred while building the embedding model
        at [email protected]/ai.djl.repository.zoo.Criteria.loadModel(Criteria.java:151)
        at [email protected]/org.jabref.logic.ai.ingestion.model.DeepJavaEmbeddingModel.<init>(DeepJavaEmbeddingModel.java:23)
        at [email protected]/org.jabref.logic.ai.ingestion.model.UpdateEmbeddingModelTask.call(UpdateEmbeddingModelTask.java:65)
        at [email protected]/org.jabref.logic.ai.ingestion.model.UpdateEmbeddingModelTask.call(UpdateEmbeddingModelTask.java:20)
        at [email protected]/org.jabref.gui.util.UiTaskExecutor$1.call(UiTaskExecutor.java:188)
        at javafx.graphics@23/javafx.concurrent.Task$TaskCallable.call(Task.java:1401)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.UnsatisfiedLinkError: C:\Users\Thilo\.djl.ai\pytorch\2.4.0-cu124-win-x86_64\torch_cuda.dll: Can't find dependent libraries
        at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2418)
        at java.base/java.lang.Runtime.load0(Runtime.java:852)
        at java.base/java.lang.System.load(System.java:2025)
        at [email protected]/ai.djl.pytorch.jni.LibUtils.loadNativeLibrary(LibUtils.java:379)
        at [email protected]/ai.djl.pytorch.jni.LibUtils.loadLibTorch(LibUtils.java:195)
        at [email protected]/ai.djl.pytorch.jni.LibUtils.loadLibrary(LibUtils.java:82)
        at [email protected]/ai.djl.pytorch.engine.PtEngine.newInstance(PtEngine.java:53)
        ... 17 more

This was a Windows issue.
I did install the newest Visual C++ redistributable from Microsoft and restarted and I also deleted the .djl.ai folder in my home directory, as adviced. Both of which did not solve the issue.

I then used Dependency Walker on torch_cuda.dll. It found that lots of dependencies were not satisfied, but dependency walker is a little outdated. See https://stackoverflow.com/a/36244483/873282. One should use https://github.com/lhak/Dependencies IMHO.

Funny enough, this is a regression introduced by commit c280eb7 (#11773). That one upgrades ai.djl:pytorch:pytorch-model-zoo from 0.29 to 0.30, which comes with different pytorch cuda dlls. Version 0.29.0 still works. Version 0.30.0 does not.

JabRef documenation:

DJL Documentation:

DJL related issues:
@ThiloteE ThiloteE added component: ai Related to AI Chat/Summarization [outdated] type: bug Confirmed bugs or reports that are very likely to be bugs status: has-workaround Bugs where a workaround exist labels Nov 10, 2024
@ThiloteE
Copy link
Member Author

Solutions:

  • A) Have no CUDA installed on your system (and have no CUDA paths set in your systems Environment Variables), then it should automatically detect the correct location of the CUDA path that is managed by JabRef (and DeepJavaLibrary (djl) and the system will also somehow find all dependencies.

  • B) (If you have multiple CUDA installed on your system) Add C:\Users\USER\.djl.ai\pytorch\2.4.0-cu124-win-x86_64 to "PATH" in Windows 10 Environment Variables. Can be done via searching for environment variables and manually adding the folder path. See https://www.howtogeek.com/787217/how-to-edit-environment-variables-on-windows-10-or-11/

Explanation about what happened to me:

My hypothesis about what happened: Since I had multiple CUDAs installed on my system and I had not set the PATH and System Environment Variables, the embedding model was not functioning and only the LLM was fully functional, which made it seem like embeddings were not sent to GPT4All, while in reality, no embeddings had ever been created in the first place. I confirmed LLMs being functional, while testing local API servers like GPT4All, Ollama or llama.cpp, as reported in issue #12114

I also had lots of x86 Microsoft Visual C++ Redistributables installed, which are not needed on my x64 system and that also might have caused some conflicts, but the main issue was the path issue, which caused the embeddings model to not function.

Links and comments that helped me find the answers:

This was referenced Nov 10, 2024
Merged
Merged
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
component: ai Related to AI Chat/Summarization [outdated] type: bug Confirmed bugs or reports that are very likely to be bugs status: has-workaround Bugs where a workaround exist
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ThiloteE