feat: cuda feature detection. #275
Conversation
|
Is there a good way for us to print some logs in |
|
I think you can do something like |
LLama/NativeLibraryConfig.cs
Outdated
| config.Desc = new Description(libraryPath); | ||
| } | ||
|
|
||
| /// <summary> |
There was a problem hiding this comment.
I think it'd be better to split this into various WithX methods rather than having them all in one method (more extensible in the future).
e.g.
WithCuda(bool cuda)
WithAvx(AvxLevel avx)
WithAutoFallback(bool allow)
WithSkipCheck(bool skip)There was a problem hiding this comment.
This format seems good, especially for the ability of chained call. There's one problem that it couldn't notice user the default configuration well. If we just set default value to the parameter, it may looks strange when calling WithAutoFallback().WithSkipCheck(false). 😹
Anyway, I prefer this format, too. However is there any solution about the case above?
LLama/NativeLibraryConfig.cs
Outdated
| /// <param name="useCuda"></param> | ||
| /// <param name="avxLevel"></param> | ||
| /// <param name="allowFallback">Whether to allow fall-back when your hardware doesn't support your configuration.</param> | ||
| /// <param name="skipCheck">Whether to skip the check when fallback is allowed. |
There was a problem hiding this comment.
I'm not really sure what skipCheck does from this description. it's "skipping the check", but what check?
There was a problem hiding this comment.
It's to check if the hardware support the user's choice. Let's ignore the skipCheck parameter first, then there're two basic conditions, allowing fallback and the opposite.
If fallback is allowed, we could choose another library if the user's configuration does not match the hardware. For example, the user specified cuda but there's actually no supported cuda version at that machine.
If fallback is disabled, however, we're supposed to load exactly the library specified. However the real condition may be complex and our logic may not cover all the cases (for example, linux cuda detection). Therefore if sometimes loading a library won't actually have problem but we take it as invalid by mistake, the user could force-load it without any check.
I think by this way we could leave a solution for some potential problems in the future.
|
Just a few review nits, but overall looks good. My only criticism is it's less clear than before which binaries are preferred over others. Previously you could just read down the list of |
AsakusaRinne
force-pushed
the
cuda_detection
branch
from
e11df78 to
bbbfbd2
Compare
|
The linux check has passed but the binaries seemed not to work. I can only load the library when I replaced the binaries with self-compiled ones. Not sure if it's related with #270 |
|
@AsakusaRinne I get this exception trying to load LLamaSharp: System.BadImageFormatException: Could not load file or assembly 'LLamaSharp, Version=0.8.0.0, Culture=neutral, PublicKeyToken=null'. An attempt was ma... System.BadImageFormatException at LLama.Unittest.BasicTest..ctor() |
|
@SignalRT It seems to be a problem of LLamaSharp package instead of backend packages. Is your system 32-bit or 64-bit? These packages are the ones generated by github ci workflow, maybe another try with it? |
|
@AsakusaRinne, Arm64 (M2 computer). I can run the test in the project, execute the examples, but with your packages or with the packages generated manually in my computer with prepare_release.sh it fails with the same error. I'm trying to understand the problem....that seems a little weird. |
|
@AsakusaRinne After a lot of problems, it seems that it's related to some problem with nuget cache. I clear the cache and it works with the packages generated in the CI pipeline |
Wow, that's nice! I think we could release it tonight. @martindevans The CUDA libraries have been replaced the ones with avx 2, is that right? |
Yep, the CUDA binaries should now compile with AVX2 by default. |
This PR is supposed to be merged after #258
Note that it adds two APIs for users, which are
NativeLibraryConfig.WithLibraryandNativeLibraryConfig.WithMatchRule. To be honest I think the naming is not good but I can't come up with a better one. Any idea?TODO: