Cannot load libllama from Mac with Intel CPU

[mayuanyang]

Is the provided libllama.dll only working for Mac with M1 M2 CPU?

Got the following error when trying to load it on Mac with Intel CPU, however it is working fine on Mac with M2 CPU

System.TypeInitializationException: The type initializer for 'LLama.Native.NativeApi' threw an exception.
---> LLama.Exceptions.RuntimeError: The native library cannot be found. It could be one of the following reasons:

1. No LLamaSharp backend was installed. Please search LLamaSharp.Backend and install one of them.

2. You are using a device with only CPU but installed cuda backend. Please install cpu backend instead.

3. The backend is not compatible with your system cuda environment. Please check and fix it. If the environment is expected not to be changed, then consider build llama.cpp from source or submit an issue to LLamaSharp.

4. One of the dependency of the native library is missed.

   at LLama.Native.NativeApi..cctor() in /Users/eddyma/Github/LLamaSharp/LLama/Native/NativeApi.cs:line 36
   --- End of inner exception stack trace ---
   at LLama.Native.NativeApi.llama_max_devices()
   at LLama.Abstractions.TensorSplitsCollection..ctor() in /Users/eddyma/Github/LLamaSharp/LLama/Abstractions/IModelParams.cs:line 111
   at LLama.Common.ModelParams..ctor(String modelPath) in /Users/eddyma/Github/LLamaSharp/LLama/Common/ModelParams.cs:line 89
   at LLama.WebAPI.Services.StatefulChatService..ctor(IConfiguration configuration) in /Users/eddyma/Github/LLamaSharp/LLama.WebAPI/Services/StatefulChatService.cs:line 19
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Span1& arguments, Signature sig, Boolean constructor, Boolean wrapExceptions) at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor2.VisitCallSite(ServiceCallSite callSite, TArgument argument) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope) at Microsoft.Extensions.DependencyInjection.ServiceProvider.CreateServiceAccessor(Type serviceType) at System.Collections.Concurrent.ConcurrentDictionary2.GetOrAdd(TKey key, Func`2 valueFactory)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ServicesModelBinder.BindModelAsync(ModelBindingContext bindingContext)
   at Microsoft.AspNetCore.Mvc.ModelBinding.ParameterBinder.BindModelAsync(ActionContext actionContext, IModelBinder modelBinder, IValueProvider valueProvider, ParameterDescriptor parameter, ModelMetadata metadata, Object value, Object container)

[nappa0326]

Hello. I also have the same issue, and when I asked a question at #58, I received this reply.

[mayuanyang]

Hello. I also have the same issue, and when I asked a question at #58, I received this reply.

thanks for your info, do you have the intel dylib for handy?

[mayuanyang]

I decided to build it myself, for those who need libllama.dylib for Mac Intel, you might try this one
libllama.dylib.zip
Hope this helps
Cheers

[SignalRT]

@mayuanyang , @nappa0326. I'm not able to test this, but I give a try in this branch to generate Intel binaries. If you can give try to the libllama.dylib generated in the deps binaries that you can download from this run:

https://github.com/SignalRT/LLamaSharp/actions/runs/6754571177#artifacts

The library is in the folder macOS-x86_64:

If this works should be possible to include this in the nuget packages. Let me know if it works.

[mayuanyang]

Hi @SignalRT
I tried both the dylib i built and the one you mentioned from the Action, they both seems loading fine, however i got weird result as screenshot below, any thought?

[SignalRT]

@mayuanyang I'm not sure that metal works on Intel CPUs. Theoretically the binary of my compilation should disable METAL, but it seems that it's using metal....

Please try first this solution:

You can explicitly disable GPU inference with the --n-gpu-layers|-ngl 0 command-line argument.

And as second option this solution:

To disable the Metal build at compile time use the LLAMA_NO_METAL=1 flag or the LLAMA_METAL=OFF cmake option.

[mayuanyang]

@SignalRT have tried that, still generate weird output

[SignalRT]

@mayuanyang . I think that this is not related with the intel library. It's related to the llama.cpp version. I have the same result in arm64 if I build the library with the current version. And it works ok with the version:

ggerganov/llama.cpp@207b519

It seems that the changes in the main.yaml to introduce the build to support macOS Intel works, because it loads and gets the same results that I get with the current llama.cpp

To close this issue If you can build the library from the 207b519 commit id from llama.cpp and check if every thing works find it would help a lot.

It will also help to check it with the Intel build metal works and can be deactivated with the gpu layers. It will help to create the right nuget package to the MacOS intel.

[SignalRT]

I can confirm that the garbage in the output is related with the changes in latest llama.cpp. I make a PR #249 with the changes in the source code. If you test the intel binaries with this branch it should work ok.

[mayuanyang]

@SignalRT FYI, I manage to compile the llama.cpp on Mac Intel and run it, things seems to be working fine, hope that helps, happy to submit PR for any help if required :)

[martindevans]

See #258 which is introducing Intel Mac support. It should be included in the next release (very soon).

[SignalRT]

@mayuanyang , Please confirm that the current version solve all the issues with OSX Intel.

[mayuanyang]

it is working now, thanks