Android Backend Support #717

AmSmart · 2024-05-04T23:44:23Z

As discussed in #695, Here's a PR that generates the shared library for Android. I have also added the new binaries to be copied in the CPU Backend .nuspec file. Please let me know if I am missing anything else.

I have tested the generated .so by loading manually into a .NET MAUI app which ran smoothly on my Android device. See video below

Screen_Recording_20240427_141010.mp4

martindevans · 2024-05-05T00:08:24Z

Looks good, thanks for putting this together.

Could you please manually trigger a run of the "Update Binaries" action on your repo as a test run. To do that:

Go to actions (top bar of your repo)
Select "Update Binaries" on the left
Click "Run Workflow" at the top of the run list (right side)
Change the branch to add-android-support
Click run workflow
Link the run here

AsakusaRinne

Thank you for this contribution! The overall looks good except one concern. I prefer to putting the android native libraries in a new package LLamaSharp.Backend.Android because those who want to use android will never use libraries of win/linux/osx. For those who want to publish a cross-platform app, they could install both LLamaSharp.Backend.Android and LLamaSharp.Backend.Cpu. Please refer to #489 to see how to add a new backend package.

The following items will further improve it but are not required in this PR.

Add a workflow to run test in android. I know little about running C# apps on android but it seems that github action has already provided such support (reference).
Add logics to support dynamic native library selection on android.

I'd like to work with item 2 after having this PR merged so please don't worry about it. :) Would you like to work with item 1 (in this PR or after this PR are both okay).

AmSmart · 2024-05-07T13:00:38Z

Totally aligned. I was pondering that it deserved its own backend as well but wasn't sure. Will work on these over the next few days. Should share an update soon.

PS: Prefer to finish it up in this PR

cmpktheo · 2024-05-13T12:02:12Z

Could you upload the MAUI app's source with the android runtime libraries to test it?

AmSmart · 2024-05-24T04:16:17Z

Hi @cmpktheo Attaching a binary compliled for android-arm64-v8a

libllama.zip

cmpktheo · 2024-05-24T15:48:02Z

Thanks @AmSmart !
I have been trying to build llama.cpp on a linux VM and test a MAUI Android app with LlamaSharp using a Visual Studio x86_64 emulator.

I followed the android build instructions on llama.cpp page, but when I try to use the .so file, I get a null pointer calling the llama_model_default_params method (which basically returns a dummy structure) . It makes me think that there's either a flaw in my build process or some issue related to MAUI. This is my first attempt to build anything in MAUI so I might miss something.
I've created a windows MAUI app that loads the GPT2 guff model and works perfectly.

I would appreciate if you could share the build process so I can apply it to x86_64 architecture or a ready build x86_64 runtime and ideally a MAUI-Android sample app.

martindevans · 2024-05-24T17:56:24Z

@cmpktheo did you ensure you're using the right version of llama.cpp? The C# source is only compatible with one exact version! See the table in the readme here for the correct versions.

cmpktheo · 2024-05-27T09:17:45Z

@martindevans the issue seems to be with my build process since the native method returns null pointer.

AmSmart · 2024-05-29T20:54:40Z

@cmpktheo You can check the compile.yml file from my PR. I've updated it to include other Android architectures.

AmSmart · 2024-05-29T21:30:23Z

@AsakusaRinne I've been a bit busy since our last conversation but I've finally gotten some time to address some of the concerns you listed earlier and made some other changes as well.

Things to note

I have created a different Nuspec for the Android backend as discussed
I noticed there was a .targets file but it wasn't been used by any of the generated packages (I double checked this by unzipping existing packages on Nuget). That said, I needed the targets file to Embed the library and link for Android so I removed all the dormant build directives in the file and have just the Android ones in it. I plan to open another PR for iOS and it would need the targets file as well. If you do not like these change, or need a bit more clarification. I'm happy to explain more
Since Android doesn't support additional enhancements (CUDA, CUBLAS, AVX etc) at the moment, there is really no need to manually resolve the native library. I have added an if statement to exclude Android from the Manual Resolution in NativeApi.Load.cs
Updated the compile.yml file to remove file extensions from directory names. I think it may confuse new contributors who may confuse the directories for files (I was confused for a moment).
With regards to providing tests, I'm thinking of a world where we have a Demo/Example .NET Android Project instead. The UnitTests already provide coverage for the underlying code and a demo app should suffice. That said, we could still proceed to write tests against the demo app and have them run in the workflow. Let me know what you think, I'm happy to contribute this in another PR.

Let me know if there are anymore requested changes for this PR to be good to go.

AsakusaRinne

Thank you for your effort! It looks good to me except a few minor issues.

With regards to providing tests, I'm thinking of a world where we have a Demo/Example .NET Android Project instead. The UnitTests already provide coverage for the underlying code and a demo app should suffice. That said, we could still proceed to write tests against the demo app and have them run in the workflow. Let me know what you think, I'm happy to contribute this in another PR.

Sure, the workflow is not required in this PR. It's a good idea to add an example/demo first. The main motivation for the android test workflow is that our core team members do not have android environment to test it. Thus it will be hard for us to know whether a PR breaks this feature. But no hurry, adding an example is a good first step. :)

Since Android doesn't support additional enhancements (CUDA, CUBLAS, AVX etc) at the moment, there is really no need to manually resolve the native library. I have added an if statement to exclude Android from the Manual Resolution in NativeApi.Load.cs

The native library loading system is also responsible for deciding the file path to load it. However, please don't worry, I'll do it after this PR.

AsakusaRinne · 2024-05-30T17:21:06Z

LLama/LLamaSharp.Runtime.targets


-      <None Include="$(MSBuildThisFileDirectory)runtimes/deps/llama.dll">


These targets are used in the unittest and example projects, for example:

https://github.com/SciSharp/LLamaSharp/blob/master/LLama.Examples/LLama.Examples.csproj#L2

https://github.com/SciSharp/LLamaSharp/blob/master/LLama.Unittest/LLama.Unittest.csproj#L2

Therefore, though I'm not sure whether this targets file should also be used in other backend packages, I don't think we should remove it. May I ask which behavior will be different if a targets file is used in the backend package?

AsakusaRinne · 2024-05-30T17:22:30Z

LLama/Native/NativeApi.Load.cs

@@ -57,6 +57,11 @@ private static void SetDllImportResolver()
            // NativeLibrary is not available on older runtimes. We'll have to depend on
            // the normal runtime dll resolution there.
 #if NET5_0_OR_GREATER
+
+            // We don't need special dll resolution on Android
+            if (OperatingSystem.IsAndroid())


The master branch has refactored the native library loading system so I'm afraid this code would not work. Don't worry, I can do it after having this PR merged.

AsakusaRinne · 2024-05-30T17:31:41Z

.github/workflows/compile.yml

@@ -51,12 +51,12 @@ jobs:
      - uses: actions/upload-artifact@v4
        with:
          path: ./build/libllama.so
-          name: llama-bin-linux-${{ matrix.build }}-x64.so
+          name: llama-bin-linux-${{ matrix.build }}-x64


@martindevans Could you please take a look of this? I'm not sure what it's expected to do with the original code.

cmpktheo · 2024-06-07T16:50:02Z

@cmpktheo You can check the compile.yml file from my PR. I've updated it to include other Android architectures.

@AmSmart , Your compiled .so file for the android-arm64-v8a architecture, is ~12 MB.
When I run the Update Binaries workflow, the resulting .so file for the same architecture is ~17MB.
I am asking because, when I use the compiled .so file for the x86_64 architecture I get a [libc] Fatal signal 11 error, which means that there's an issue with the build process or MAUI/.NET.

How did you get the 12MB version which is considerably smaller than my run of Update Binaries workflow?

martindevans · 2024-07-21T14:44:19Z

Since this PR has been stalled for a while I've taken the first step (generating the binaries) and added it in #861, hopefully we can start to make some progress on Android support :)

martindevans · 2024-07-24T21:08:29Z

Some of this work was pulled out into a separate PR (#861). The rest of the work will need to be updated to fix the conflicts in new PRs, so I'll close this one. It should still serve as a handy reference for future PRs.

AmSmart · 2024-07-25T19:06:04Z

Will pick up the rest of the effort in another PR as I find time.

martindevans · 2024-07-25T20:14:46Z

Hopefully I've got the most complex bit out of the way for you with the updated build script :)

AsakusaRinne requested changes May 6, 2024

View reviewed changes

add android support

7ebe28e

AmSmart force-pushed the feature/add-android-support branch from 82e5020 to cb08e1b Compare May 29, 2024 18:58

AmSmart added 2 commits May 29, 2024 20:36

add support for other architectures

8b2c686

add logic to auto load library

c7b6980

AmSmart force-pushed the feature/add-android-support branch from cb08e1b to c7b6980 Compare May 29, 2024 19:37

remove file extension from directory names

0511b96

AmSmart force-pushed the feature/add-android-support branch from 7541e8b to 0511b96 Compare May 29, 2024 21:25

AsakusaRinne requested changes May 30, 2024

View reviewed changes

AsakusaRinne added enhancement New feature or request backend native lib labels May 30, 2024

martindevans mentioned this pull request Jul 21, 2024

Android Build #861

Merged

martindevans closed this Jul 24, 2024

AmSmart deleted the feature/add-android-support branch November 6, 2024 11:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Android Backend Support #717

Android Backend Support #717

AmSmart commented May 4, 2024

martindevans commented May 5, 2024

AsakusaRinne left a comment

AmSmart commented May 7, 2024 •

edited

Loading

cmpktheo commented May 13, 2024

AmSmart commented May 24, 2024 •

edited

Loading

cmpktheo commented May 24, 2024

martindevans commented May 24, 2024

cmpktheo commented May 27, 2024

AmSmart commented May 29, 2024

AmSmart commented May 29, 2024 •

edited

Loading

AsakusaRinne left a comment

AsakusaRinne May 30, 2024

AsakusaRinne May 30, 2024

AsakusaRinne May 30, 2024

cmpktheo commented Jun 7, 2024 •

edited

Loading

martindevans commented Jul 21, 2024

martindevans commented Jul 24, 2024

AmSmart commented Jul 25, 2024

martindevans commented Jul 25, 2024


		<None Include="$(MSBuildThisFileDirectory)runtimes/deps/llama.dll">

Android Backend Support #717

Android Backend Support #717

Conversation

AmSmart commented May 4, 2024

martindevans commented May 5, 2024

AsakusaRinne left a comment

Choose a reason for hiding this comment

AmSmart commented May 7, 2024 • edited Loading

cmpktheo commented May 13, 2024

AmSmart commented May 24, 2024 • edited Loading

cmpktheo commented May 24, 2024

martindevans commented May 24, 2024

cmpktheo commented May 27, 2024

AmSmart commented May 29, 2024

AmSmart commented May 29, 2024 • edited Loading

AsakusaRinne left a comment

Choose a reason for hiding this comment

AsakusaRinne May 30, 2024

Choose a reason for hiding this comment

AsakusaRinne May 30, 2024

Choose a reason for hiding this comment

AsakusaRinne May 30, 2024

Choose a reason for hiding this comment

cmpktheo commented Jun 7, 2024 • edited Loading

martindevans commented Jul 21, 2024

martindevans commented Jul 24, 2024

AmSmart commented Jul 25, 2024

martindevans commented Jul 25, 2024

AmSmart commented May 7, 2024 •

edited

Loading

AmSmart commented May 24, 2024 •

edited

Loading

AmSmart commented May 29, 2024 •

edited

Loading

cmpktheo commented Jun 7, 2024 •

edited

Loading