Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for generating native libraries for Linux and macOS aarch64 platforms (fixes #46). #50

Merged
merged 16 commits into from
Jul 23, 2024

Conversation

kirkrodrigues
Copy link
Member

@kirkrodrigues kirkrodrigues commented Jul 21, 2024

Description

This PR will allow users of aarch64 (a.k.a. arm64) platforms like Apple Silicon and ARM processors to use the Java clp-ffi library.

To achieve this, this PR:

  • Adds a GH workflow job (build-lib-for-linux-aarch64) that uses Docker + QEMU to build the native library for aarch64 and upload it to GH Actions storage.
    • To run the build-env setup and compilation steps in the container, this PR also adds a shell script (install-deps-and-build-native-lib.sh).
  • Updates the build-lib-for-macos GH workflow job to also build the native library for Apple Silicon using a macos-14 GH-hosted runner, and upload the new binary to GH Actions storage.
    • This PR also removes the native-lib-build GH action since it was only being used on macOS and it's simple-enough to inline.
  • Updates the build-and-release GH workflow job to download the new binaries and add them to the JAR.
  • Modifies the build-and-release GH workflow job to build and test, add the binaries to the package, and then build the final JAR, rather than unnecessarily building a JAR before adding the additional binaries.
  • Adds a GH job step to upload the final JAR to GH Actions storage for debugging.
  • Adds an integration test which is an independent Maven project that uses the built JAR to run a simple program that checks whether the native library can be loaded and used correctly.
    • This PR also adds an integration-tests task to run this test, renames the current test to unit-tests, and adds a new test task to run unit-tests and integration-tests.

Validation performed

  • Downloaded the JAR built by the GH workflow.
  • Ran task integration-tests to set up and run the integration tests.
  • Copied the downloaded JAR to to target/repo/com/yscope/clp/clp-ffi/0.4-SNAPSHOT
  • Validated that running the integration test manually (without using task integration-tests, since that would overwrite the downloaded JAR) passed on:
    • Ubuntu 20.04 (Focal) on the following platforms:
      • x86_64
      • aarch64
    • Ubuntu 22.04 (Jammy) on the following platforms:
      • x86_64
      • aarch64
    • macOS 14.2.1 on the following platforms:
      • aarch64

@kirkrodrigues kirkrodrigues marked this pull request as draft July 21, 2024 20:06
@kirkrodrigues kirkrodrigues marked this pull request as ready for review July 22, 2024 05:19
@kirkrodrigues
Copy link
Member Author

Verified with @LinZhihao-723's help that the built jar works on an Apple Silicon Mac.

@davemarco
Copy link

davemarco commented Jul 23, 2024

Tested integration test on AWS x86 and arm64 22.04 ubuntu instances with this script. Printed output from test as expected. Did not test mac os as expensive on AWS. Maybe for future PR add test directly in workflow but not required for now.

# Ubuntu setup
sudo apt update
sudo apt install -y build-essential cmake git maven openjdk-11-jdk
# Install task using your preferred method

# Testing steps
git clone [email protected]:y-scope/clp-ffi-java.git
cd clp-ffi-java
git submodule update --init --recursive
task integration-tests
cd target/repo/com/yscope/clp/clp-ffi/0.4-SNAPSHOT/
# Copy the .jar files into the dir above
cd ../../../../../../../integration-tests/jar-load-native-lib
MAVEN_OPTS="-Dmaven.repo.local=$(readlink -f ../../target/repo)" mvn compile
MAVEN_OPTS="-Dmaven.repo.local=$(readlink -f ../../target/repo)" mvn exec:java -Dexec.mainClass="com.yscope.clp.Main"

@kirkrodrigues kirkrodrigues requested a review from davemarco July 23, 2024 19:33
DEBIAN_FRONTEND=noninteractive apt-get install -y build-essential cmake maven

cd "$project_dir"
mvn --batch-mode validate generate-resources test assembly:single@assemble-lib-dir
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I moved validate before generate-resources to match the order that Maven executes them in (see mvn buildplan:list).

@davemarco
Copy link

Tested again integration test had no errors after new changes. Checks all passes

@kirkrodrigues kirkrodrigues merged commit 0afc76a into y-scope:main Jul 23, 2024
6 checks passed
@kirkrodrigues kirkrodrigues deleted the add-arm64-bins branch July 23, 2024 20:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants