Packaging and Maven-Repo artifact deployment

[rdhabalia]

hi,

Thanks for providing java solution for crc32c-sse4.2 using native library. I can see you have uploaded SNAPSHOT version of this artifact at : Sonatype-Staging_maven-repo

However, we are not able to use this solution due to following issues:

1. Library not present into maven-repo: The available jni library in maven-repo is only available for windows OS and we can't use it for "Mac OS" or "Linux OS".

Solutions:
1. Upload jni library to maven repo for all well-known OS: Right now, Maven repo only contains jni for windows so other system can't use this library. So, can you also please upload library for MacOSX and Linux as well.

2. One Packaging of all OS specific JNI library using vagrant: Checkout RocksDB Build it uses vagrant to build jni for all OS environments and packages them into one artifact which you can deploy to maven repo

1. Can't use artifact as it as:
   When we build circe-crc32c-sse42 using maven build:

• It internally generates "jni" library based on OS where we build and it packages this generated jni library under .nar artifact. e.g: circe-crc32c-sse42.nar
• Nar Plugin goal also generates NarSystem.java which tries to load this jni library at runtime using System.loadLibrary("circe-crc32c-sse42-0.1-SNAPSHOT"); which fails obviously because it asks ClassLoader to load library from java.library.path or sun.boot.library.path. And as this jni library is packaged under .nar so, definitely it is not present in either library.path and hence, initialization of the interface which provides API to compute checksum Sse42Crc32C.java fails.

Solution

• So, I created a pull request : which generates circe-crc32c-sse42.jar instead circe-crc32c-sse42.nar artifact and internally loads generated jni library without messing up with library.path and user can use this artifact without performing any additional magic.
  I have also added a test case which cleanly uses Sse42Crc32C.java to compute checksum by simply adding dependency of artifact com.scurrilous:circe-crc32c-sse42:${project.version} and without worrying about configuring library.path.

[trevorr]

Hi Raj,

I appreciate your interest in Circe.

The snapshot release you mention was never intended to be used by anyone.
It was a test of the release process. The reason there is no official
release is the difficulty you describe: there was not an easy way to
include native libraries for all platforms in the release. While I think it
can be manually done, I was looking for a better way when my schedule
filled up with other projects. This Maven NAR thread describes where I
ended up: https://groups.google.com/d/msg/maven-nar/55381nAYeLw/nVYq0SbnnCgJ

I can attempt to perform your first solution to get an initial release
available. However, since it will take some time to re-familiarize myself
with the project and configure build environments, it may be a few weeks
before I can complete this, given my other obligations. Also, since any
builds I create would be on recent OS versions, there would be no guarantee
they would work on older versions. For instance, I often hear of people
running into C library incompatibility when deploying code built with
recent Ubuntu onto older RHEL/Centos instances. Of course, the simplicity
of this native library should make that less likely.

I don't really understand the cause of your .nar vs .jar issue. I'd need to
be more convinced that the approach advocated by the Nar plugin (i.e.
loading .nar artifacts using NarSystem) is fundamentally flawed to consider
replacing it. Also, your pull request has some minor issues, such as using
tabs instead of spaces in new files and adding a redundant test case that
will fail on any machine not reporting support for SSE 4.2 (either due to
lack of processor, or more commonly, VM support). There is already a test
case for SSE 4.2 CRC-32C that only runs if support is detected.

I'm still not sure what the real solution is going forward, as any build
process that doesn't allow any developer to easily build for all platforms
doesn't seem viable. The RocksDB build is an interesting data point, but it
doesn't support Windows. Because the native library is so simple and free
of platform dependencies, it's tempting to try cross-compilation using only
Linux. Cross-compiling for OS X used to be problematic, but this project
looks promising: https://github.com/tpoechtrager/osxcross. The remaining
question would be how well Nar works with cross-compilation.

[rdhabalia]

Hi Trevor,

Thank you for your prompt response.

1. I don't really understand the cause of your .nar vs .jar issue...(i.e. loading .nar artifacts using NarSystem) is fundamentally flawed to consider replacing it...adding a redundant test case..

I have added test-case to explain issue: how NarSystem creates issue while using circe-crc32c-sse42 by a different project and I also explained similar problem it in earlier thread as well.

• when we build circe-crc32c-sse42 it generates .nar artifact which contains generated jni library in it.
• Now, while accessing Sse42Crc32C.java, NarSystem.java tries to load generated jni circe-crc32c-sse42-0.1-SNAPSHOT using following code which you can see in NarSystem.java

System.loadLibrary("circe-crc32c-sse42-0.1-SNAPSHOT");

• It essentially asks ClassLoader to load circe-crc32c-sse42-0.1-SNAPSHOT library from java.library.path or sun.boot.library.path. And of course as this jni is a part of nar package and not present into either library.path, NarSystem class initialization will fail to load it and we can't use Sse42Crc32C.java

So, to resolve this issue:

• we need to unpack nar using nar-plugin and need to pass absolute-path of the directory where nar plugin will unpack the library as a jvm argument: -Djava.library.path=/yh/git/crc/my-dir/nar/circe-crc32c-sse42-0.1-SNAPSHOT-x86_64-MacOSX-gpp-jni/lib/x86_64-MacOSX-gpp/jni
• then only NarSystem can load it correctly and one can use this utility.

Therefore, I created a pull request: which creates a jar and it auto loads library and user don't have to pass -Djava.library.path argument to access the utility. And test-case was essentially part of different artifact which shows how a separate artifact can use utility without performing any additional steps.

change in pull request

• formatted with space rather tabs
• remove test-case as it may fail in OS which doesn't support SSE4.2

Request

• As we are using your other artifacts by pointing to Sonatype-Staging_maven-repo and it is a SNAPSHOT version. So, can you please upload release version for other non-nar artifacts.
• We have done similar jar stuff on top of your utility to use it without performing any additional steps. So, if we can get jni for various OS: uploaded in maven-repo then we can use it directly.

Thanks for your work.