building jep via mingw-w64 fails with "cannot export PyInit_jep: symbol not defined"

[maveric2]

Hi jep-Team,

I like to build jep with mingw-w64. This is what I prepared:

• download: msys2-x86_64-20240727.exe from https://www.msys2.org/
• install python3, numpy for mingw-w64
• download: https://download.eclipse.org/justj -> org.eclipse.justj.openjdk.hotspot.jre.full.win32.x86_64_21.0.2.v20240123-0840
• gcc-13.1.0 available
• start mingw-w64.exe shell

Now I've downloaded jep-4.2.0.tar.gz and untar it. Entering the jep-4.2.0 directory and called
> python3 setup.py build.

It fails with:
Error: Please set the environment variable JAVA_HOME to a path containing the JDK.

what i expect - now I've set the JAVA_HOME to jre root folder:
> export JAVA_HOME <PATH_TO_JAVA>/jre

Now it starts building, but did not find the compiler (have a external located gcc-13.1.0) and fails with:
error: [WinError 2] the system cannot find the specified file

Setting CC and CXX to the external gcc:
> export CC=<PATH_TO_GCC>/bin/gcc.exe

resolves the issue and building starts. But fails at the very end during the linking phase:
ld.exe: cannot find -ljvm: No such file or directory
ld.exe: cannot find -ldl: No such file or directory

The missing jvm library I resolved by setting the LIBRARY_PATH to the location where jvm.dll is located:
> export LIBRARY_PATH=${JAVA_HOME}/bin/server

Now the -ljvm is found and only -ldl is missing. The libdl is usually a linux library and not existing within a mingw-w64 environment.
So I took the whole gcc command and removed the -ldl in the linking command.
Now the link command proceed a bit further, but I bumped into the following linker error:
ld.exe: cannot export PyInit_jep: symbol not defined

Conclusion
To get rid of the -ldl error it is simply a check/fix within the build flow, that libdl is not linked if the build runs within Windows.
The PyInit_jep issue is caused by missing exports!? - I don't know.

[maveric2]

I proceed a bit further by removing the jep.cp311-mingw_x86_64.def from the link command-line.
Linking went through and I was able to run:
> python3 setup.py install

Within the python3 shell I execute:
>>> help('modules')
Which shows me jep in the available module list.

However importing jep fails with:
>>> import jep Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: DLL load failed while importing jep: Cannot find Module.

Checking the the site-packages folder:
msys64\mingw64\lib\python3.11\site-packages\jep-4.2.0-py3.11-mingw_x86_64.egg\jep
I see a libjep.so which is very unusual for a mingw-w64 build.

Further conclusions
I think the compile flow is not prepared to work with mingw-w64, since I would expect a jep.dll.

[maveric2]

I wondering why there is a libjep.so and no jep.dll. So I checked the build flow and find the root cause:
def is_windows(): return 'win' in get_platform()
This returns false since get_platform() returns mingw_x86_64. So I fixed it by replacing 'win' by 'mingw_x86_64'.
def is_windows(): return 'mingw_x86_64' in get_platform()

This:
def is_windows():
platform = get_platform().lower()
if 'mingw_x86_64' in platform or 'win' in platform:
return True
return False
should cover both cases.

Now the whole build process went through directly to the point where the PyInit_jep symbol is not defined.
(no need to handle the -lld issue).
Removing again build/temp.mingw_x86_64-cpython-311/src/main/c/jep/jep.cp311-mingw_x86_64.def from the link-command-line the process went through:

• jep.cp311-mingw_x86_64.dll
• jep.dll
  exists.

BUT still DLL load failed while importing jep.

I've analyzed the jep.dll with "Dependency Walker" and see the jep.dll depends on jvm.dll as expected. Now I've copied the jvm.dll beside the jep.dll and got the following error:
ImportError: dynamic module does not define module export function (PyInit_jep)

This is because I got rid of the jep.cp311-mingw_x86_64.def while linking I guess. But keeping it causes linking to fail.

[maveric2]

After removing the build/temp.mingw_x86_64-cpython-311/src/main/c/jep/jep.cp311-mingw_x86_64.def from the link-command-line the process went through...

Update:
Now I've executed:
python3 setup.py build
again - which seems to be required and lead to a working jep.
Afterwards I run:
python3 setup.py install

Testing:
python3 -c "import jep"
returns:

Traceback (most recent call last):
		  File "D:/msys_20240827/msys64/mingw64/lib/python3.11/site-packages/jep-4.2.0-py3.11-mingw_x86_64.egg/jep/__init__.py", line 27, in <module>
		    from _jep import *
		ModuleNotFoundError: No module named "_jep"

		During handling of the above exception, another exception occurred:

		Traceback (most recent call last):
		  File "<string>", line 1, in <module>
		  File "D:/msys_20240827/msys64/mingw64/lib/python3.11/site-packages/jep-4.2.0-py3.11-mingw_x86_64.egg/jep/__init__.py", line 29, in <module>
		    raise ImportError("Jep is not supported in standalone Python, it must be embedded in Java.")
		ImportError: Jep is not supported in standalone Python, it must be embedded in Java.

Which I expect, since Jep is not supported in standalone Python - fingers crossed to get it finally working.

[maveric2]

Finally I get jep working under windows using a msys2/msys64/mingw-w64/gcc-13.1.0 toolchain. So the steps above were necessary to getting this done.
However there are two things which are left over to improve the build flow for the future to be able to compile jep smooth:

• modify is_windows() to accept 'win' and 'mingw_x86_64' as a return value of get_platform()
• remove jep.cp311-mingw_x86_64.def within the command-line while linking, or fix it

[ndjensen]

Thank you for sharing your progress and solution! I'm sure others will find it useful.

If you want to submit a pull request we could put in the change to the is_windows() method.

Regarding the linker command line, if you look at Jep's commands/build_ext.py and commands/_msvccompiler.py you can see how we extended and overrode specific functions to get the compilation to work on Windows with MSVC. It's using built-in Python support for building extension modules but slightly altering it to get it to work with Jep. I see distutils, which has now moved into setuptools, has a cygwinccompiler.py. I wonder if that's what's being used when you run the build? There's a link() method in there and also a Mingw32CCompiler class. If that's what's being used, maybe you could extend and override that method to fix the linker command as necessary, similar to how Jep is overriding a step in the MSVC build? Just an idea to consider.