MacOS M4 - ModuleNotFoundError: No module named 'encodings'

[conapi-stefan]

Problem:
I try to run a Python script from my Java app (similar code works for me on Windows).
I am using a virtual environment for my python setup. This is also set as the default via .zshrc.
Below is the Java code I use and the exact error. I read through all similar issues here but none of the hints worked so far.

Happy to do some debugging or troubleshooting if that helps improving jep.

Setup steps taken:
python -m venv --system-site-packages /Users/stefan/venvs/default
source ~/venvs/default/bin/activate
pip install jep
pip list
--> shows jep 4.2.1
Java is 17 or 21, tested both

 .zshrc:
     export PATH="$(brew --prefix)/opt/python@3/libexec/bin:$PATH"
     # Created by `pipx` on 2024-11-19 07:43:54
     export PATH="$PATH:/Users/stefan/.local/bin"
     source ~/venvs/default/bin/activate

Java sample used to reproduce:

public class TestPython {
    public static void main(String[] args) {
        try {
            String home="/Users/stefan/venvs/default";
            String jepPath = "/Users/stefan/venvs/default/lib/python3.13/site-packages/jep/libjep.jnilib";

            PyConfig pyConfig = new PyConfig();
            pyConfig.setPythonHome(home); // if commented --> no module named 'jep'

            MainInterpreter.setInitParams(pyConfig);
            MainInterpreter.setJepLibraryPath(jepPath);

            SubInterpreter jep = new SubInterpreter(new JepConfig());

        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

Error:

> Task :messagemanager-scripts:TestPython.main() FAILED
Fatal Python error: Failed to import encodings module
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x000000016f4cb000 (most recent call first):
  <no Python frame>

Environment (please complete the following information):

• OS Platform, Distribution, and Version: MacOS 15.1, M4 ARM
• Python Distribution and Version: 3.13.0
• Java Distribution and Version: Amazon Corretto 17.0.13.11.1 (also tried Java 21)
• Jep Version: 4.2.1
• Python packages used (e.g. numpy, pandas, tensorflow): pip list output: jep 4.2.1, packaging 24.2, pip 24.2, wheel 0.44.0

Not sure it matters or is related, but I noticed that the jep script in ~/venvs/default/bin is not correct for my machine. there is no jep jar in the referenced pipx jep dir (/Users/stefan/.local/pipx/venvs/jep/lib/python3.13/site-packages/jep) . This folder does not exist and therefore running jep causes a ClassNotFoundException.
If I set jep_dir in that script to /Users/stefan/venvs/default/lib/python3.13/site-packages/jep (where the jar is), then i get:

jep.JepException: <class 'ModuleNotFoundError'>: No module named 'jep'
	at <string>.<module>(<string>:1)
	at jep.Jep.eval(Native Method)
	at jep.Jep.eval(Jep.java:328)
	at jep.Jep.configureInterpreter(Jep.java:193)
	at jep.SharedInterpreter.configureInterpreter(SharedInterpreter.java:68)
	at jep.Jep.<init>(Jep.java:172)
	at jep.SharedInterpreter.<init>(SharedInterpreter.java:60)
	at jep.Run.run(Run.java:49)
	at jep.Run.main(Run.java:146)

JEP Script

#!/bin/sh

PYTHONHOME="/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.13"; export PYTHONHOME

jep_dir=/Users/stefan/.local/pipx/venvs/jep/lib/python3.13/site-packages/jep

if [ ! -d "$jep_dir" ]
then
    jep_dir=/Users/stefan/.local/pipx/venvs/jep/lib/python3.13/site-packages/jep-4.2.1-py3.13-macosx-15.0-arm64.egg/jep
fi

cp="$jep_dir/jep-4.2.1.jar"
if test "x$CLASSPATH" != "x"; then
    cp="$cp":"$CLASSPATH"
fi

jni_path=$jep_dir

args=$*
if test "x$args" = "x"; then
  args="$jep_dir/console.py"
fi

exec java -classpath "$cp" -Djava.library.path="$jni_path" jep.Run $args
```

[bsteffensmeier]

I don't have time to look closely right now but some quick thoughts to help you troubleshoot

1. I think the encodings error happens anytime you set python home in a venv, I am pretty sure those options conflict within python so I recommend removing the java code that sets python home
2. When we generate the jep script we have code that detects a vent and sets PYTHONEXECUTABLE. Since PYTHONEXECUTABLE is not set in your jep script it looks like it was not built in a vent. Maybe this is a pip cache issue? You could try setting PYTHONEXECUTABLE to the location of python and that might help python resolve the location of your packages without PYTHONHOME.

[conapi-stefan]

Thanks @bsteffensmeier but if I do not set Python home in my Java code, then I get:

No module named 'jep'

[bsteffensmeier]

Thanks @bsteffensmeier but if I do not set Python home in my Java code, then I get:

No module named 'jep'

By setting python home you are causing errors earlier in initialization and don't make it to the code that tries to load jep. I am able to replicate the encoding problems in python without jep by using a venv and setting python home. Since the behavior of jep in a venv with python home set matches the behavior of python I do not think this is a problem jep should solve.

bsteffensmeier@mac-mini ~ % python3 -m venv --system-site-packages /Users/bsteffensmeier/venvs/default
bsteffensmeier@mac-mini ~ % source ~/venvs/default/bin/activate 

(default) bsteffensmeier@mac-mini ~ % export PYTHONHOME=/Users/bsteffensmeier/venvs/default 
(default) bsteffensmeier@mac-mini ~ % python3
Fatal Python error: Failed to import encodings module
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00000001eee37ac0 (most recent call first):
  <no Python frame>

After I pip3 install jep in this environment(without PYTHONHOME set) I am able to run the jep script and get an interpreter. Below is the jep script that was autogenerated:

#!/bin/sh

. /Users/bsteffensmeier/venvs/default/bin/activate
export PYTHONEXECUTABLE="/Users/bsteffensmeier/venvs/default/bin/python3.13"




jep_dir=/Users/bsteffensmeier/venvs/default/lib/python3.13/site-packages/jep

if [ ! -d "$jep_dir" ]
then
    jep_dir=/Users/bsteffensmeier/venvs/default/lib/python3.13/site-packages/jep-4.2.1-py3.13-macosx-10.13-universal2.egg/jep
fi

cp="$jep_dir/jep-4.2.1.jar"
if test "x$CLASSPATH" != "x"; then
    cp="$cp":"$CLASSPATH"
fi

jni_path=$jep_dir

args=$*
if test "x$args" = "x"; then
  args="$jep_dir/console.py"
fi

exec java -classpath "$cp" -Djava.library.path="$jni_path" jep.Run $args

Since jep detected a venv during install it is activating the venv and also setting PYTHONEXECUTABLE, both steps your jep script is missing. When I remove the line that set PYTHONEXECUTABLE then I get No module named 'jep', just like you, so setting PYTHONEXECUTABLE will be required to run jep in a venv on a Mac. I am not sure why Mac needs this when linux doesn't but there is quite a bit of Mac specific code in cpython when it is resolving paths and a suspect somewhere in there it is adding behavior that is inconsistent.

[conapi-stefan]

@bsteffensmeier Thanks a lot for this test!

If I do not call pyConfig.setPythonHome in Java an set PYTHONEXECUTABLE environment variable before running my Java program it works.

Questions:

Is there a way to set PYTHONEXECUTABLE via i.e. PyConfig?
(My real code tries to detect the jep library on the path, then calls setJepLibraryPath. I want to avoid setting PYTHONEXECUTABLE outside the Java app)

Is it a separate defect/issue that "pip install jep" does not generate a proper script on MacOS?

[bsteffensmeier]

Is there a way to set PYTHONEXECUTABLE via i.e. PyConfig? (My real code tries to detect the jep library on the path, then calls setJepLibraryPath. I want to avoid setting PYTHONEXECUTABLE outside the Java app)

There is currently no way to set PYTHONEXECUTABLE from Java. Looking through the Python c-api I don't see an api for setting that exact option either. In the c-api there is an option to set the program name and it looks like this could be used for the same purpose. We have already had a request to set the program name(#515) and I plan to make that available from java in a future release which may help you.

In the mean time the "No module named jep" seems to just be a problem with sys.path, you should be able to add the site-packages to the path manually before creating an interpreter by using JepConfig.setIncludePath(). I was able to start an interpreter in a venv on a Mac without setting PYTHONEXECUTABLE by calling SharedInterpreter.setConfig(new JepConfig().setIncludePath("/Users/bsteffensmeier/venvs/default/lib/python3.13/site-packages/")); before I started my interpreter.

Is it a separate defect/issue that "pip install jep" does not generate a proper script on MacOS?

The problem is that pip will cache any package you install and reinstall the cached package if you install again, even if the install is in a different venv or not in a venv at all. In your case it looks like your initial build was not in a venv and when you later installed in a venv it reused the jep script without the venv options. I can't find anyway to control this behavior for our project, it is a pip setting and something that they haven't done until recently. If you uninstall jep and install again using the --no-cache-dir option with pip then it should give you a proper jep script. I will update the readme to mention the --no-cache-dir option.

[conapi-stefan]

Thanks again!!

It works perfectly now with setIncludePath and jep script works also after reinstalling with the no cache dir open!