Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Need to downgrade py4j during install #401

Open
jpdna opened this issue Jun 26, 2018 · 4 comments
Open

Need to downgrade py4j during install #401

jpdna opened this issue Jun 26, 2018 · 4 comments

Comments

@jpdna
Copy link
Member

jpdna commented Jun 26, 2018

When I attempt to install from Master as below, I get the following error:

git clone [email protected]:bigdatagenomics/mango.git
virtualenv venv
source venv/bin/activate
cd mango/mango-viz/
make prepare
cd ../mango-python/
make prepare
cd ..
export SPARK_HOME=/Users/paschalj/test9/apr10_mango_graph/spark-2.2.1-bin-hadoop2.7
export MANGO_HOME=/Users/paschalj/test9/june14_mango_master/mango

PY4J_ZIP="$(ls -1 "${SPARK_HOME}/python/lib" | grep py4j)"
export PYTHONPATH=${SPARK_HOME}/python:${SPARK_HOME}/python/lib/${PY4J_ZIP}:${PYTHONPATH}
mvn clean package -DskipTests

ASSEMBLY_DIR="${MANGO_HOME}/mango-assembly/target"
ASSEMBLY_JAR="$(ls -1 "$ASSEMBLY_DIR" | grep "^mango-assembly[0-9A-Za-z\_\.-]*\.jar$" | grep -v javadoc | grep -v sources || true)"
export PYSPARK_SUBMIT_ARGS="--jars ${ASSEMBLY_DIR}/${ASSEMBLY_JAR} --driver-class-path ${ASSEMBLY_DIR}/${ASSEMBLY_JAR} pyspark-shell"

mvn clean package -P python -DskipTests

...
[INFO] --- exec-maven-plugin:1.6.0:exec (dev-python) @ mango-python ---
pip install -e .
Obtaining file:///Users/paschalj/test9/june26_mango/mango/mango-python
Exception:
Traceback (most recent call last):
  File "/Users/paschalj/test9/june26_mango/venv/lib/python2.7/site-packages/pip/_internal/basecommand.py", line 228, in main
    status = self.run(options, args)
  File "/Users/paschalj/test9/june26_mango/venv/lib/python2.7/site-packages/pip/_internal/commands/install.py", line 291, in run
    resolver.resolve(requirement_set)
  File "/Users/paschalj/test9/june26_mango/venv/lib/python2.7/site-packages/pip/_internal/resolve.py", line 103, in resolve
    self._resolve_one(requirement_set, req)
  File "/Users/paschalj/test9/june26_mango/venv/lib/python2.7/site-packages/pip/_internal/resolve.py", line 257, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/Users/paschalj/test9/june26_mango/venv/lib/python2.7/site-packages/pip/_internal/resolve.py", line 194, in _get_abstract_dist_for
    req, self.require_hashes, self.use_user_site, self.finder,
  File "/Users/paschalj/test9/june26_mango/venv/lib/python2.7/site-packages/pip/_internal/operations/prepare.py", line 354, in prepare_editable_requirement
    req.check_if_exists(use_user_site)
  File "/Users/paschalj/test9/june26_mango/venv/lib/python2.7/site-packages/pip/_internal/req/req_install.py", line 930, in check_if_exists
    self.req.name
  File "/Users/paschalj/test9/june26_mango/venv/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 472, in get_distribution
    dist = get_provider(dist)
  File "/Users/paschalj/test9/june26_mango/venv/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 344, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/Users/paschalj/test9/june26_mango/venv/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 892, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/Users/paschalj/test9/june26_mango/venv/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 783, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
ContextualVersionConflict: (py4j 0.10.7 (/Users/paschalj/test9/june26_mango/venv/lib/python2.7/site-packages), Requirement.parse('py4j==0.10.4'), set(['pyspark']))
make: *** [develop] Error 2
[ERROR] Command execution failed.
org.apache.commons.exec.ExecuteException: Process exited with an error: 2 (Exit value: 2)
    at org.apache.commons.exec.DefaultExecutor.executeInternal (DefaultExecutor.java:404)
    at org.apache.commons.exec.DefaultExecutor.execute (DefaultExecutor.java:166)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:804)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:751)
    at org.codehaus.mojo.exec.ExecMojo.execute (ExecMojo.java:313)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:208)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:194)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] mango 0.0.2-SNAPSHOT ............................... SUCCESS [  7.278 s]
[INFO] mango: CORE ........................................ SUCCESS [01:19 min]
[INFO] mango: CLI ......................................... SUCCESS [ 43.223 s]
[INFO] Mango: Assembly .................................... SUCCESS [01:00 min]
[INFO] Mango: Python visualization hooks .................. SUCCESS [01:13 min]
[INFO] Mango: Python APIs 0.0.2-SNAPSHOT .................. FAILURE [  3.535 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 04:28 min
[INFO] Finished at: 2018-06-26T11:16:32-04:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:exec (dev-python) on project mango-python: Command execution failed.: Process exited with an error: 2 (Exit value: 2) -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :mango-python

This can be resolve by doing:

pip install py4j==0.10.4

prior to running maven
@akmorrow13
Copy link
Contributor

A couple questions:

  • Are you running in a virtualenv?
  • what is your python version?
  • Can you print out and report the output for your env PY4J_ZIP?

@jpdna
Copy link
Member Author

jpdna commented Jun 26, 2018

Yes, used a virtualenv, seen in the commands included at start of the issue.

echo $PY4J_ZIP
py4j-0.10.4-src.zip

echo $PYTHONPATH
/Users/paschalj/test9/apr10_mango_graph/spark-2.2.1-bin-hadoop2.7/python:/Users/paschalj/test9/apr10_mango_graph/spark-2.2.1-bin-hadoop2.7/python/lib/py4j-0.10.4-src.zip:

python --version
Python 2.7.10

@ricky-lim
Copy link

I am also running into similar problems. Any advice yet to solve?

Cheers

@akmorrow13
Copy link
Contributor

Can you please specify the specific problem you are having, as well as the Spark, Mango, and python version?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@akmorrow13 @jpdna @ricky-lim