offline install fails with 'ImportError: No module named vsc.install.shared_setup' #1926

CJCShadowsan · 2016-09-21T08:38:10Z

Hi,

So following the advanced bootstrapping guide, and specifying tar files separately... I am presented with:

+++ STAGE 1: installing EasyBuild in temporary dir with easy_install...


[[INFO]] Fetching sources from /etc/easybuild.d/...
[[INFO]] Found /etc/easybuild.d/vsc-base.tar.gz for vsc-base package
[[INFO]] Found /etc/easybuild.d/easybuild-framework.tar.gz for easybuild-framework package
[[INFO]] Found /etc/easybuild.d/easybuild-easyblocks.tar.gz for easybuild-easyblocks package
[[INFO]] Found /etc/easybuild.d/easybuild-easyconfigs.tar.gz for easybuild-easyconfigs package
[[INFO]] installing EasyBuild with 'easy_install --quiet --upgrade --prefix=/tmp/tmpzAzJTm/eb_stage1 /etc/easybuild.d/vsc-base.tar.gz /etc/easybuild.d/easybuild-framework.tar.gz /etc/easybuild.d/easybuild-easyblocks.tar.gz /etc/easybuild.d/easybuild-easyconfigs.tar.gz /etc/easybuild.d/vsc-base.tar.gz'
Traceback (most recent call last):
  File "/etc/easybuild.d//bootstrap_eb.py", line 793, in <module>
    main()
  File "/etc/easybuild.d//bootstrap_eb.py", line 593, in main
    templates = stage1(tmpdir, sourcepath, distribute_egg_dir)
  File "/etc/easybuild.d//bootstrap_eb.py", line 340, in stage1
    easy_install.main(cmd)
  File "/tmp/tmpzAzJTm/lib/python2.7/site-packages/distribute-0.6.49-py2.7.egg/setuptools/command/easy_install.py", line 1973, in main
    with_ei_usage(lambda:
  File "/tmp/tmpzAzJTm/lib/python2.7/site-packages/distribute-0.6.49-py2.7.egg/setuptools/command/easy_install.py", line 1954, in with_ei_usage
    return f()
  File "/tmp/tmpzAzJTm/lib/python2.7/site-packages/distribute-0.6.49-py2.7.egg/setuptools/command/easy_install.py", line 1977, in <lambda>
    distclass=DistributionWithoutHelpCommands, **kw
  File "/usr/lib64/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/lib64/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/tmp/tmpzAzJTm/lib/python2.7/site-packages/distribute-0.6.49-py2.7.egg/setuptools/command/easy_install.py", line 360, in run
    self.easy_install(spec, not self.no_deps)
  File "/tmp/tmpzAzJTm/lib/python2.7/site-packages/distribute-0.6.49-py2.7.egg/setuptools/command/easy_install.py", line 584, in easy_install
    return self.install_item(None, spec, tmpdir, deps, True)
  File "/tmp/tmpzAzJTm/lib/python2.7/site-packages/distribute-0.6.49-py2.7.egg/setuptools/command/easy_install.py", line 634, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "/tmp/tmpzAzJTm/lib/python2.7/site-packages/distribute-0.6.49-py2.7.egg/setuptools/command/easy_install.py", line 829, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/tmp/tmpzAzJTm/lib/python2.7/site-packages/distribute-0.6.49-py2.7.egg/setuptools/command/easy_install.py", line 1109, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/tmp/tmpzAzJTm/lib/python2.7/site-packages/distribute-0.6.49-py2.7.egg/setuptools/command/easy_install.py", line 1095, in run_setup
    run_setup(setup_script, args)
  File "/tmp/tmpzAzJTm/lib/python2.7/site-packages/distribute-0.6.49-py2.7.egg/setuptools/sandbox.py", line 33, in run_setup
    lambda: execfile(
  File "/tmp/tmpzAzJTm/lib/python2.7/site-packages/distribute-0.6.49-py2.7.egg/setuptools/sandbox.py", line 81, in run
    return func()
  File "/tmp/tmpzAzJTm/lib/python2.7/site-packages/distribute-0.6.49-py2.7.egg/setuptools/sandbox.py", line 35, in <lambda>
    {'__file__':setup_script, '__name__':'__main__'}
  File "setup.py", line 36, in <module>
ImportError: No module named vsc.install.shared_setup

Yet running the same install in the same directory without the EASYBUILD_BOOTSTRAP_SOURCEPATH defined, works fine.

Any thoughts?

Chris

The text was updated successfully, but these errors were encountered:

boegel · 2016-09-21T15:29:14Z

@CJCShadowsan The problem is that vsc-install (https://pypi.python.org/pypi/vsc-install) is not available. Unfortunately, the bootstrap script doesn't support specifying your own tarball for vsc-install yet, which means you're stuck between a rock and a hard place... see also #1880

It shouldn't be too hard to update it to also take into account a vsc-install tarball though...

@ocaisa thoughts?

CJCShadowsan · 2016-09-21T15:40:06Z

The issue here is the documentation clearly states this is possible, when in fact it's not. Else i'd not have bothered trying to take it offline ;)

But yeah, surely vsc-install tarball would make this an offline process then (using the advanced bootstrapping options)...

...Though it wouldn't just be vsc-install - it'd need to be distribute as well wouldn't it for Stage 0?

ocaisa · 2016-09-22T16:07:33Z

The bootstrap script already handles distribute, it just needs to do the same for vsc-install/vsc-base and then it could leverage what's in #1880

ocaisa · 2016-09-22T16:10:04Z

Whoops, sorry, it doesn't handle distribute...but it could with #1880 and then treat the others the same way. It would mean you're stuck with a hard version of each though.

boegel · 2016-12-10T17:26:24Z

the bootstrap script is now able to also install vsc-install using a provided tarball, see #1955

with #1946 and #1984, it will now also auto-skip stage 0 of the bootstrap procedure in case a suitable setuptools installation is found on the system

with this combined, a fully offline installation should work...

arif-ali · 2016-12-17T00:34:35Z

I am following up on this, and we have tried, this, and we still get a problem, (I work with @CJCShadowsan)

[[INFO]] +++ STAGE 1: installing EasyBuild in temporary dir with easy_install...

[[INFO]] Fetching sources from /etc/easybuild.d/...
[[INFO]] Found /etc/easybuild.d/vsc-install.tar.gz for vsc-install package
[[INFO]] Found /etc/easybuild.d/vsc-base.tar.gz for vsc-base package
[[INFO]] Found /etc/easybuild.d/easybuild-framework.tar.gz for easybuild-framework package
[[INFO]] Found /etc/easybuild.d/easybuild-easyblocks.tar.gz for easybuild-easyblocks package
[[INFO]] Found /etc/easybuild.d/easybuild-easyconfigs.tar.gz for easybuild-easyconfigs package
[[INFO]] installing EasyBuild with 'easy_install --quiet --upgrade --prefix=/tmp/tmpC4ztQF/eb_stage1 /etc/easybuild.d/vsc-install.tar.gz /etc/easybuild.d/vsc-base.tar.gz /etc/easybuild.d/easybuild-framework.tar.gz /etc/easybuild.d/easybuild-easyblocks.tar.gz /etc/easybuild.d/easybuild-easyconfigs.tar.gz /etc/easybuild.d/vsc-base.tar.gz'
[[ERROR]] Running 'easy_install --quiet --upgrade --prefix=/tmp/tmpC4ztQF/eb_stage1 /etc/easybuild.d/vsc-install.tar.gz /etc/easybuild.d/vsc-base.tar.gz /etc/easybuild.d/easybuild-framework.tar.gz /etc/easybuild.d/easybuild-easyblocks.tar.gz /etc/easybuild.d/easybuild-easyconfigs.tar.gz /etc/easybuild.d/vsc-base.tar.gz' failed: no file to get name from
Traceback (most recent call last):
  File "/etc/easybuild.d//bootstrap_eb.py", line 323, in run_easy_install
    easy_install.main(args)
  File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1992, in main
    with_ei_usage(lambda:
  File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1979, in with_ei_usage
    return f()
  File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1996, in <lambda>
    distclass=DistributionWithoutHelpCommands, **kw
  File "/usr/lib64/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/lib64/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 380, in run
    self.easy_install(spec, not self.no_deps)
  File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 604, in easy_install
    return self.install_item(None, spec, tmpdir, deps, True)
  File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 653, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 849, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1130, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1115, in run_setup
    run_setup(setup_script, args)
  File "/usr/lib/python2.7/site-packages/setuptools/sandbox.py", line 69, in run_setup
    lambda: execfile(
  File "/usr/lib/python2.7/site-packages/setuptools/sandbox.py", line 120, in run
    return func()
  File "/usr/lib/python2.7/site-packages/setuptools/sandbox.py", line 71, in <lambda>
    {'__file__':setup_script, '__name__':'__main__'}
  File "setup.py", line 1518, in <module>
  File "setup.py", line 1496, in action_target
  File "setup.py", line 1482, in action_target
  File "setup.py", line 1277, in parse_target
  File "setup.py", line 299, in get_name_url
Exception: no file to get name from

any ideas on this would be great

boegel · 2016-12-17T09:17:29Z

@arif-ali Are all the /etc/easybuild.d/*.tar.gz listed in that command actually there?

Which versions of vsc-base and vsc-install did you download here (and why don't you version the tarballs)?

arif-ali · 2016-12-17T16:33:52Z

@boegel, The files are there

We automate the download using the following script, which downloads the latest versions from the master branches from all the repos

wget https://raw.githubusercontent.com/hpcugent/easybuild-framework/develop/easybuild/scripts/bootstrap_eb.py

wget https://github.com/hpcugent/easybuild-framework/archive/master.tar.gz -O easybuild-framework.tar.gz

wget https://github.com/hpcugent/easybuild-easyblocks/archive/master.tar.gz -O easybuild-easyblocks.tar.gz

wget https://github.com/hpcugent/easybuild-easyconfigs/archive/master.tar.gz -O easybuild-easyconfigs.tar.gz

wget https://github.com/hpcugent/vsc-base/archive/master.tar.gz -O vsc-base.tar.gz

wget https://github.com/hpcugent/vsc-install/archive/master.tar.gz -O vsc-install.tar.gz

boegel · 2016-12-17T16:44:36Z

@arif-ali You shouldn't use the tarball from GitHub for vsc-install and vsc-base, but those from PyPI (via https://pypi.python.org/pypi/vsc-install and https://pypi.python.org/pypi/vsc-base).

The source tarballs available from GitHub are not the same, the ones on PyPI are 'pre-processed', see also hpcugent/vsc-base#232 . For the EasyBuild tarballs, it doesn't matter, there's no difference between GitHub & PyPI.

That does complicate auto-downloading a bit though, because the download URLs on PyPI are messed up...

arif-ali · 2016-12-18T02:05:50Z

@boegel Thanks for that direction, I have managed to get to a point, where I can automate the downloads, by running the following commands

wget https://raw.githubusercontent.com/hpcugent/easybuild-framework/develop/easybuild/scripts/bootstrap_eb.py

wget https://github.com/hpcugent/easybuild-framework/archive/master.tar.gz -O easybuild-framework.tar.gz

wget https://github.com/hpcugent/easybuild-easyblocks/archive/master.tar.gz -O easybuild-easyblocks.tar.gz

wget https://github.com/hpcugent/easybuild-easyconfigs/archive/master.tar.gz -O easybuild-easyconfigs.tar.gz

pip download vsc-base --no-deps

pip download vsc-install --no-deps

And then was able to do a local install. Thanks for helping to get this working

boegel · 2016-12-18T08:49:48Z

Excellent!

So, can we close this issue?

arif-ali · 2016-12-18T12:24:38Z

I think the key thing here is, to document this somewhere, so that other people can benefit from this.

Other than that, this should be good, can be closed

boegel · 2016-12-18T12:55:44Z

@arif-ali updated documentation at easybuilders/easybuild#289, please review?

arif-ali · 2016-12-19T10:41:06Z

Looks good to me, thanks

boegel added this to the v2.x milestone Sep 21, 2016

boegel closed this as completed Dec 19, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

offline install fails with 'ImportError: No module named vsc.install.shared_setup' #1926

offline install fails with 'ImportError: No module named vsc.install.shared_setup' #1926

CJCShadowsan commented Sep 21, 2016

boegel commented Sep 21, 2016

CJCShadowsan commented Sep 21, 2016

ocaisa commented Sep 22, 2016

ocaisa commented Sep 22, 2016

boegel commented Dec 10, 2016

arif-ali commented Dec 17, 2016

boegel commented Dec 17, 2016

arif-ali commented Dec 17, 2016 •

edited

Loading

boegel commented Dec 17, 2016

arif-ali commented Dec 18, 2016

boegel commented Dec 18, 2016

arif-ali commented Dec 18, 2016

boegel commented Dec 18, 2016

arif-ali commented Dec 19, 2016

offline install fails with 'ImportError: No module named vsc.install.shared_setup' #1926

offline install fails with 'ImportError: No module named vsc.install.shared_setup' #1926

Comments

CJCShadowsan commented Sep 21, 2016

boegel commented Sep 21, 2016

CJCShadowsan commented Sep 21, 2016

ocaisa commented Sep 22, 2016

ocaisa commented Sep 22, 2016

boegel commented Dec 10, 2016

arif-ali commented Dec 17, 2016

boegel commented Dec 17, 2016

arif-ali commented Dec 17, 2016 • edited Loading

boegel commented Dec 17, 2016

arif-ali commented Dec 18, 2016

boegel commented Dec 18, 2016

arif-ali commented Dec 18, 2016

boegel commented Dec 18, 2016

arif-ali commented Dec 19, 2016

arif-ali commented Dec 17, 2016 •

edited

Loading