support Python 3.x

[boegel]

(old internal ticket 231)

We should try and make sure EasyBuild runs (correctly) with Python v3.x.

Supporting Python 2.4 -> 3.2 can be done in a single code base.

For example handling the changed named Exception syntax can be made to work in both 2.4 and 3.1 by using

try:
  #something
except SomeException:
  ex = sys.exc_info()
  #handle exception

[JensTimmerman]

See http://packages.python.org/six/ for a compatibility layer.

[boegel]

Documentation on porting to Python3: http://docs.python.org/dev/howto/pyporting.html

The Python 2/3 Compatible Source is as special interest to me now, but I have no idea whether it's feasible while keeping the code reasonably clean.

[JensTimmerman]

I'm not in for supporting Python 2.4 -> 3.2

2.6 -> 3.2 with the 2to3 tool gives you more readable code.

I propose dropping python 2.4 support in EB 2.0 and making a bootstrap script that uses EB 1.9 to install python 2.6 and EB 2.x for you.

But maybe we should look at our userbase and do a questionaire.
Ask people:

Are you using python 2.4
are you planing on still using 2.4 in 1 year
are you planing on still using 2.4 in 2 year
are you planing on still using 2.4 in 4 year
Are you using python 2.6
are you planing on still using 2.6 in 1 year
are you planing on still using 2.6 in 2 year
are you planing on still using 2.6 in 4 year
Are you using python 2.7
are you planing on still using 2.7 in 1 year
are you planing on still using 2.7 in 2 year
are you planing on still using 2.7 in 4 year
are you planing on using python 3 as a default python this year
are you planing on using python 3 as a default in 1 year
are you planing on using python 3 as a default in 2 year
are you planing on using python 3 as a default in 4 year

This will give us some guidelines, instead of guessing what people will want in the future.

[fgeorgatos]

i don't know if this helps, but I suspect asking which distros people are
about to rely on,
might give you more clueful answers (for those who maintain that as the
default).

On Tue, Oct 29, 2013 at 2:23 PM, Jens Timmerman [email protected]:

I'm not in for supporting Python 2.4 -> 3.2

2.6 -> 3.2 with the 2to3 tool gives you more readable code.

I propose dropping python 2.4 support in EB 2.0 and making a bootstrap
script that uses EB 1.9 to install python 2.6 and EB 2.x for you.

But maybe we should look at our userbase and do a questionaire.
Ask people:

{{{
Are you using python 2.4
are you planing on still using 2.4 in 1 year
are you planing on still using 2.4 in 2 year
are you planing on still using 2.4 in 4 year
Are you using python 2.6
are you planing on still using 2.6 in 1 year
are you planing on still using 2.6 in 2 year
are you planing on still using 2.6 in 4 year
Are you using python 2.7
are you planing on still using 2.7 in 1 year
are you planing on still using 2.7 in 2 year
are you planing on still using 2.7 in 4 year
are you planing on using python 3 as a default python this year
are you planing on using python 3 as a default in 1 year
are you planing on using python 3 as a default in 2 year
are you planing on using python 3 as a default in 4 year
}}}

This will give us some guidelines, instead of guessing what people will
want in the future.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/133#issuecomment-27298183
.

[vsoch]

hey @boegel ! This issue is really old, but we are hitting the time when the sun is actually setting for python 2 for many of the distributions. For example (and I think there are others, but Ubuntu is what I use most) - https://wiki.ubuntu.com/Python/Python36Transition.

What do you see as challenges to making this transition, other than updating a bunch of print / except / iteration lines? I'd be happy to help / give a shot at this (I think it will be more important as python2 is deprecated) if you want to discuss the potential issues. I'll include @shahzebsiddiqui in this discussion too!

[JensTimmerman]

@vsoch we would follow this guide: https://portingguide.readthedocs.io/en/latest/process.html
the dropping python2.4 and python2.5 part has already been done. We are contemplating dropping 2.6 but that's not really needed.

Our idea was to have a single codebase that supports both python2 and 3, having 2 releases with py2to3 seems brittle and cumbersome, especially if you target to eventually switch to python3 only.

First part is to have our dependencies supporting python 3
that would be: vsc-install, vsc-base and optionally
'coloredlogs', and 'humanfriendly'

Our team (HPC UGent) also develops vsc-install and vsc-base and we certainly would not mind getting some test branches with python3 support.
We can start with extra tests that are running all code trough python modernize and run the tests for the generated code with both python2 and python3.
Once these tests are in place we can mandate that all new code should at least work for these tests, and then start updating module by module to make all code python2 and python3 compatible.

vsc-install and vsc-base are transitioned this way we can start doing the same for easybuild.

This has been on our plan for a few years now, but we feel no real pressure in our team to take on this task already, python2 will be supported by RedHat and CentOS for a few years to come, and we have limited python coding resources.

Are you up for this task?

[vsoch]

Definitely! Would you like help with the dependency libraries or should I hold tight and start contribution for just easy build? If you can point me to the right places (links) I would be happy to take a shot at the first task too.

[boegel]

@vsoch There's little point on starting to point EasyBuild when vsc-base hasn't been ported...

We need vsc-install too, but currently only for running the tests (and if we need to we can probably get around that fairly easily, since I suspect it won't be that easy to port vsc-install).

So, if you would like to contribute: vsc-base is definitely a good place to start, see https://github.com/hpcugent/vsc-base.

The main modules we use there are:

• vsc.utils.generaloption
• vsc.utils.fancylogger
• vsc.utils.missing
• vsc.utils.rest
• vsc.utils.affinity
• vsc.utils.patterns
• vsc.utils.wrapper

[boegel]

@vsoch This may be helpful, to get an idea what you're getting into... hpcugent/vsc-base#142

[boegel]

Some progress on this, thanks to @vsoch:

• PR to add support for running vsc-base on top of Python 3.x: [WIP] update python 3.0 for vsc-base hpcugent/vsc-base#258
  • which, in turn, requires vsc-install to be compatible with Python 3.x: minimal tweaks to update for python3 hpcugent/vsc-install#83

[boegel]

Python 3.x support is one of the main goals now for the upcoming EasyBuild 4.0 (see easybuilders/easybuild#447); preliminary ETA is summer 2019 (but this will likely be revised at some point).

There's been quite a bit of progress on this already in a separate 4.x branch, where the initial step was to ingest the parts of the code that we need from vsc-base and vsc-install (effectively removing them as dependencies), see #2708.

All PRs related to Python 3 support are grouped under the python3 label, see python3 Python 3 compatibility .

[boegel]

Update: with #2789, the full framework test suite is passing with Python 3.6 🎉

[vsoch]

Congrats!!

[boegel]

More updates:

• framework test suite now passes with Python 2.6, 2.7, 3.5, 3.6 and 3.7 (using 4.x branch)
• same for easyblocks test suite (using 4.x branch)
• same for easyconfigs test suite (using 4.x branch)

Successful builds using eb on top of Python 3.6:

• Python-2.7.15-foss-2018b.eb + Python-3.6.6-foss-2018b.eb + all deps (from scratch)` (from scratch)
• OpenFOAM-6-foss-2018b.eb + all deps

So I'm calling it, I consider that EasyBuild is fully ported to Python 3 in the 4.x branch, and I'm closing this issue.
No doubt there some problems will still pop up with further testing, but that should be corner cases, which will be handled on a case by case basis.

The plan is to include these changes in the EasyBuild v4.0 release that we intend to release by summer 2019.