Issues upgrading from 2.8.6 to 2.10.5 "failed to create virtual environment"

[hell1ano]

• O/S: Oracle Linux 7.9
• Python version: 2.7.5
• NetBox version: 2.8.6 trying upgrade to 2.10.5

I've been struggling to first migrate and then upgrade my Netbox installation for the past few hours.
Background is that I'm moving my installation from a local Docker container to an Enterprise server (running Linux) to prepare production use.

While the migration worked fine (doing a pg_dump basically) and all my data is now at the Linux-server, my next "to-do" was to upgrade to the latest stable version. My plan was to go from the current version 2.8.6 to the latest 2.10.5.

Following the guide at https://netbox.readthedocs.io/en/stable/installation/upgrading/
I have copied the local requirements, configuration aswell as the media from my previous installation and re-created the symbolic link.
I get to the step where I need to run the upgrade script, and the script doesnt even begin to process but throws an error immediately. While googling this gives alot of simliar errors, nothing seem to be comparable with my issue.
See below for a full output of the command.

[root@XXXXX netbox]# ./upgrade.sh
Removing old virtual environment...
Creating a new virtual environment at /opt/netbox-2.10.5/venv...
Error: Command '['/opt/netbox-2.10.5/venv/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.
--------------------------------------------------------------------
ERROR: Failed to create the virtual environment. Check that you have
the required system packages installed and the following path is
writable: /opt/netbox-2.10.5/venv
--------------------------------------------------------------------

Any feedback or pointers is appreciated!

[candlerb]

Python version: 2.7.5

Netbox only works with python 3.6+. What does python3 -V show?

Since you're essentially building a server from scratch, rather than upgrading an existing server in-place, I think you will be better off following the regular Netbox installation instructions. Then just restore your pg_dump into that, and finally run ./upgrade.sh to apply the migrations to your database.

[hell1ano]

Obviously I followed the installation instructions for version Netbox 2.8.6 on the server, and its running fine. (Maybe this was unclear from my first message)

[candlerb]

I guess that ought to work, but there was no need to install 2.8.6 on the new server, as you can just install 2.10.5 directly. After importing the database, run ./upgrade.sh to bring the database up to the same level.

That is: if you can install 2.10.5 directly, where running ./upgrade.sh is part of that installation process, then presumably a second ./upgrade.sh should work just fine too.

[hell1ano]

I guess that ought to work, but there was no need to install 2.8.6 on the new server, as you can just install 2.10.5 directly. After importing the database, run ./upgrade.sh to bring the database up to the same level.

That is: if you can install 2.10.5 directly, where running ./upgrade.sh is part of that installation process, then presumably a second ./upgrade.sh should work just fine too.

Yeah did that, obviously I get stuck on the same step.

[candlerb]

To be clear, are you saying that your problem is that you can't make a fresh install of 2.10.5 under Oracle Linux 7.9?

Can you try changing the first line of upgrade.sh from

#!/bin/bash

to

#!/bin/bash -x

and running again. This will make it clear exactly which line of shell script is triggering the problem.

[hell1ano]

To be clear, are you saying that your problem is that you can't make a fresh install of 2.10.5 under Oracle Linux 7.9?

Can you try changing the first line of upgrade.sh from

#!/bin/bash

to

#!/bin/bash -x

and running again. This will make it clear exactly which line of shell script is triggering the problem.

[root@XXXXXXX netbox]# ./upgrade.sh
++ dirname ./upgrade.sh
+ cd .
++ pwd -P
+ VIRTUALENV=/opt/netbox-2.10.5/venv
+ '[' -d /opt/netbox-2.10.5/venv ']'
+ COMMAND='rm -rf /opt/netbox-2.10.5/venv'
+ echo 'Removing old virtual environment...'
Removing old virtual environment...
+ eval rm -rf /opt/netbox-2.10.5/venv
++ rm -rf /opt/netbox-2.10.5/venv
+ COMMAND='/usr/bin/python3 -m venv /opt/netbox-2.10.5/venv'
+ echo 'Creating a new virtual environment at /opt/netbox-2.10.5/venv...'
Creating a new virtual environment at /opt/netbox-2.10.5/venv...
+ eval /usr/bin/python3 -m venv /opt/netbox-2.10.5/venv
++ /usr/bin/python3 -m venv /opt/netbox-2.10.5/venv
Error: Command '['/opt/netbox-2.10.5/venv/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.
+ echo --------------------------------------------------------------------
--------------------------------------------------------------------
+ echo 'ERROR: Failed to create the virtual environment. Check that you have'
ERROR: Failed to create the virtual environment. Check that you have
+ echo 'the required system packages installed and the following path is'
the required system packages installed and the following path is
+ echo 'writable: /opt/netbox-2.10.5/venv'
writable: /opt/netbox-2.10.5/venv
+ echo --------------------------------------------------------------------
--------------------------------------------------------------------
+ exit 1

[candlerb]

OK, to rule out the obvious:

ls -l /opt/netbox-2.10.5/venv    # presumably will say it doesn't exist
mkdir /opt/netbox-2.10.5/venv    # does this succeed?

Other than that, I'm inclined to try

strace -f -s 128 /usr/bin/python3 -m venv /opt/netbox-2.10.5/venv 2>error.out

and try to find the system call which fails.

It's not impossible that SELinux policies are breaking things.

[hell1ano]

You ran the wrong command. The first line of your error.out shows:

execve("/usr/bin/python3", ["/usr/bin/python3", "-m", "/venv", "/opt/netbox-2.10.5/venv"], 0x7ffdea0a0c60 /* 22 vars */) = 0

You should have given -m venv not -m /venv

Lets try again.. sorry
error_2.zip

[candlerb]

There is some traceback buried in there, but the error is 890 bytes long and it only captured the first 128 (-s 128).

[pid 12243] write(2, "Traceback (most recent call last):\n  File \"/usr/lib64/python3.6/runpy.py\", line 193, in _run_module_as_main\n    \"__main__\", mod_"..., 890 <unfinished ...>
[pid 12242] <... read resumed>"Traceback (most recent call last):\n  File \"/usr/lib64/python3.6/runpy.py\", line 193, in _run_module_as_main\n    \"__main__\", mod_"..., 8192) = 890
[pid 12243] <... write resumed>)        = 890
...
[pid 12243] +++ exited with 1 +++

Re-run the strace command with -s 1024 instead of -s 128.

[hell1ano]

There is some traceback buried in there, but the error is 890 bytes long and it only captured the first 128 (-s 128).

[pid 12243] write(2, "Traceback (most recent call last):\n  File \"/usr/lib64/python3.6/runpy.py\", line 193, in _run_module_as_main\n    \"__main__\", mod_"..., 890 <unfinished ...>
[pid 12242] <... read resumed>"Traceback (most recent call last):\n  File \"/usr/lib64/python3.6/runpy.py\", line 193, in _run_module_as_main\n    \"__main__\", mod_"..., 8192) = 890
[pid 12243] <... write resumed>)        = 890
...
[pid 12243] +++ exited with 1 +++

Re-run the strace command with -s 1024 instead of -s 128.

error_3.zip

[candlerb]

Traceback (most recent call last):
  File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/lib64/python3.6/ensurepip/__main__.py", line 5, in <module>
    sys.exit(ensurepip._main())
  File "/usr/lib64/python3.6/ensurepip/__init__.py", line 232, in _main
    default_pip=args.default_pip,
  File "/usr/lib64/python3.6/ensurepip/__init__.py", line 129, in _bootstrap
    shutil.copy(whl, tmpdir)
  File "/usr/lib64/python3.6/shutil.py", line 245, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib64/python3.6/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/lib64/python3.6/ensurepip/_bundled/setuptools-40.6.2-py2.py3-none-any.whl'

Looks like python in Oracle Linux is borked. I found this which suggests you may need third-party repos to get python to work on OL. Sorry I can't help you further; you may have better luck with CentOS or Ubuntu, which most Netbox users seem to deploy.

[hell1ano]

Traceback (most recent call last):
  File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/lib64/python3.6/ensurepip/__main__.py", line 5, in <module>
    sys.exit(ensurepip._main())
  File "/usr/lib64/python3.6/ensurepip/__init__.py", line 232, in _main
    default_pip=args.default_pip,
  File "/usr/lib64/python3.6/ensurepip/__init__.py", line 129, in _bootstrap
    shutil.copy(whl, tmpdir)
  File "/usr/lib64/python3.6/shutil.py", line 245, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib64/python3.6/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/lib64/python3.6/ensurepip/_bundled/setuptools-40.6.2-py2.py3-none-any.whl'

Looks like python in Oracle Linux is borked. I found this which suggests you may need third-party repos to get python to work on OL. Sorry I can't help you further; you may have better luck with CentOS or Ubuntu, which most Netbox users seem to deploy.

Thanks for all your help.
Guess I will have to stay on old Netbox version for the time being...
Is there any easy way for me to know which is the latest version that would run on the "borked version"?

Gonna dive into that link aswell, seeing if any of those other repos would help.