POC: Attempt to package jdaviz in Pyinstaller

[duytnguyendtn]

Description

This PR encompasses the learning lessons I've made to package jdaviz in pyinstaller. Requires #1890

Some background:
The goal of pyinstaller is great: it produces either a single executable, or a "portable folder" with an executable that contains everything necessary to run your program. No python, no dependencies, no installing necessary; you just run the executable.

Pyinstaller is "script-oriented", that being it doesn't simply hook into our existing setuptools.entry_points. It effectively runs a .py file "with no arguments" (quotes will be explained later). To start, I added a if name == '__main__' flag at the bottom of cli.py as the starting point.

Pyinstaller encodes all the arguments in a .spec configuration file. This spec config file should live "as close" to the script as possible, hence why the cli.spec file is in jdaviz/jdaviz:

set the current directory to the location of your program
https://pyinstaller.org/en/stable/usage.html

Pyinstaller tries to identify all the necessary modules by traversing your script through all its imports and finding all the modules called by all its imports recursively. The side-effect of this approach is the following:

Anything that is not directly imported is not included by default

This can include icons, data files (like our line lists), ALL of our .vue files, etc. These need to be listed and included manually. To expedite the development of this PR, any repository with missing assets, I copied over the whole package, rather than try and isolate each and every file that was missing. This by consequence increases the size of the packaged executable. If we were to actually deploy this, care could be taken to locate all the individual files, rather than blindly copying over the entire repo.

To build the executable:

pip install pyinstaller
cd jdaviz/jdaviz
pyinstaller ./cli.spec

This will build the executable in jdaviz/dist. It should be as simple as calling jdaviz/dist/cli/cli.exe

This results in the following bug I'm stuck on: the exe enters an infinite loop where it attempts to open multiple copies of jdaviz. None of these copies load properly and just hang on the loading screen:

Console snippet

(envhacking) PS E:\STScI\gitRepos\jdaviz\jdaviz> .\dist\cli\cli.exe
[Voila] Using C:\Users\Duy\AppData\Local\Temp to store connection files
[Voila] Storing connection files in C:\Users\Duy\AppData\Local\Temp\voila_zoe9pi_l.
[Voila] Serving static files from E:\STScI\gitRepos\jdaviz\jdaviz\dist\cli\voila\static.
[Voila] Voilà is running at:
http://localhost:8866/
[Voila] WARNING | Notebook notebook.ipynb is not trusted
0.02s - Debugger warning: The os.path.realpath.__code__.co_filename (ntpath.py)
0.00s - is not absolute, which may make the debugger miss breakpoints.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
[Voila] WARNING | Kernel Provisioning: The 'local-provisioner' is not found.  This is likely due to the presence of multiple jupyter_client distributions and a previous distribution is being used as the source for entrypoints - which does not include 'local-provisioner'.  That distribution should be removed such that only the version-appropriate distribution remains (version >= 7).  Until then, a 'local-provisioner' entrypoint will be automatically constructed and used.
The candidate distribution locations are: []
[Voila] Kernel started: 4c1d15b1-0f37-4cd4-82ee-17e483791b64
[Voila] Using C:\Users\Duy\AppData\Local\Temp to store connection files
[Voila] Storing connection files in C:\Users\Duy\AppData\Local\Temp\voila_eg9w5mlj.
[Voila] Serving static files from E:\STScI\gitRepos\jdaviz\jdaviz\dist\cli\voila\static.
[Voila] The port 8866 is already in use, trying another port.
[Voila] Voilà is running at:
http://localhost:8867/
[Voila] WARNING | Notebook notebook.ipynb is not trusted
0.00s - Debugger warning: The os.path.realpath.__code__.co_filename (ntpath.py)
0.00s - is not absolute, which may make the debugger miss breakpoints.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
[Voila] WARNING | Kernel Provisioning: The 'local-provisioner' is not found.  This is likely due to the presence of multiple jupyter_client distributions and a previous distribution is being used as the source for entrypoints - which does not include 'local-provisioner'.  That distribution should be removed such that only the version-appropriate distribution remains (version >= 7).  Until then, a 'local-provisioner' entrypoint will be automatically constructed and used.
The candidate distribution locations are: []
[Voila] Kernel started: ecaf82f1-d9ef-4588-a2da-6f175ae5e15a
[Voila] Using C:\Users\Duy\AppData\Local\Temp to store connection files
[Voila] Storing connection files in C:\Users\Duy\AppData\Local\Temp\voila_nqqte4qf.
[Voila] Serving static files from E:\STScI\gitRepos\jdaviz\jdaviz\dist\cli\voila\static.
[Voila] The port 8866 is already in use, trying another port.
[Voila] The port 8867 is already in use, trying another port.
[Voila] Voilà is running at:
http://localhost:8868/
[Voila] WARNING | Notebook notebook.ipynb is not trusted
0.00s - Debugger warning: The os.path.realpath.__code__.co_filename (ntpath.py)
0.00s - is not absolute, which may make the debugger miss breakpoints.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
[Voila] WARNING | Kernel Provisioning: The 'local-provisioner' is not found.  This is likely due to the presence of multiple jupyter_client distributions and a previous distribution is being used as the source for entrypoints - which does not include 'local-provisioner'.  That distribution should be removed such that only the version-appropriate distribution remains (version >= 7).  Until then, a 'local-provisioner' entrypoint will be automatically constructed and used.
The candidate distribution locations are: []
[Voila] Kernel started: 7ef9a815-bf98-405a-9d1a-7724803dbd64
[Voila] Using C:\Users\Duy\AppData\Local\Temp to store connection files
[Voila] Storing connection files in C:\Users\Duy\AppData\Local\Temp\voila_4jdmrrvh.
[Voila] Serving static files from E:\STScI\gitRepos\jdaviz\jdaviz\dist\cli\voila\static.
[Voila] The port 8866 is already in use, trying another port.
[Voila] The port 8867 is already in use, trying another port.
[Voila] The port 8868 is already in use, trying another port.
[Voila] Voilà is running at:
http://localhost:8869/
....

If you wait long enough, the jupyter client times out waiting for the kernel:

Timeout Traceback

Task exception was never retrieved
future: <Task finished name='Task-9' coro=<VoilaHandler.get_generator.<locals>.put_html() done, defined at voila\handler.py:201> exception=RuntimeError("Kernel didn't respond in 60 seconds")>
Traceback (most recent call last):
  File "voila\handler.py", line 202, in put_html
  File "voila\exporter.py", line 101, in generate_from_notebook_node
  File "jinja2\environment.py", line 1367, in generate_async
  File "jinja2\environment.py", line 925, in handle_exception
  File "E:\STScI\gitRepos\jdaviz\jdaviz\dist\cli\share\jupyter\voila\templates\jdaviz-default\index.html.j2", line 1, in top-level template code
    {%- extends 'nbconvert/templates/jdaviz-default/index.html.j2' -%}
  File "E:\STScI\gitRepos\jdaviz\jdaviz\dist\cli\share\jupyter\nbconvert\templates\jdaviz-default\index.html.j2", line 43, in top-level template code
    {% block notebook_execute %}
  File "E:\STScI\gitRepos\jdaviz\jdaviz\dist\cli\share\jupyter\voila\templates\jdaviz-default\index.html.j2", line 4, in block 'notebook_execute'
    {%- set kernel_id = kernel_start(nb) -%}
  File "jinja2\async_utils.py", line 56, in auto_await
  File "voila\notebook_renderer.py", line 152, in inner_kernel_start
  File "voila\notebook_renderer.py", line 213, in _jinja_kernel_start
  File "nbclient\client.py", line 532, in async_start_new_kernel_client
  File "nbclient\util.py", line 96, in ensure_async
  File "jupyter_client\client.py", line 205, in _async_wait_for_ready
RuntimeError: Kernel didn't respond in 60 seconds

I can't explain why this is happening... My best guess is that pyinstaller is getting confused at which executable it should be calling (we call multiple in the background: jdaviz, voila, jupyter). At this point, I think it would be worth it to pivot to explore other options rather than continue down this route. I'll preserve it here in case we want to pick it up later

Other things to note:

1. I'm running off of astropy dev because of an issue where pyinstaller wasn't including the astropy.convolution C code. This issue went away when moving to astropy dev
2. Pyinstaller seemed to not work with argparse, hence why I skipped the argument parsing in cli._main and called cli.main directly (see the underline). I kept getting a unrecognized arguments: -m path/to/some/file. Calling cli.main directly seemed to sidestep this issue

Change log entry

• Is a change log needed? If yes, is it added to CHANGES.rst? If you want to avoid merge conflicts,
  list the proposed change log here for review and add to CHANGES.rst before merge. If no, maintainer
  should add a no-changelog-entry-needed label.

Checklist for package maintainer(s)

This checklist is meant to remind the package maintainer(s) who will review this pull request of some common things to look for. This list is not exhaustive.

• Are two approvals required? Branch protection rule does not check for the second approval. If a second approval is not necessary, please apply the trivial label.
• Do the proposed changes actually accomplish desired goals? Also manually run the affected example notebooks, if necessary.
• Do the proposed changes follow the STScI Style Guides?
• Are tests added/updated as required? If so, do they follow the STScI Style Guides?
• Are docs added/updated as required? If so, do they follow the STScI Style Guides?
• Did the CI pass? If not, are the failures related?
• Is a milestone set? Set this to bugfix milestone if this is a bug fix and needs to be released ASAP; otherwise, set this to the next major release milestone.
• After merge, any internal documentations need updating (e.g., JIRA, Innerspace)?

[duytnguyendtn]

@pllim found pyinstaller/pyinstaller#1921 (comment), so I tested adding that to our cli.py script, but unfortunately that didn't seem to fix it :( I'm still getting the multiple tabs. Next going to investigate whether I can package just voila by itself, as suggested by @kecnry

[rosteen]

@mariobuikhuizen Any idea why pyinstaller and voila might be interacting in this way?

[maartenbreddels]

I noticed

• POC: Attempt to package jdaviz in PyOxidizer #1923 and
• POC: Attempt to package jdaviz in cx_Freeze #1936

Just wanted to check if this approach is the preferred one, if so, I'm happy to take a look!

[rosteen]

I noticed

* [POC: Attempt to package jdaviz in PyOxidizer #1923](https://github.com/spacetelescope/jdaviz/pull/1923) and

* [POC: Attempt to package jdaviz in cx_Freeze #1936](https://github.com/spacetelescope/jdaviz/pull/1936)

Just wanted to check if this approach is the preferred one, if so, I'm happy to take a look!

@duytnguyendtn hit roadblocks with all three approaches, but it seemed like this one was the most promising.

[duytnguyendtn]

@maartenbreddels Yes! The team would like to explore this Pyinstaller route and I was going to bring it up today at our tagup

[maartenbreddels]

Some notes while getting this working:

Due to Python 3.10 I got this message when running the standalone binary:

  File "asdf/util.py", line 30, in <module>
AttributeError: 'zipimporter' object has no attribute 'exec_module'

This was not easy to find, but it appears this requires Python 3.10
move to 3.10 https://docs.python.org/3/whatsnew/3.10.html#zipimport

The 'loop' behaviour was due to starting a 'kernel' using the `sys.executable', which was set to the binary itself, causing jdaviz to start again and again.

This issue was noted here: ipython/ipython#4779

The solutions are to recognize this (see this thread or @astrofrog 's solution https://github.com/astrofrog/voila-qt-app/blob/master/voila_demo.py ) which we use here.

[pllim]

Superseded by #1960