Python 3.8 & Ubuntu20.04 support

[Achyut-BK]

I was installing the deb binary for Dragonfire, but it kept saying that I need python version less that 3.7. So, I tried the Debian package build instructions, but it errors out after a few seconds. I tried building it from source, but it didn't create the dragonfire command (I ran it with python3 -m dragonfire), or install most of the required deps. I tried installing the deps manually, but realhud doesn't look like it can be built manually. I then gave up and tried sudo dpkg -i --force-all dragonfire_1.1.1_amd64.deb. This time it built everything, and formed the dragonfire command. Only problem is that when running dragonfire, it tells me it needs deeppavlov==0.7.1, which is incompatible with 3.8. Is there any way we can get it to run under python 3.8 ? Also, it doesn't look like it installs into a virtualenv during manual building, not sure if this is just me or it is supposed be like that.
EDIT my GPU doesn't support CUDA, but I'm hoping that it will work. Not sure if this has any relationship to the problems

[Achyut-BK]

Got it. The deeppavlov library seems to only run with CUDA. so this seems to be an error in the README instead of an error in the app, as it only says the CUDA Library is preferred

[mertyildiran]

@Achyut-BK I'm having hard-time to support Ubuntu 20.04 because there are too many dependency issues. I've removed several dependencies and lost some features on 1.2.0 branch. Maybe you can figure out a way to upgrade while preserving the features.

[Achyut-BK]

@mertyildiran I can try my luck with it. IDK how well I can do though, as like I mentioned, I don't have an Nvidia CUDA supporting GPU. Perhaps we can make another branch for AMD and intel-integrated GPUs that uses OpenCL or installs the CUDA drivers, like mentioned here.I was actually also wondering, can we support more TTS engines, like mycroft does. Mimic2 seems Nice.

[Achyut-BK]

I dug around a bit more in the source code. I uninstalled the deb package and built the master branch from source. It still didn't compile the realhud library, so I tried manually compiling the realhud library, and got the following error

xxx@xxx-Lubuntu:~/Dragonfire/dragonfire/realhud$ gcc -shared -pthread -fPIC -fwrapv -O2 -fno-strict-aliasing -I/usr/include/python3.8 -o realhud realhud.c `pkg-config --libs gtk+-3.0` `pkg-config --cflags gtk+-3.0` -w
realhud.c: In function ‘screen_changed’:
realhud.c:173:5: error: unknown type name ‘GdkColormap’; did you mean ‘Colormap’?
  173 |     GdkColormap *colormap = gdk_screen_get_rgba_colormap(screen);
      |     ^~~~~~~~~~~
      |     Colormap
realhud.c: In function ‘expose’:
realhud.c:194:42: error: ‘GtkWidget’ {aka ‘struct _GtkWidget’} has no member named ‘window’
  194 |     cairo_t *cr = gdk_cairo_create(widget->window);
      |                                          ^~

So this seems to be the problem with the builiding of Realhud. I checked the source again, but the compile instructions given there didn't work. I fixed it with

`gcc -shared -pthread -fPIC -fwrapv -O2 -fno-strict-aliasing -I/usr/include/python3.8 -o realhud realhud.c `pkg-config --libs gtk+-2.0` `pkg-config --cflags gtk+-2.0` -w`

I renamed it realhud.so and copied it to the root Dragonfire directory.
After installing a couple more deps...
I ran into deeppavlov once again.
I tried installing it, and guess what
deeppavlov 0.12.1 requires pydantic==1.3, but you'll have pydantic 1.6.1 which is incompatible.
thinc 8.0.0a11 requires pydantic<2.0.0,>=1.4.0, but you'll have pydantic 1.3 which is incompatible.
bruh.
But at least I fixed that realhud problem I mentioned.
I fixed this thinc mess with the --use-feature=2020-resolver. I was finally able to install deeppavlov.
I ran it AAAND
ModuleNotFoundError: No module named 'thinc.v2v'
(✖╭╮✖) 😭
This seems to be a problem with thinc, as I opened the python shell, and tried thinc myself. It imports thinc, but it doesn't import 'thinc.v2v'.

[Achyut-BK]

Fixed thinc modulenotfound problem by downgrading it to 7.0.0.
We now suffer from
ImportError: cannot import name 'prefer_gpu' from 'thinc.api' (/home/xxx/.local/lib/python3.8/site-packages/thinc/api.py)

[Achyut-BK]

We seem to be in dependency hell. Thinc version 8.0.0a11 is required by spacy, but thinc version 7.0.0 is required for someother scripts.
On installing thinc 7.0.0

spacy 3.0.0.dev12 requires preshed<3.1.0,>=3.0.2, but you'll have preshed 2.0.1 which is incompatible.                                
spacy 3.0.0.dev12 requires srsly<3.0.0,>=2.1.0, but you'll have srsly 1.0.2 which is incompatible.                                    
spacy 3.0.0.dev12 requires thinc==8.0.0a11, but you'll have thinc 7.0.0 which is incompatible.                                        

with Thinc 8 :
ModuleNotFoundError: No module named 'thinc.v2v'
with Thinc 7:
ImportError: cannot import name 'prefer_gpu' from 'thinc.api' (/home/xxx/.local/lib/python3.8/site-packages/thinc/api.py)
I think this is fixable by downgrading spacy. I will figure out the precise versions later

[Achyut-BK]

Fixed the aforementioned problem
pip install spacy==2.3.2 installs spacy and Thinc in a way that is compatible. YAY.
After a couple problems with pyUserInput and missing spacy models, its FINALLY building the AI. So what I'll try doing is installing dragonfire on a fresh version of Ubuntu, go through installing everything, and then post python3 -m pip list and the installed packages here.

[mertyildiran]

@Achyut-BK I'm sorry for my late reply.

pip3 install -e . should compile the realhud Python C extension without any problem. We should replace realhud with an Electron based or low-level WebKit API based solution in the future. Replacing it with something else is out of scope of this issue. This commit proves that realhud has no compilation issues on Ubuntu 20.04

I hit that thinc and spacy dependency issue in the past and I thought I've fixed it somewhere though I couldn't find the commit.

It's great that you've overcome all the dependency issues. I'm looking forward to receive your pull request. Remember that it should pass the GitHub workflow in here. Feel free to modify and/or remove the problematic tests located in here.

[Achyut-BK]

@mertyildiran That sounds great. However, it seems to have a problem with tensorflow 2 (AttributeError: module 'tensorflow' has no attribute 'orthogonal_initializer' ). Tensorflow 1 may work, but there are no precompiled wheels for TF 1 for python3.8. I don't have the processing power to build the wheels myself. Could you help?

[mertyildiran]

@Achyut-BK yeah they drop support for TensorFlow 1.x as of Python 3.8. Therefore we should instead migrate to TensorFlow 2 and replace or remove the models broken in TensorFlow 2.

An alternative approach would be making this project rely on pyenv instead of system packages and use an older Python version like 3.6 or 3.7. But I'm not sure how we can do that. Like; it should install pyenv if it does not exists on the system then it should pyenv install 3.6.10 and then define an environment pyenv virtualenv 3.6.10 dragonfire-env-3.6.10 and finally activate it and use it pyenv activate dragonfire-env-3.6.10. But again I'm not sure how can we achieve this automatically.

[Achyut-BK]

@mertyildiran That would actually be good, as currently, when it is using my systems pip, its basically locked all my ML libraries at one version. Not much of a problem for me, but I'm pretty sure this can get annoying if you use these frameworks regularly. Also, if we succeed with pyenv, we could probably try for packaging it as a snap or appimage for anyone who doesn't want to get down and dirty with the source code. Also, we could maybe update the code itself too. I don't know Coding AI, but I can help out with pyenv.

[mertyildiran]

@Achyut-BK it's good to hear that you can help. Distributing with snap, flatpak and appimage is also in the plan. I don't have enough time to focus on this repository right now. If you can fix these dependency issues, I would be so grateful. I'm ready to answer all kinds of questions of yours and help to you using this discussion chain.