flatpak-pip-generator does not download arch independent packages

[radiodee1]

flatpak-builder version

1.2.2

Linux distribution and version

Ubuntu 22.04

Affected flatpak-builder tool

pip/flatpak-pip-generator

flatpak-builder tool cli args

-r ./../py/requirements.txt --output py-package

Source repository URL

No response

Flatpak-builder manifest URL

No response

Description

This is the flatpak-pip-generator call.

    flatpak-pip-generator -r ./../py/requirements.txt --output py-package # --runtime org.freedesktop.Platform

This is the requirements.txt. You can see, I've tried to use torch as a requirement:

    --extra-index-url https://download.pytorch.org/whl/cpu
    torch==1.12.1
    happytransformer==2.4.1

If I remove the --runtime option from the flatpak-pip-generator call and use the requirements.txt file, I get a lot of output that ends up here:

    Successfully downloaded torch happytransformer datasets tqdm transformers protobuf sentencepiece typing-extensions dill huggingface-hub numpy packaging pyarrow pyyaml regex requests responses tokenizers aiohttp filelock multiprocess pandas xxhash aiosignal async-timeout attrs certifi charset-normalizer frozenlist idna multidict pyparsing python-dateutil pytz urllib3 yarl fsspec six
    ========================================================================
    Downloading arch independent packages
    ========================================================================
    Traceback (most recent call last):
      File "/home/dave/workspace/electron-gpt/flatpak/./tools/pip/flatpak-pip-generator", line 259, in <module>
        url = get_tar_package_url_pypi(name, version)
      File "/home/dave/workspace/electron-gpt/flatpak/./tools/pip/flatpak-pip-generator", line 72, in get_tar_package_url_pypi
        with urllib.request.urlopen(url) as response:
      File "/usr/lib/python3.10/urllib/request.py", line 216, in urlopen
        return opener.open(url, data, timeout)
      File "/usr/lib/python3.10/urllib/request.py", line 525, in open
        response = meth(req, response)
      File "/usr/lib/python3.10/urllib/request.py", line 634, in http_response
        response = self.parent.error(
      File "/usr/lib/python3.10/urllib/request.py", line 563, in error
        return self._call_chain(*args)
      File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
        result = func(*args)
      File "/usr/lib/python3.10/urllib/request.py", line 643, in http_error_default
        raise HTTPError(req.full_url, code, msg, hdrs, fp)
    urllib.error.HTTPError: HTTP Error 404: Not Found

I'm not really sure what causes the HTTP Error.

[DEFCON001]

Having this issue too, but on manjaro, but for a different dependency. The link that is causing my issue is "https://pypi.org/pypi/shiboken2-5.15.2.1/5.15.2/json" (I just printed the url before line 72). The actual package its meant to fetch is "https://pypi.org/pypi/shiboken2/5.15.2/json", so this seems to be some sort of error caused by " " being replaced with "-" (at least in my case).

[DEFCON001]

Seems to be an issue with the get_package_name module processing the value "shiboken2-5.15.2.1-5.15.2-cp35.cp36.cp37.cp38.cp39.cp310-abi3-manylinux1_x86_64.whl". More specifically line 89 to 102:

 89     elif filename.endswith('whl'):
 90         segments = filename.split('-')
 91         if len(segments) == 5:                                                                                                                              
 92             return segments[0]
 93         candidate = segments[:len(segments) - 4]
 94         # Some packages list the version number twice
 95         # e.g. PyQt5-5.15.0-5.15.0-cp35.cp36.cp37.cp38-abi3-manylinux2014_x86_64.whl
 96         if candidate[-1] == segments[len(segments) - 4]:
 97             return '-'.join(candidate[:-1])
 98         return '-'.join(candidate)
 99     else:
100         raise Exception(
101             'Downloaded filename: {} does not end with bz2, gz, xz, zip, or whl'.format(filename)
102         )   ```

[DEFCON001]

I have found the issue, "shiboken2-5.15.2.1-5.15.2-cp35.cp36.cp37.cp38.cp39.cp310-abi3-manylinux1_x86_64.whl" gets broken down into
['shiboken2', '5.15.2.1', '5.15.2', 'cp35.cp36.cp37.cp38.cp39.cp310', 'abi3', 'manylinux1_x86_64.whl']
the logic takes into account duplicate values as described when index 2 and 3 are identical, but here they are not:
'5.15.2.1' vs '5.15.2'
Replacing line 96 with:

if candidate[-1].__contains__(segments[len(segments) - 4]) | segments[len(segments) - 4].__contains__(candidate[-1]):

More errors follow this however, so I'll look into those.

[DEFCON001]

The program then throws the error:

Traceback (most recent call last):
  File "/home/berret/Documents/FLATPAK-TESTING/extras/flatpak-pip-generator", line 264, in <module>
    url = get_tar_package_url_pypi(name, version)
  File "/home/berret/Documents/FLATPAK-TESTING/extras/flatpak-pip-generator", line 80, in get_tar_package_url_pypi
    raise Exception(err)
Exception: Failed to get shiboken2-5.15.2.1 source from https://pypi.org/pypi/shiboken2/5.15.2.1/json

This is correct, as the page does not have any source files directly on it, however it does have a link to a repository, this appears in the https://pypi.org/pypi/shiboken2/5.15.2/json file as a .download_ul value, however it's source code is only one of the files in said repository (see https://download.qt.io/official_releases/QtForPython/ and how /shiboken2 is only one of the sub-directories located there). This might be fixed by parsing the json for the .download_url attribute, and then checking if that link has a sub-directory with the same name as the name variable, if not, then possibly default to the whole repository. Doing some pattern matching to see if the name is inside of the .download_ul might also help but this all gets very inconsistent when every single repo can be different as they are off site. The fix in the above comment will fix the first error, but I don't know if this is a related bug or a separate bug that needs to be opened as a new issue, so please respond whether or not the above change fixed your issues.

[radiodee1]

I made the change you suggested, but the error message I got remained unchanged. This does not work for my original problem. Also, I replaced line 95, not 96. Maybe my editor counts lines different from yours.

[DEFCON001]

No I think I'd added a print command while debugging higher up in the program so it makes sense it would be 95 for you.

[radiodee1]

I came back to this today. Below is some of my output again.

    Downloading arch independent packages
    ========================================================================
    Deleting regex-2022.8.17-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
    Downloading https://files.pythonhosted.org/packages/f7/e1/8b46ff54516faea4ea9469c4f5b50f2e9e73131f54d19e6827141698a822/regex-2022.8.17.tar.gz
    Traceback (most recent call last):
      File "/home/dave/workspace/electron-gpt/flatpak/./tools/pip/flatpak-pip-generator", line 259, in <module>
        url = get_tar_package_url_pypi(name, version)
      File "/home/dave/workspace/electron-gpt/flatpak/./tools/pip/flatpak-pip-generator", line 72, in get_tar_package_url_pypi
        with urllib.request.urlopen(url) as response:
      File "/usr/lib/python3.10/urllib/request.py", line 216, in urlopen
        return opener.open(url, data, timeout)
      File "/usr/lib/python3.10/urllib/request.py", line 525, in open
        response = meth(req, response)
      File "/usr/lib/python3.10/urllib/request.py", line 634, in http_response
        response = self.parent.error(
      File "/usr/lib/python3.10/urllib/request.py", line 563, in error
        return self._call_chain(*args)
      File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
        result = func(*args)
      File "/usr/lib/python3.10/urllib/request.py", line 643, in http_error_default
        raise HTTPError(req.full_url, code, msg, hdrs, fp)
    urllib.error.HTTPError: HTTP Error 404: Not Found

can somebody tell me what this means?

[radiodee1]

Hi,

so the answer to my problem is complex. I want to use a python library that is like pytorch. This library is not on a public repository, and only has wheels available for download. At the same time there are other packages from python I'm interested in. The program flatpak-pip-generator does not complete it's work. It cannot find the repository for torch for the arch independent package.

The problem, if you ask me, is that the flatpak-pip-generator does not print the 'generated-sources' file for the other libraries I'm interested in. I think it should tell you it cannot find the source for one, but then go on to specify the arch independent packages for the others in the output file. As it is now, the entire file is useless if just one of the files cannot be found.

Then, if I want to I can go to the central yaml manifest file and try to download the wheel for the package that upset the flatpak-pip-generator program myself. This might be done by specifying a special flag on the command line when running flatpak-pip-generator. If you ask me, something like this would be useful.

[Kentoseth]

Just updating this ticket to indicate I am experiencing this issue on Ubuntu 22.04 running:

python3 fpg.py --runtime='org.freedesktop.Sdk//23.08' --requirements-file='requirements-mac-linux.txt' --output caster.yml --yaml

(fpg.py is my renaming of the contents of the flatpak-pip-generator.py file)

Output:

========================================================================
Downloading arch independent packages
========================================================================
Deleting pillow-10.2.0-cp311-cp311-manylinux_2_28_x86_64.whl
Downloading https://files.pythonhosted.org/packages/f8/3e/32cbd0129a28686621434cbf17bb64bf1458bfb838f1f668262fefce145c/pillow-10.2.0.tar.gz
Traceback (most recent call last):
  File "/home/Caster/fpg.py", line 291, in <module>
    url = get_tar_package_url_pypi(name, version)
  File "/home/Caster/fpg.py", line 91, in get_tar_package_url_pypi
    raise Exception(err)
Exception: Failed to get PySide2-5.13.2 source from https://pypi.org/pypi/PySide2/5.13.2/json

The error seems self-explanatory considering no platform-independent option exists for PySide2:

https://pypi.org/project/PySide2/#files

Not sure how this might be resolved.

The requirements file is online for testing:

https://github.com/dictation-toolbox/Caster/blob/master/requirements-mac-linux.txt

[bbhtt]

You can use --ignore-pkg on it to exclude. Pyside2 doesn't have any aarch64 wheels either, if you're building for both arches it needs to be built from source, follow https://github.com/flathub/org.freecadweb.FreeCAD/blob/105074876d9b26ab0469d6b68589faeeff2fd59a/org.freecadweb.FreeCAD.yaml#L35

Otherwise take the changes from #331 and you'll get only x86_64 support.

[hfiguiere]

PySide2 is a big sore point because the package is only design to build on the developer system. and version 5.15 require clang 15 or lower or you get incomprehensible errors. we shall feel grateful the last two release build with python 3.11...

tl;dr PySide WILL have to be built it's own way. and it takes a while.