Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pip install in arm64 container #51

Closed
sinsalabim opened this issue Jul 5, 2022 · 5 comments
Closed

Pip install in arm64 container #51

sinsalabim opened this issue Jul 5, 2022 · 5 comments

Comments

@sinsalabim
Copy link

Hi there!

I'm currently trying to implement pyOpenDDS into an ARM64 container based python application.
I'll attach the Dockerfile of my container and the commands + output within this issue post, in order to let you reproduce the steps.

Sorry for the dumb question, I'm such a newbie with opendds and optimized precompiled messages (like protobuffer and stuff).
I do know that I have to precompile messages structure, before start any test/develop.

However, two things do confuse me:

  1. Once $DDS_ROOT/setenv.sh has been sourced or the equivalent, run the commands below in this directory.

Which should be the $DDS_ROOT ? is it the base folder of the cloned repository? I'm asking 'cause there's no setenv.sh across the cloned repository.

  1. CMake Error at CMakeLists.txt:6 (find_package): Error

Within the cloned repository there are four different CMakeLists.txt files

  • pyopendds/ext/CMakeLists.txt
  • tests/basic_test/CMakeLists.txt
  • tests/itl2py_test/CMakeLists.txt
  • pyopendds/dev/itl2py/templates/CMakeLists.txt

However, the error-involved file should be the fourth, and the code line that do triggers the Error, is the following:

find_package(Python3 COMPONENTS Development REQUIRED)

But those libraries are installed (check the container dockerfile).
Therefore, I'm quite confused.

Can anyone reproduce the steps?
Docker environment should help a lot.

Thank you very much for the given attention.

Dockerfile:

FROM --platform=linux/arm64/v8 python:3.10.4

# Install basic ubuntu packages
RUN apt-get update && apt-get install -y \
        software-properties-common \
        sqlite3 \
        gnupg

# Install Python related packages
RUN apt-get update && apt-get install -y \
        python \
        python3 \
        python-dev \
        python3-dev \
        python3-pip

# Install pygobject dependencies
RUN apt-get update && apt-get install -y \
        libgirepository1.0-dev \
        gcc \
        libcairo2-dev \
        pkg-config \
        gir1.2-gtk-3.0 \
        freetds-dev

RUN python -m pip install --upgrade pip

RUN pip3 install pipenv pycairo PyGObject essential_generators Cython

RUN pip3 install pymssql

ENV TZ Europe/Rome

RUN apt-get update && apt-get install -y net-tools iputils-ping netcat patchelf

Then I'll just clone the pyopendds repository into the opt directory (these are the command I've ran in the container terminal):

# cd /opt
# git clone https://github.com/oci-labs/pyopendds.git
# cd pyopendds
# pip3 install .

Output is the following:

Processing /opt/pyopendds
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Collecting cmake-build-extension
  Using cached cmake_build_extension-0.5.1-py3-none-any.whl (13 kB)
Collecting jinja2
  Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.1/133.1 kB 2.3 MB/s eta 0:00:00
Collecting GitPython
  Using cached GitPython-3.1.27-py3-none-any.whl (181 kB)
Collecting cmake
  Using cached cmake-3.22.5-py2.py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (21.9 MB)
Collecting ninja
  Using cached ninja-1.10.2.3-py2.py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (126 kB)
Collecting setuptools-scm
  Using cached setuptools_scm-7.0.4-py3-none-any.whl (42 kB)
Collecting MarkupSafe>=2.0
  Downloading MarkupSafe-2.1.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (26 kB)
Collecting gitdb<5,>=4.0.1
  Using cached gitdb-4.0.9-py3-none-any.whl (63 kB)
Collecting tomli>=1.0.0
  Using cached tomli-2.0.1-py3-none-any.whl (12 kB)
Requirement already satisfied: setuptools in /usr/local/lib/python3.10/site-packages (from setuptools-scm->cmake-build-extension->pyopendds==0.2.0) (58.1.0)
Collecting packaging>=20.0
  Using cached packaging-21.3-py3-none-any.whl (40 kB)
Collecting typing-extensions
  Using cached typing_extensions-4.3.0-py3-none-any.whl (25 kB)
Collecting smmap<6,>=3.0.1
  Using cached smmap-5.0.0-py3-none-any.whl (24 kB)
Collecting pyparsing!=3.0.5,>=2.0.2
  Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
Building wheels for collected packages: pyopendds
  Building wheel for pyopendds (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for pyopendds (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [129 lines of output]
      /tmp/pip-build-env-zkeo_x_5/overlay/lib/python3.10/site-packages/setuptools/config/setupcfg.py:463: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
        warnings.warn(msg, warning_class)
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-aarch64-cpython-310
      creating build/lib.linux-aarch64-cpython-310/pyopendds
      copying pyopendds/Subscriber.py -> build/lib.linux-aarch64-cpython-310/pyopendds
      copying pyopendds/exceptions.py -> build/lib.linux-aarch64-cpython-310/pyopendds
      copying pyopendds/DataWriter.py -> build/lib.linux-aarch64-cpython-310/pyopendds
      copying pyopendds/Publisher.py -> build/lib.linux-aarch64-cpython-310/pyopendds
      copying pyopendds/constants.py -> build/lib.linux-aarch64-cpython-310/pyopendds
      copying pyopendds/util.py -> build/lib.linux-aarch64-cpython-310/pyopendds
      copying pyopendds/init_opendds.py -> build/lib.linux-aarch64-cpython-310/pyopendds
      copying pyopendds/DomainParticipant.py -> build/lib.linux-aarch64-cpython-310/pyopendds
      copying pyopendds/DataReader.py -> build/lib.linux-aarch64-cpython-310/pyopendds
      copying pyopendds/__init__.py -> build/lib.linux-aarch64-cpython-310/pyopendds
      copying pyopendds/Topic.py -> build/lib.linux-aarch64-cpython-310/pyopendds
      creating build/lib.linux-aarch64-cpython-310/pyopendds/dev
      copying pyopendds/dev/util.py -> build/lib.linux-aarch64-cpython-310/pyopendds/dev
      copying pyopendds/dev/__init__.py -> build/lib.linux-aarch64-cpython-310/pyopendds/dev
      creating build/lib.linux-aarch64-cpython-310/pyopendds/dev/itl2py
      copying pyopendds/dev/itl2py/__main__.py -> build/lib.linux-aarch64-cpython-310/pyopendds/dev/itl2py
      copying pyopendds/dev/itl2py/generate.py -> build/lib.linux-aarch64-cpython-310/pyopendds/dev/itl2py
      copying pyopendds/dev/itl2py/Output.py -> build/lib.linux-aarch64-cpython-310/pyopendds/dev/itl2py
      copying pyopendds/dev/itl2py/PythonOutput.py -> build/lib.linux-aarch64-cpython-310/pyopendds/dev/itl2py
      copying pyopendds/dev/itl2py/ast.py -> build/lib.linux-aarch64-cpython-310/pyopendds/dev/itl2py
      copying pyopendds/dev/itl2py/CppOutput.py -> build/lib.linux-aarch64-cpython-310/pyopendds/dev/itl2py
      copying pyopendds/dev/itl2py/__init__.py -> build/lib.linux-aarch64-cpython-310/pyopendds/dev/itl2py
      copying pyopendds/dev/itl2py/itl.py -> build/lib.linux-aarch64-cpython-310/pyopendds/dev/itl2py
      creating build/lib.linux-aarch64-cpython-310/pyopendds/dev/include
      creating build/lib.linux-aarch64-cpython-310/pyopendds/dev/include/pyopendds
      copying pyopendds/dev/include/pyopendds/common.hpp -> build/lib.linux-aarch64-cpython-310/pyopendds/dev/include/pyopendds
      copying pyopendds/dev/include/pyopendds/user.hpp -> build/lib.linux-aarch64-cpython-310/pyopendds/dev/include/pyopendds
      creating build/lib.linux-aarch64-cpython-310/pyopendds/dev/itl2py/templates
      copying pyopendds/dev/itl2py/templates/CMakeLists.txt -> build/lib.linux-aarch64-cpython-310/pyopendds/dev/itl2py/templates
      copying pyopendds/dev/itl2py/templates/user.cpp -> build/lib.linux-aarch64-cpython-310/pyopendds/dev/itl2py/templates
      copying pyopendds/dev/itl2py/templates/setup.py -> build/lib.linux-aarch64-cpython-310/pyopendds/dev/itl2py/templates
      copying pyopendds/dev/itl2py/templates/user.py -> build/lib.linux-aarch64-cpython-310/pyopendds/dev/itl2py/templates
      running build_ext
      -- The CXX compiler identification is GNU 10.2.1
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Found Python3: /usr/local/include/python3.10 (found version "3.10.4") found components: Development Development.Module Development.Embed
      CMake Error at CMakeLists.txt:6 (find_package):
        By not providing "FindOpenDDS.cmake" in CMAKE_MODULE_PATH this project has
        asked CMake to find a package configuration file provided by "OpenDDS", but
        CMake did not find one.

        Could not find a package configuration file provided by "OpenDDS" with any
        of the following names:

          OpenDDSConfig.cmake
          opendds-config.cmake

        Add the installation prefix of "OpenDDS" to CMAKE_PREFIX_PATH or set
        "OpenDDS_DIR" to a directory containing one of the above files.  If
        "OpenDDS" provides a separate development package or SDK, be sure it has
        been installed.


      -- Configuring incomplete, errors occurred!
      See also "/opt/pyopendds/build/temp.linux-aarch64-cpython-310__pyopendds/CMakeFiles/CMakeOutput.log".

      ==> Configuring:
      $ cmake -S /opt/pyopendds/pyopendds/ext -B /opt/pyopendds/build/temp.linux-aarch64-cpython-310__pyopendds -GNinja -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX:PATH=/opt/pyopendds/build/lib.linux-aarch64-cpython-310/_pyopendds -DCMAKE_MAKE_PROGRAM=/tmp/pip-build-env-zkeo_x_5/overlay/bin/ninja -DPython3_ROOT_DIR=/usr/local -DCALL_FROM_SETUP_PY:BOOL=ON -DPYOPENDDS_INCLUDE=/opt/pyopendds/pyopendds/dev/include -DCMAKE_MAKE_PROGRAM=/tmp/pip-build-env-zkeo_x_5/overlay/bin/ninja

      ==> Building:
      $ cmake --build /opt/pyopendds/build/temp.linux-aarch64-cpython-310__pyopendds --config Release

      ==> Installing:
      $ cmake --install /opt/pyopendds/build/temp.linux-aarch64-cpython-310__pyopendds

      Traceback (most recent call last):
        File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 261, in build_wheel
          return _build_backend().build_wheel(wheel_directory, config_settings,
        File "/tmp/pip-build-env-zkeo_x_5/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 244, in build_wheel
          return self._build_with_temp_dir(['bdist_wheel'], '.whl',
        File "/tmp/pip-build-env-zkeo_x_5/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 229, in _build_with_temp_dir
          self.run_setup()
        File "/tmp/pip-build-env-zkeo_x_5/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 174, in run_setup
          exec(compile(code, __file__, 'exec'), locals())
        File "setup.py", line 19, in <module>
          setup(
        File "/tmp/pip-build-env-zkeo_x_5/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup
          return distutils.core.setup(**attrs)
        File "/tmp/pip-build-env-zkeo_x_5/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 177, in setup
          return run_commands(dist)
        File "/tmp/pip-build-env-zkeo_x_5/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 193, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-zkeo_x_5/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 968, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-zkeo_x_5/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 1217, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-zkeo_x_5/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-zkeo_x_5/normal/lib/python3.10/site-packages/wheel/bdist_wheel.py", line 299, in run
          self.run_command('build')
        File "/tmp/pip-build-env-zkeo_x_5/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 317, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-zkeo_x_5/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 1217, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-zkeo_x_5/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-zkeo_x_5/overlay/lib/python3.10/site-packages/setuptools/command/build.py", line 24, in run
          super().run()
        File "/tmp/pip-build-env-zkeo_x_5/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/build.py", line 131, in run
          self.run_command(cmd_name)
        File "/tmp/pip-build-env-zkeo_x_5/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 317, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-zkeo_x_5/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 1217, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-zkeo_x_5/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-zkeo_x_5/overlay/lib/python3.10/site-packages/cmake_build_extension/build_extension.py", line 113, in run
          self.build_extension(ext)
        File "/tmp/pip-build-env-zkeo_x_5/overlay/lib/python3.10/site-packages/cmake_build_extension/build_extension.py", line 229, in build_extension
          subprocess.check_call(configure_command)
        File "/usr/local/lib/python3.10/subprocess.py", line 369, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cmake', '-S', '/opt/pyopendds/pyopendds/ext', '-B', '/opt/pyopendds/build/temp.linux-aarch64-cpython-310__pyopendds', '-GNinja', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_INSTALL_PREFIX:PATH=/opt/pyopendds/build/lib.linux-aarch64-cpython-310/_pyopendds', '-DCMAKE_MAKE_PROGRAM=/tmp/pip-build-env-zkeo_x_5/overlay/bin/ninja', '-DPython3_ROOT_DIR=/usr/local', '-DCALL_FROM_SETUP_PY:BOOL=ON', '-DPYOPENDDS_INCLUDE=/opt/pyopendds/pyopendds/dev/include', '-DCMAKE_MAKE_PROGRAM=/tmp/pip-build-env-zkeo_x_5/overlay/bin/ninja']' returned non-zero exit status 1.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pyopendds
Failed to build pyopendds
ERROR: Could not build wheels for pyopendds, which is required to install pyproject.toml-based projects
@iguessthislldo
Copy link
Member

Which should be the $DDS_ROOT ? is it the base folder of the cloned repository? I'm asking 'cause there's no setenv.sh across the cloned repository.

setenv.sh isn't part of PyOpenDDS, it is generated by OpenDDS when it's configured and should be in the root of the OpenDDS source directory. This is what CMake is complaining about in the middle of the log you posted; it can't find OpenDDS.

@sinsalabim
Copy link
Author

Quite embarrassing.
Thought that pyopendds would have installed OpenDDS automagically.
Quite naive of me.
However, I did managed to install open dds, sourced out setenv.sh and successfully installed pyopendds.

Absolutely easy peasy.

Only curios thing is that setenv.sh had no sha-bang at the very top.
Therefore I've added execution right with chmod +x and ran the script, and everything went right.
Just like one shot one kill.

This is a container built specifying arm64 architecture (as you can see in the docker file).
Right now I'll close the issue right away and let you know if it will run correctly into the real environment (https://developer.nvidia.com/embedded/jetson-nano-developer-kit)

Thank you very much for the given attention and super quick support.

@iguessthislldo
Copy link
Member

Thought that pyopendds would have installed OpenDDS automagically.

That's something I have an issue for: #29

However in the case of arm64, I'm not sure if it'd be be automatic or not.

Only curios thing is that setenv.sh had no sha-bang at the very top.

It doesn't need a sha-bang. It's meant to be sourced from the shell or scripts using something like source OpenDDS/setenv.sh.

@mitza-oci
Copy link
Member

Only curios thing is that setenv.sh had no sha-bang at the very top.

It doesn't need a sha-bang. It's meant to be sourced from the shell or scripts using something like source OpenDDS/setenv.sh.

And the same concept applies to the file mode bits (chmod).

@sinsalabim
Copy link
Author

sinsalabim commented Jul 7, 2022 •
edited
Loading

@iguessthislldo I've realized why sourcing wasn't working.
Container has shell terminal, not bash (check the ps aux command output)

# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.4 163320 36224 pts/0    Ssl+ 11:12   0:00 /usr/bin/qemu-aarch64 /usr/local/bin/python3 python3
root         8  0.0  0.0 148844  7188 pts/1    Ssl  11:12   0:00 /usr/bin/qemu-aarch64 /bin/sh /bin/sh
root        69  0.0  0.1 155252 10564 ?        Rl+  10:31   0:00 ps aux

Therefore the equivalent of "source" is the "." command.

That's why source gives me the following error:

# source OpenDDS/setenv.sh
/bin/sh: 35: source: not found

Instead I should have used the following:

# . OpenDDS/setenv.sh

Which runs smoothly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@iguessthislldo @mitza-oci @sinsalabim