From 94dd864877da761beb89275fc5852b050657c4ef Mon Sep 17 00:00:00 2001 From: David Shrewsbury Date: Thu, 15 Aug 2024 11:10:32 -0400 Subject: [PATCH 1/3] Add 3.1 porting guide --- docs/index.rst | 2 +- docs/porting_guides/porting_guide.rst | 13 ++ .../porting_guide_v3.0.rst} | 9 +- docs/porting_guides/porting_guide_v3.1.rst | 120 ++++++++++++++++++ 4 files changed, 139 insertions(+), 5 deletions(-) create mode 100644 docs/porting_guides/porting_guide.rst rename docs/{porting_guide_v3.rst => porting_guides/porting_guide_v3.0.rst} (95%) create mode 100644 docs/porting_guides/porting_guide_v3.1.rst diff --git a/docs/index.rst b/docs/index.rst index 61b9253a..ea282e20 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -109,7 +109,7 @@ Collection maintainers can learn to correctly declare dependencies for their col definition usage collection_metadata - porting_guide_v3 + porting_guides/porting_guide glossary .. toctree:: diff --git a/docs/porting_guides/porting_guide.rst b/docs/porting_guides/porting_guide.rst new file mode 100644 index 00000000..ffaf4786 --- /dev/null +++ b/docs/porting_guides/porting_guide.rst @@ -0,0 +1,13 @@ +****************************** +Ansible Builder Porting Guides +****************************** + +This section lists porting guides that can help you in updating your Execution Environment files between +versions of ``ansible-builder``. + + +.. toctree:: + :maxdepth: 1 + + porting_guide_v3.1 + porting_guide_v3.0 diff --git a/docs/porting_guide_v3.rst b/docs/porting_guides/porting_guide_v3.0.rst similarity index 95% rename from docs/porting_guide_v3.rst rename to docs/porting_guides/porting_guide_v3.0.rst index 4dd1af16..f0ea4e1d 100644 --- a/docs/porting_guide_v3.rst +++ b/docs/porting_guides/porting_guide_v3.0.rst @@ -1,7 +1,8 @@ -.. _porting_guide_v3: +.. _porting_guide_v3.0: -Ansible Builder version 3 Porting Guide -======================================= +********************************* +Ansible Builder 3.0 Porting Guide +********************************* This section discusses the behavioral changes between ``ansible-builder`` version 1.2 and version 3.0. @@ -110,4 +111,4 @@ version Must be ``3`` if you are using Ansible Builder 3.x. -See the :ref:`version ` section for more details. \ No newline at end of file +See the :ref:`version ` section for more details. diff --git a/docs/porting_guides/porting_guide_v3.1.rst b/docs/porting_guides/porting_guide_v3.1.rst new file mode 100644 index 00000000..3e455e9f --- /dev/null +++ b/docs/porting_guides/porting_guide_v3.1.rst @@ -0,0 +1,120 @@ +********************************* +Ansible Builder 3.1 Porting Guide +********************************* + +This section discusses the behavioral changes between ``ansible-builder`` version 3.0 and version 3.1. + +.. note:: + + We highly advise running ``ansible-builder`` with increased verbosity (``--v3``) to fully expose any error + messages that may help in diagnosing any problems. + +.. contents:: Topics + +Python Requirements Handling +============================ + +The 3.1 release significantly changes how Python and system requirements are handled by simplifying dependency +parsing. This release removes the use of an external library that was unmaintained and parsed many types of +Python dependencies either partially or completely incorrectly. The changes are described below. + +PEP 508 Standard +---------------- + +Python requirements files are expected to follow the `PEP 508 standard `_. +Builder will *expect* the requirements file to be in this format, but it makes two exceptions: + +#. Comments (lines beginning with ``#``) are ignored. +#. Any line from the requirements file that is not compliant with PEP508 causes a warning to be omitted, and + the line is passed through to ``pip`` unmodified. It is recommended to *not* depend on this behavior as we + cannot guarantee that these lines will not cause issues later during the dependency installation process. + +The passthrough of lines not following PEP508 may cause issues that may not have been present with version 3.0 +since many of these lines were simply ignored and not passed to ``pip`` with that version. + +Dependency Sanitization +----------------------- + +Dependency sanitization (the combining of duplicate dependencies into a single dependency entry) is no longer performed. + +.. note:: + + The ``--sanitize`` option to the ``ansible-builder introspect`` command still exists, but is now undocumented + and does nothing. + +The effect of this change is that builder will now pass a listed dependency multiple times for each requirement file +in which it is found. For example, with version 3.0, if collection A listed the Python dependency ``foo``, and +collection B listed the dependency ``foo>=1.0``, then it would have appeared in the combined Python requirements file +as a single entry: + +:: + + foo,foo>=1.0 # from collection A, B + +Now, with version 3.1, those dependencies are no longer combined and will appear in the combined Python requirements +file as separate entries: + +:: + + foo # from collection A + foo>=1.0 # from collection B + +If your container image has an older version of ``pip``, this change might cause an error during the image build +process. This guide covers how to deal with this situation below. + +Common Issues +============= + +This section lists some common errors that might be encountered when running ``ansible-builder`` version 3.1. + +ERROR: Double requirement given +------------------------------- + +Because dependency sanitization has been removed, *all* Python requirements from included collections and user +requirement files are passed along to the ``pip`` command that installs those requirements. Due to this change, +the image build may exit abnormally with an error similar to the following: + +:: + + ERROR: Double requirement given: netaddr>=0.10.1 (from -r /tmp/src/requirements.txt (line 13)) (already in netaddr (from -r /tmp/src/requirements.txt (line 4)), name='netaddr') + Error: building at STEP "RUN /output/scripts/assemble": while running runtime: exit status 1 + +This error comes from ``pip`` within one of the intermediate container images when attempting to install the Python +requirements from included collections and/or from the user supplied requirements. This intermediate image is based +on the base image defined within the Execution Environment file, and it means that the version of ``pip`` installed +within that image is too old to handle duplicate requirement entries. + +To determine where the duplicate requirements are coming from, run ``ansible-builder`` with the ``--v3`` option +to get more verbose output, then look for the output from the introspection phase. It will look similar to below: + +:: + + [3/4] STEP 12/13: RUN $PYCMD /output/scripts/introspect.py introspect --write-bindep=/tmp/src/bindep.txt --write-pip=/tmp/src/requirements.txt + Creating parent directory for /tmp/src/requirements.txt + --- + python: + - 'netaddr # from collection ansible.netcommon' + - 'netaddr>=0.10.1 # from collection ansible.utils' + +In the example output above, the double Python requirement for ``netaddr`` is coming from the collections +``ansible.netcommon`` and ``ansible.utils``. + +The solution requires upgrading ``pip`` within the base image to a version that contains an updated dependency resolver. +Beginning with ``pip`` version 20.3, the dependency resolver can handle duplicate requirements whose versions do not +conflict, so that version is the minimum required. Upgrade ``pip`` in the base image from within the Execution +Environment file by adding these lines to it: + +.. code-block:: yaml + + additional_build_steps: + append_base: + - RUN pip3 install -U pip + +That will upgrade ``pip`` to the latest version within the base image. To restrict the upgrade to a specific +version of ``pip``, alter the upgrade command to specify that version. For example: + +.. code-block:: yaml + + additional_build_steps: + append_base: + - RUN pip3 install -U pip==20.3 From 2c6d9d1463e0fb7d596b602be680b83de965c152 Mon Sep 17 00:00:00 2001 From: David Shrewsbury Date: Fri, 16 Aug 2024 07:44:53 -0400 Subject: [PATCH 2/3] Use PYCMD for pip upgrade --- docs/porting_guides/porting_guide_v3.1.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/porting_guides/porting_guide_v3.1.rst b/docs/porting_guides/porting_guide_v3.1.rst index 3e455e9f..16c9c5f8 100644 --- a/docs/porting_guides/porting_guide_v3.1.rst +++ b/docs/porting_guides/porting_guide_v3.1.rst @@ -108,7 +108,7 @@ Environment file by adding these lines to it: additional_build_steps: append_base: - - RUN pip3 install -U pip + - RUN $PYCMD -m pip install -U pip That will upgrade ``pip`` to the latest version within the base image. To restrict the upgrade to a specific version of ``pip``, alter the upgrade command to specify that version. For example: @@ -117,4 +117,4 @@ version of ``pip``, alter the upgrade command to specify that version. For examp additional_build_steps: append_base: - - RUN pip3 install -U pip==20.3 + - RUN $PYCMD -m pip install -U pip==20.3 From fbe3f89292f41c6ee6b35b7f803b1374cb259436 Mon Sep 17 00:00:00 2001 From: David Shrewsbury Date: Wed, 21 Aug 2024 10:20:02 -0400 Subject: [PATCH 3/3] Apply suggestions from code review Co-authored-by: Matt Davis <6775756+nitzmahone@users.noreply.github.com> --- docs/porting_guides/porting_guide_v3.1.rst | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/porting_guides/porting_guide_v3.1.rst b/docs/porting_guides/porting_guide_v3.1.rst index 16c9c5f8..fc162f81 100644 --- a/docs/porting_guides/porting_guide_v3.1.rst +++ b/docs/porting_guides/porting_guide_v3.1.rst @@ -6,8 +6,8 @@ This section discusses the behavioral changes between ``ansible-builder`` versio .. note:: - We highly advise running ``ansible-builder`` with increased verbosity (``--v3``) to fully expose any error - messages that may help in diagnosing any problems. + We highly advise running ``ansible-builder`` with increased verbosity using the ``-vvv`` option (``--v3`` for + versions older than 3.1) to fully expose any error messages that may help in diagnosing any problems. .. contents:: Topics @@ -25,17 +25,17 @@ Python requirements files are expected to follow the `PEP 508 standard