From 02712cd7e127a4237a6ebbdeab53ee139630b7c4 Mon Sep 17 00:00:00 2001 From: carschandler <92899389+carschandler@users.noreply.github.com> Date: Wed, 27 Mar 2024 16:46:24 -0500 Subject: [PATCH 1/2] Add instructions for using micromamba offline Additionally, remove `$` from shell code blocks where the output isn't also shown, which agrees better with the other docs pages. Added a note to the conda-lock yaml guide noting that the lock file must end in `-lock.yml`. --- docs/source/user_guide/micromamba.rst | 333 ++++++++++++++++++++++++-- 1 file changed, 313 insertions(+), 20 deletions(-) diff --git a/docs/source/user_guide/micromamba.rst b/docs/source/user_guide/micromamba.rst index 5bf223d2b3..a51857e53d 100644 --- a/docs/source/user_guide/micromamba.rst +++ b/docs/source/user_guide/micromamba.rst @@ -15,7 +15,7 @@ Quickstarts | ``micromamba`` supports a subset of all ``mamba`` or ``conda`` commands and implements a command line interface from scratch. | You can see all implemented commands with ``micromamba --help``: -.. code:: +.. code-block:: $ micromamba --help @@ -50,14 +50,14 @@ For more details, please read about :ref:`configuration`. After :ref:`activation`, you can run ``install`` to add new packages to the environment. -.. code:: +.. code-block:: - $ micromamba install xtensor -c conda-forge + micromamba install xtensor -c conda-forge Using ``create``, you can also create environments: -.. code:: +.. code-block:: $ micromamba create -n xtensor_env xtensor xsimd -c conda-forge __ @@ -107,9 +107,9 @@ Using ``create``, you can also create environments: After the installation is finished, the environment can be :ref:`activated` with: -.. code:: +.. code-block:: - $ micromamba activate xtensor_env + micromamba activate xtensor_env Specification files @@ -126,7 +126,7 @@ Simple text spec files The ``txt`` file contains *one spec per line*. For example, this could look like: -.. code:: +.. code-block:: xtensor numpy 1.19 @@ -135,9 +135,9 @@ The ``txt`` file contains *one spec per line*. For example, this could look like To use this file, pass: -.. code:: +.. code-block:: - $ micromamba create -n from_file -f spec_file.txt -c conda-forge + micromamba create -n from_file -f spec_file.txt -c conda-forge .. note:: You can pass multiple text spec files by repeating the ``-f,--file`` argument. @@ -148,7 +148,7 @@ Conda YAML spec files More powerful are ``YAML`` files like the following, because they already contain a desired environment name and the channels to use: -.. code:: +.. code-block:: yaml name: testenv channels: @@ -160,9 +160,9 @@ More powerful are ``YAML`` files like the following, because they already contai They are used the same way as text files: -.. code:: +.. code-block:: - $ micromamba create -f env.yml + micromamba create -f env.yml .. note:: CLI options will keep :ref:`precedence` over *target prefix* or *channels* specified in spec files. @@ -175,13 +175,13 @@ Explicit spec files Using ``conda`` you can generate *explicit* environment lock files. For this, create an environment, activate it, and execute: -.. code:: +.. code-block:: - $ conda list --explicit --md5 + conda list --explicit --md5 These environment files look like the following and precisely "pin" the desired package + version + build string. Each package also has a checksum for reproducibility: -.. code:: +.. code-block:: # This file may be used to create an environment using: # $ conda create --name --file @@ -197,9 +197,9 @@ These environment files look like the following and precisely "pin" the desired To install such a file with ``micromamba``, just pass the ``-f`` flag again: -.. code:: +.. code-block:: - $ micromamba create -n xtensor -f explicit_env.txt + micromamba create -n xtensor -f explicit_env.txt .. note:: @@ -215,7 +215,7 @@ Unlike explicit spec files, these "unified" lock files are multi-platform. These files are named ``conda-lock.yml`` by default, and look like: -.. code:: +.. code-block:: yaml # This lock file was generated by conda-lock (https://github.com/conda/conda-lock). DO NOT EDIT! # @@ -270,6 +270,299 @@ These files are named ``conda-lock.yml`` by default, and look like: To install such a file with ``micromamba``, just pass the ``-f`` flag again: -.. code:: +.. code-block:: - $ micromamba create -n my-environment -f conda-lock.yml + micromamba create -n my-environment -f conda-lock.yml + +.. note:: + + The lock file must end with ``-lock.yml`` for ``micromamba`` to recognize it + as such. + +Offline Usage +============= + +Sometimes it is necessary to set up a Python environment on a machine that +cannot be connected to the internet. ``micromamba`` is a convenient solution for +this scenario since it is a statically linked binary, so no "installation" is +required, and no admin privileges are required [#f1]_. To successfully create an +environment, the offline machine needs a list of packages to be installed and +the packages themselves. You may be able to use one of the `specification file +<#specification-files>`_ formats mentioned above to satisfy the former +requirement, but this requires that the environment be re-solved on the offline +machine, which is highly likely to fail [#f2]_. Using the ``conda-lock`` +strategy discussed `above <#micromamba-conda-lock>`_, we can solve the +environment on the online machine and create an explicit, comprehensive +dependency list for multiple platforms. The explicit list of packages removes +the need to solve for the environment on the offline machine. We can then use +``micromamba`` to download the list of packages in an isolated root prefix. The +steps below outline the process for moving an environment to an offline machine +without any installations or additional packages required on the offline +machine. + +.. [#f1] While long path support is not strictly required for Windows machines, it + is recommended to be enabled for machines that do not already have it, which + requires admin privileges. + +.. [#f2] When transferring the package cache to the offline machine, it is + likely that the index JSON files in ``pkgs/cache`` will be invalidated (last + time modified will probably not match the timestamp inside the file). This + will leave the solver in a confused state that may still determine a solution + for the environment, but the solution will probably not be compatible with + the package cache that was obtained using the original solution on the online + machine. Since that is the package cache that we will have moved onto the + offline machine, the install will fail. + +**Requirements**: + +* A machine connected to the internet that has ``micromamba`` installed +* The ability to invoke the output of ``micromamba shell hook`` on the offline + machine (should not require admin privileges, but strict group policies may + prevent script execution altogether) +* The ability to transfer files from the online machine to the offline one +* *Optional*: the ability to unzip a ``.zip`` archive on the offline machine (if + this is not possible, then simply transfer all files unzipped) +* See [#f1]_ if the target machine is running Windows + +On the online machine +********************* + +The following steps should be completed on a machine that is connected to the +internet and has ``micromamba`` installed. It does not have to share the same +architecture as the target machine. + +Prepare environment spec file +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To begin with, the environment needs to be defined using any of the +`specification file <#specification-files>`_ formats mentioned above. The rest +of the steps will assume that the following ``environment.yml`` file is used: + +.. code-block:: yaml + + # environment.yml + name: offline-env + channels: + - conda-forge + dependencies: + - python=3.12 + - numpy + +Any compatible spec file can be used, but note that if the ``channels`` property +is not set in the file, you may have to set it via the ``-c/--channel`` argument +if it is not set elsewhere. + +Create a ``conda-lock.yml`` file +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. note:: + + See the `conda-lock section <#micromamba-conda-lock>`_ for more details on + ``conda-lock`` outside of the offline install context, and see ``conda-lock + lock --help`` if your use case is not covered here. + +``conda-lock`` allows us to generate an explicit list of *all* packages required +to build an environment for any number of architectures. We can install it using +``micromamba`` in a new environment: + +.. code-block:: + + micromamba create -n conda-lock -c conda-forge conda-lock + micromamba activate conda-lock + +Next, the ``conda-lock lock`` command can generate the dependency list using any +of the `specification file <#specification-files>`_ formats mentioned above. We +will use the ``environment.yml`` file listed in the previous step. + +.. code-block:: + + conda-lock lock -f environment.yml --micromamba + +By default, this will create a ``conda-lock.yml`` file with entries for the four +most common platforms/architectures/subdirs: ``linux-64``, ``osx-64``, +``osx-arm64``, ``win-64``. If the platform of your *offline* machine is not +included in this list, then list it explicitly using the ``-p, --platform`` +option. Note that you can choose a different name for this file with the +``--lockfile`` option or by just renaming it, but it must end in ``-lock.yml`` +for ``micromamba`` to properly recognize it. Also note that conda-lock should +find ``micromamba`` automatically and use it, but to be safe, we explicitly +request that it be used via ``--micromamba``. + +We can drop out of the environment now: + +.. code-block:: + + micromamba deactivate conda-lock + +Create a new root prefix directory +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Create a new directory to be used as the root prefix. Using an isolated root +prefix ensures a clean package cache that contains only the packages necessary +to be moved to the new machine. Any clean directory will work. Create the +directory and update the ``MAMBA_ROOT_PREFIX`` environment variable for the +current shell process. + +Bash: + +.. code-block:: bash + + mkdir offline-environment + export MAMBA_ROOT_PREFIX="$PWD/offline-environment" + +PowerShell: + +.. code-block:: powershell + + mkdir offline-environment + $env:MAMBA_ROOT_PREFIX="$PWD\offline-environment" + +.. note:: + + If you close your shell during this process for some reason, remember to reset + the ``MAMBA_ROOT_PREFIX`` environment variable. + +Create the environment +^^^^^^^^^^^^^^^^^^^^^^ + +If the target platform uses a different architecture from the online platform, +the ``MAMBA_PLATFORM`` or ``CONDA_SUBDIR`` environment variables can be used, +but it is easiest to just pass the ``--platform`` argument during environment +creation, which will add a ``.mambarc`` file inside of the new environment's +directory with the ``platform`` property set. We can also download all our +packages without installing them in the creation step by passing the lock file +created earlier to ``-f`` as well as the ``--download-only`` option. For +example, if an online Linux machine is targeting an offline Windows machine, the +following would be used to create and activate the environment: + +.. code-block:: + + micromamba create -n offline-environment -f conda-lock.yml --download-only --platform win-64 + +Note that the ``--platform`` chosen above must be one of the ones provided in +the ``conda-lock.yml`` file. + +``--download-only`` causes the packages to be downloaded to +``$MAMBA_ROOT_PREFIX/pkgs`` but not linked into +``$MAMBA_ROOT_PREFIX/envs/offline-environment``. It also causes the downloaded +``.conda`` and ``.tar.bz2`` archives to be extracted, which is undesirable for +this use-case since it creates a redundant copy of every package, increasing the +file transfer size. Thus, the extracted directories can be removed if desired: + +Bash: + +.. code-block:: bash + + find "$MAMBA_ROOT_PREFIX/pkgs" -type d -delete + +PowerShell: + +.. code-block:: powershell + + ls "$env:MAMBA_ROOT_PREFIX\pkgs" -Attributes Directory | rm -Recurse + +``micromamba`` will extract the archives when the environment is reproduced on +the offline machine, so there is no real disadvantage in doing this. Currently, +there is no way to decouple the download and extract steps, but this may change +in the future (see `here `_). + +Compress the package cache +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +*This step is optional and just reduces the file transfer size. If the offline +machine does not have a way to unzip* ``.zip`` *archives, then this step should be +skipped.* + +With the environment packages downloaded, we can compress the +``$MAMBA_ROOT_PREFIX/pkgs`` directory, commonly called the "package cache". Most +systems provide ``tar`` by default or some other way to unzip archives, but +check to ensure that you can unzip archives on your offline machine. Navigate +into your ``$MAMBA_ROOT_PREFIX`` directory and create a new archive containing +your package cache: + +.. code-block:: + + cd $MAMBA_ROOT_PREFIX + tar -czvf pkgs.zip ./pkgs + +Prepare files for transfer +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Move the following files into your staging area or onto an external drive where +it can be transported to the offline machine: + +* The ``pkgs.zip`` archive +* The ``micromamba`` executable *for your target architecture* (see the + `releases page `_ + to download executables for different architectures) +* The ``conda-lock.yml`` lock file +* Any additional source code required + +These can all be zipped as well before transferring if desired. + +On the offline machine +********************** + +Set up ``micromamba`` +^^^^^^^^^^^^^^^^^^^^^ + +After completing the transfer, you can set up ``micromamba`` manually using +environment variables and ``conda shell hook`` or allow it to set itself up +using + +.. code-block:: + + ./path/to/micromamba shell init + +and restarting your terminal. + +.. note:: + + On Windows, you may need change your script execution policy using + ``Set-ExecutionPolicy Unrestricted CurrentUser`` which will apply to your user + always, or use ``Process`` as the final argument to set it for only this + process. You may also see a request to enable long filename support after + running ``micromamba shell init``. This requires admin to enable, but is not + strictly necessary, although it is certainly recommended if possible. Some + systems may have this enabled already. + +Unzip the package cache +^^^^^^^^^^^^^^^^^^^^^^^ + +After setting up your new prefix through ``micromamba shell init`` or otherwise, +navigate to it and unzip the packages cache. + +.. code-block:: + + cd $MAMBA_ROOT_PREFIX + tar -xzvf /path/to/pkgs.zip + +The ``.conda`` and ``.tar.bz2`` package archives should be present under the +``$MAMBA_ROOT_PREFIX/pkgs`` directory now. + +Create the new environment +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +At this point, all that's left is to create the new environment, which will +extract the cached packages and link them into the new environment. Thanks to +``conda-lock``, the environment spec file will match one-to-one with the package +cache, so no downloading or solving is required. Do note that ``conda-lock.yml`` +does not specify an environment name, so this must be done manually. + +.. code-block:: + + micromamba create -n offline-environment -f /path/to/conda-lock.yml + +.. note:: + + The ``--offline`` argument should not be required in the creation step, but + note that it does exist in case this breaks without it in the future. + Additionally, if ``micromamba`` complains about not being able to find the + ``pkgs_dir``, then add the following lines to your ``.mambarc``: + +.. code-block:: yaml + + pkgs_dirs: + - /path/to/$MAMBA_ROOT_PREFIX/pkgs + +The environment should now be working! From 5c74cbc3ec710b8c6bea23cf085c80d953a2b570 Mon Sep 17 00:00:00 2001 From: carschandler <92899389+carschandler@users.noreply.github.com> Date: Mon, 1 Apr 2024 14:05:27 -0500 Subject: [PATCH 2/2] Remove trailing whitespace --- docs/source/user_guide/micromamba.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/user_guide/micromamba.rst b/docs/source/user_guide/micromamba.rst index a51857e53d..6f4693d7d0 100644 --- a/docs/source/user_guide/micromamba.rst +++ b/docs/source/user_guide/micromamba.rst @@ -350,7 +350,7 @@ of the steps will assume that the following ``environment.yml`` file is used: Any compatible spec file can be used, but note that if the ``channels`` property is not set in the file, you may have to set it via the ``-c/--channel`` argument -if it is not set elsewhere. +if it is not set elsewhere. Create a ``conda-lock.yml`` file ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^