Issue installing dev environment

[pkzli]

Hi, I'm trying to install a development environment of kraken, and ... dependency management is a nightmare...

I'm currently on fd3df08 . I tried this on two computers and had various issues . First, I have a weird bug where

$ conda env create -f environment_cuda.yml

runs into the error

libmambapy.bindings.specs.ParseError: Error parsing version "'>=2.0.6". Version contains invalid characters in '>=2.0.6.

which makes no sense. It occurs sporadically on one system or the other, without a cause I can identify.

Anyway, sometimes the command go through, and end up in a dependency error from conda (don't have one in my history right now), which is extremely verbose and difficult to understand. And off course, sometimes there is the error, sometimes not.

Finally, if the command terminates correctly, I end up with the message

Successfully installed aiohappyeyeballs-2.4.6 aiohttp-3.11.12 aiosignal-1.3.2 atpublic-5.1 cattrs-24.1.2 coremltools-8.2 flufl.lock-8.1.0 frozenlist-1.5.0 htrmopo-0.3.0 kraken-5.3.1.dev31 markdown-3.7 multidict-6.1.0 numpy-2.0.2 platformdirs-4.3.6 propcache-0.2.1 protobuf-5.29.3 psutil-6.1.1 pyaml-25.1.0 python-dateutil-2.9.0.post0 scikit-learn-1.5.2 sickle-0.7.0 six-1.17.0 yarl-1.18.3

But numpy 2 seems to be incompatible with some other package (error at runtime). So I try to force

- numpy<2.0.0

In environment_cuda.yml ... and end up with exactly the same message (conda still installs numpy 2.0.2 ...). I tried numpy<=1.26 as well, with no success.

So, following some advices on stackoverflow, I tried mamba instead of conda. By running this command

$ mamba env create -f environment_cuda.yml

mamba offers me to install an environment with

+ numpy                                       1.26.4  py311h24aa872_0            pkgs/main       Cached

Great, so I do this. And when I load the environment, and Python, I get

>>> import numpy
>>> print(numpy.__version__)
2.0.2

I don't have a system wide install of numpy. I checked where are the numpy files, they are in the environment created by mamba (or conda). And it's still numpy 2 that is installed.

So, sorry to bother you with that kind of dependency management stuff, and maybe I'm doing something stupid, but do you have any hint to help me ? I'm not an expert with mamba, I suspect it calls conda on the backend, and still installs numpy 2 (but why ?).

Would it be feasible for you to put an environment.yml and environment_cuda.yml where each package is pinned ? Thank you.

[mittagessen]

On 25/02/12 06:53AM, Pierre Kunzli wrote: Would it be feasible for you to put an `environment.yml` and `environment_cuda.yml` where each package is pinned ? Thank you.

As kraken is used both as a library and an application I can't really pin everything as it breaks integration with other workflows. Which OS, hardware,and conda version are you on? With conda 24.11.1 on amd64 and debian is runs through without issue, so it's possible you're just not getting a resolvable environment because some dependency isn't available for your system. Or your conda version is too ancient, they've played around with that a bit lately (although I see the error comes from the libmamba solver which points to a recent enough version).

[pkzli]

One system is a tuxedo OS (based on unbuntu 22.04) and the other a Endeavour OS (Arch based). Some info I get when the error occurs on the tuxedo system. I literally ran this command one hour ago without problem on this system (except the compatibility issue with numpy at runtime).

    libmambapy.bindings.specs.ParseError: Error parsing version "'>=2.0.6". Version contains invalid characters in '>=2.0.6.

`$ /home/pierre/miniconda3/bin/conda env create -f environment_cuda.yml`

  environment variables:
                 CIO_TEST=<not set>
                CONDA_EXE=/home/pierre/miniconda3/bin/conda
         CONDA_PYTHON_EXE=/home/pierre/miniconda3/bin/python
               CONDA_ROOT=/home/pierre/miniconda3
              CONDA_SHLVL=0
           CURL_CA_BUNDLE=<not set>
            DEFAULTS_PATH=/usr/share/gconf/plasmax11.default.path
               LD_PRELOAD=<not set>
           MANDATORY_PATH=/usr/share/gconf/plasmax11.mandatory.path
                     PATH=/home/pierre/miniconda3/condabin:/usr/local/cuda/bin:/home/pierre/.loc
                          al/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/u
                          sr/games:/usr/local/games:/snap/bin
       REQUESTS_CA_BUNDLE=<not set>
            SSL_CERT_FILE=<not set>
            XDG_SEAT_PATH=/org/freedesktop/DisplayManager/Seat0
         XDG_SESSION_PATH=/org/freedesktop/DisplayManager/Session1

     active environment : None
            shell level : 0
       user config file : /home/pierre/.condarc
 populated config files : /home/pierre/miniconda3/.condarc
                          /home/pierre/.condarc
          conda version : 25.1.1
    conda-build version : not installed
         python version : 3.12.2.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=zen3
                          __conda=25.1.1=0
                          __cuda=12.6=0
                          __glibc=2.35=0
                          __linux=6.11.0=0
                          __unix=0=0
       base environment : /home/pierre/miniconda3  (writable)
      conda av data dir : /home/pierre/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/pierre/miniconda3/pkgs
                          /home/pierre/.conda/pkgs
       envs directories : /home/pierre/miniconda3/envs
                          /home/pierre/.conda/envs
               platform : linux-64
             user-agent : conda/25.1.1 requests/2.32.3 CPython/3.12.2 Linux/6.11.0-108013-tuxedo tuxedo/22.04 glibc/2.35 solver/libmamba conda-libmamba-solver/25.1.1 libmambapy/2.0.5 aau/0.5.0 c/. s/. e/.
                UID:GID : 1000:1001
             netrc file : None
           offline mode : False

and the Endeavour OS

   libmambapy.bindings.specs.ParseError: Error parsing version "'>=2.0.6". Version contains invalid characters in '>=2.0.6.

`$ /opt/miniconda3/bin/conda env create -f environment_cuda.yml`

  environment variables:
                 CIO_TEST=<not set>
                CONDA_EXE=/opt/miniconda3/bin/conda
         CONDA_PYTHON_EXE=/opt/miniconda3/bin/python
               CONDA_ROOT=/opt/miniconda3
              CONDA_SHLVL=0
                CUDA_PATH=/opt/cuda
           CURL_CA_BUNDLE=<not set>
               LD_PRELOAD=<not set>
                     PATH=/home/pierrekunzli/.pyenv/shims:/opt/miniconda3/condabin:/usr/local/sb
                          in:/usr/local/bin:/usr/bin:/opt/cuda/bin:/opt/cuda/nsight_compute:/opt
                          /cuda/nsight_systems/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/
                          bin/core_perl
       REQUESTS_CA_BUNDLE=<not set>
            SSL_CERT_FILE=<not set>

     active environment : None
            shell level : 0
       user config file : /home/pierrekunzli/.condarc
 populated config files : /opt/miniconda3/.condarc
          conda version : 25.1.1
    conda-build version : not installed
         python version : 3.12.2.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=zen2
                          __conda=25.1.1=0
                          __cuda=12.7=0
                          __glibc=2.40=0
                          __linux=6.12.10=0
                          __unix=0=0
       base environment : /opt/miniconda3  (read only)
      conda av data dir : /opt/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /opt/miniconda3/pkgs
                          /home/pierrekunzli/.conda/pkgs
       envs directories : /home/pierrekunzli/.conda/envs
                          /opt/miniconda3/envs
               platform : linux-64
             user-agent : conda/25.1.1 requests/2.32.3 CPython/3.12.2 Linux/6.12.10-arch1-1 endeavouros/rolling glibc/2.40 solver/libmamba conda-libmamba-solver/25.1.1 libmambapy/2.0.5 aau/0.4.4 c/. s/. e/.
                UID:GID : 1002:1002
             netrc file : None
           offline mode : False

[pkzli]

BTW, what numpy version are you currently using in your dev environment ? should I insist on trying to get numpy < 2 ?

Both machines are amd64, forgot to mention it. I will try to use the same conda version as you.

[mittagessen]

Looks like it is a bug in libmamba-solver/mamba [0] and has nothing to do with kraken as such. Either downgrade conda/mamba to a release from last year or run the pip install. [0] conda/conda-libmamba-solver#581

[mittagessen]

On 25/02/12 07:32AM, Pierre Kunzli wrote: BTW, what numpy version are you currently using in your dev environment ? should I insist on trying to get numpy < 2 ?

I'm using numpy 2.0.2. IIRC recent shapely versions need numpy 2.x and that's a library that has significant numerical differences between releases so it's better not to mess too much with it.

[pkzli]

Ok, so now by downgrading conda to 24.11.1 I manage to create the environment, but I run into the issue related to numpy.

$ kraken get 10.5281/zenodo.10592716
Processing ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 22.9/22.9 MB 0:00:00 0:00:00
Model dir: /home/pierrekunzli/.local/share/htrmopo/d96caf7a-122e-5576-ab2b-a246c4e64221 (model files: catmus-print-fondue-large.mlmodel)
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/pierrekunzli/miniconda3/envs/kraken/bin/kraken:8 in <module>                               │
│                                                                                                  │
│   5 from kraken.kraken import cli                                                                │
│   6 if __name__ == '__main__':                                                                   │
│   7 │   sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])                         │
│ ❱ 8 │   sys.exit(cli())                                                                          │
│   9                                                                                              │
│                                                                                                  │
│ /home/pierrekunzli/miniconda3/envs/kraken/lib/python3.11/site-packages/click/core.py:1157 in     │
│ __call__                                                                                         │
│                                                                                                  │
│ /home/pierrekunzli/miniconda3/envs/kraken/lib/python3.11/site-packages/click/core.py:1078 in     │
│ main                                                                                             │
│                                                                                                  │
│ /home/pierrekunzli/miniconda3/envs/kraken/lib/python3.11/site-packages/click/core.py:1720 in     │
│ invoke                                                                                           │
│                                                                                                  │
│ /home/pierrekunzli/miniconda3/envs/kraken/lib/python3.11/site-packages/click/core.py:1657 in     │
│ _process_result                                                                                  │
│                                                                                                  │
│ /home/pierrekunzli/miniconda3/envs/kraken/lib/python3.11/site-packages/click/core.py:783 in      │
│ invoke                                                                                           │
│                                                                                                  │
│ /home/pierrekunzli/miniconda3/envs/kraken/lib/python3.11/site-packages/kraken/kraken.py:366 in   │
│ process_pipeline                                                                                 │
│                                                                                                  │
│   363 │                                                                                          │
│   364 │   from threadpoolctl import threadpool_limits                                            │
│   365 │                                                                                          │
│ ❱ 366 │   from kraken.containers import ProcessingStep                                           │
│   367 │   from kraken.lib.progress import KrakenProgressBar                                      │
│   368 │                                                                                          │
│   369 │   ctx = click.get_current_context()                                                      │
│                                                                                                  │
│ /home/pierrekunzli/miniconda3/envs/kraken/lib/python3.11/site-packages/kraken/containers.py:30   │
│ in <module>                                                                                      │
│                                                                                                  │
│    27 import bidi.algorithm as bd                                                                │
│    28 import numpy as np                                                                         │
│    29                                                                                            │
│ ❱  30 from kraken.lib.segmentation import compute_polygon_section                                │
│    31                                                                                            │
│    32 if TYPE_CHECKING:                                                                          │
│    33 │   from os import PathLike                                                                │
│                                                                                                  │
│ /home/pierrekunzli/miniconda3/envs/kraken/lib/python3.11/site-packages/kraken/lib/segmentation.p │
│ y:35 in <module>                                                                                 │
│                                                                                                  │
│     32 from scipy.spatial.distance import pdist, squareform                                      │
│     33 from shapely.ops import nearest_points, unary_union                                       │
│     34 from shapely.validation import explain_validity                                           │
│ ❱   35 from skimage import draw, filters                                                         │
│     36 from skimage.filters import sobel                                                         │
│     37 from skimage.graph import MCP_Connect                                                     │
│     38 from skimage.measure import (approximate_polygon, label, regionprops,                     │
│                                                                                                  │
│ /home/pierrekunzli/miniconda3/envs/kraken/lib/python3.11/site-packages/skimage/__init__.py:124   │
│ in <module>                                                                                      │
│                                                                                                  │
│   121 │   # process, as it may not be compiled yet                                               │
│   122 else:                                                                                      │
│   123 │   try:                                                                                   │
│ ❱ 124 │   │   from ._shared import geometry                                                      │
│   125 │   │                                                                                      │
│   126 │   │   del geometry                                                                       │
│   127 │   except ImportError as e:                                                               │
│                                                                                                  │
│ in init skimage._shared.geometry:1                                                               │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

Which is weird because it seems I have no other numpy version installed on the system.

[pkzli]

I manage to make it work by bumping scikit-image>=0.25.0 in environment_cuda.yml. Then,

conda list

still displays

scikit-image              0.24.0                   pypi_0    pypi

but ... it works ?

Do you think it's a safe way of solving the problem ?

So to summarize

• Downgrading to conda 24.11.1
• Bumping scikit-image to >=0.25

Solved my problem

[mittagessen]

On 25/02/12 08:28AM, Pierre Kunzli wrote: I manage to make it work by bumping scikit-image>=0.25.0 in environment_cuda.yml. Then, ```conda list``` still displays ``` scikit-image 0.24.0 pypi_0 pypi ```

conda installs kraken with pip so if the dependencies in setup.cfg are different than the ones in the environment file it will pull the dependency from pip and uninstall the conda-sourced one. Unfortunately, it isn't possible to install pip dependencies in conda envs in no-deps mode.

but ... it works ? Do you think it's a safe way of solving the problem ?

Yes, it should be safe. In general, taking a virgin conda env and just installing kraken with pip in there is going to produce a working install.