non-standard location of ucp headers and setup.py

[zdemat]

setup.py assumes ucp headers be present in a location relative to python include directories root

include_dirs = [os.path.dirname(get_python_inc())]
. . .
with open(include_dirs[0] + "/ucp/api/ucp_version.h") as f:

There can be deployments where UCX is installed in a specific location and location of its headers and libraries is specified by CPATH and LIBRARY_PATH env variables. I can understand this is complex to analyse, however in such situations it is hard to install ucx-py with automation tools, so this is the issue reported.

To be constructive I would suggest to use e.g. value of 'UCX_MODULE_DIR' key provided by ucx_info -c. The simple code below, that can be further optimized, can easily help to locate the headers relative to underlying UCX installation.

import os

def get_ucx_incpath():
    import subprocess
    res = include_dirs[0]
    # get output of ucx_info configuration
    p = subprocess.Popen("ucx_info -c", stdout=subprocess.PIPE, shell=True)
    (output, err) = p.communicate()
    p.wait()
    # get value of UCX_MODULE_DIR key and return path relative to it
    lines = output.decode("utf-8").split('\n')
    ucx_config = {}
    for line in lines:
        if line:
            key, val = line.split('=')
            ucx_config[key] = val
    if 'UCX_MODULE_DIR' in ucx_config.keys():
        res = ucx_config['UCX_MODULE_DIR']
        # remove lib/ucx
        res = os.path.dirname(os.path.dirname(res))
        # add include
        res = os.path.join(res,'include')
    return res

def get_ucp_version():
    with open(os.path.join(get_ucx_incpath(),"ucp/api/ucp_version.h")) as f:

[pentschev]

Thanks for the report @zdemat . This sounds like a sensible approach to me, would there be any impact on building conda packages if we change it as per the proposal above @jakirkham ?

[zdemat]

Thanks @pentschev for considering this issue. It is a good question I hope this is a more general approach utilizing the underlying UCX and I hope it works with conda but I have not considered all consequences.

It is also only the first simple proposal.

[pentschev]

Yes, definitely, @jakirkham will likely know the answer to that question or perhaps a smart alternative, let's wait for his reply later today.