Is it possible to point auditwheel at a local path for .so files?

[ndw]

I'm trying to use cibuildwheel to construct wheels for a package that includes a third party .so file. (In fact, the .so file is our own, but it would be complicated to configure the CI system to do the complete build, so I'm building in local docker containers derived from the same multi-linux distribution that cibuildwheel is using.)

So I put the library files in libs/nix and in setup.py, I add:

if system() == 'Darwin':
    extra_link_args.append('-Llibs/darwin')
else:
    extra_link_args.append('-Llibs/nix')

(I haven't actually tried testing the MacOS one yet and I haven't a clue what I'm going to do about Windows. One step at a time.)

With this configuration, the wheel is successfully built. The linker gets -Llibs/nix and finds the library.

Then auditwheel runs and claims it can't find the .so file. I've banged my head against a bunch of rocks without success. Actually, that's not strictly true. Passing

export CIBW_BEFORE_ALL='cp libs/nix/* /usr/local/lib64'

makes some of the builds work, but that's seem like such an awful hack I'm almost ashamed.

Am I overlooking something?

[Czaki]

Did you try to use LD_LIBRARY_PATH environment variable? This is a universal way to provide a list of directories to search for so files.

[ndw]

I did, but it didn't have any effect. I also tried something called RPATH based on some random web searching. It's not completely clear to me if auditwheel is running in the same or a different container.

[ndw]

I thought I was winning for a minute, when it occurred to me to do something like this:

export CIBW_ENVIRONMENT="LD_LIBRARY_PATH=/project/linux/eec_pypi/libs/nix"

But like before, the build succeeds and auditwheel fails saying it can't locate the .so. For added confusion, when I try testing this in a local Ubuntu VM with --platform linux, this trick seems to work partially.

So I'm no closer to understanding what is going on, I guess.

[joerick]

I can't see your file structure, but based on the above, perhaps it should be this?

export CIBW_ENVIRONMENT="LD_LIBRARY_PATH=/project/libs/nix"

[ndw]

No, that path is correct, because reasons.

[ndw]

I can't find any evidence that LD_LIBRARY_PATH is being supplied in the container when auditwheel is run. I assume that wheel is doing some of that work for the build but auditwheel is not. But I could be wrong.

[ndw]

I appreciate that it's difficult to help if you can't see the repository, so I've made it public. It's a bit of a mess, but I'm just trying to learn how to make this work. I have every intention of throwing it all away and starting over once I know how to make it work. Making the CI build start from sources and build the whole thing is probably never going to be practical, and I don't see why that should be necessary except in as much as I run the risk of getting the wrong .so and related binaries. I think I can manage that.

The repo is https://github.com/Saxonica/BuildWheels

[ndw]

Still no progress. I think the next thing I'm going to have to try is building my own hacked version of cibuildtools that explicitly sets the path or something.

[joerick]

I just took a look at your repo, I am also confused. This run should have worked, I think. auditwheel runs inside the container so setting with CIBW_ENVIRONMENT is right.

[joerick]

Wait. I saw something. You are cd'ing into the linux/eec_pypi dir before running cibuildwheel. Did you try my previous suggestion?

export CIBW_ENVIRONMENT="LD_LIBRARY_PATH=/project/libs/nix"

[ndw]

I did try making the LD_LIBARARY_PATH absolute. I'm cd'ing into linux/eec_pypi before running cibuildwheel because that's where the setup.py file and all of the rest of the build environment is.

[joerick]

Yes, that's fine, but the cwd when cibuildwheel is run is mapped to /project inside the container.

[ndw]

Right. I think the penny has just dropped on that point and I'm trying again in my local rig. It will not be the first or last time that the utterly obvious defeats me, I fear. More when I know more.

[ndw]

Right. I failed to grok that the cwd when cibuildwheels runs was being mounted at /project in the container. I assumed, for no obvious reason, that the root of my repository would be mounted. With that misunderstanding sorted out, I can build and repair wheels in a GitHub action.

Thank you for your help and your patience. There are other problems, but I'm pretty sure they're on the C++ side of the fence, not the cibuildwheels side!