Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Should MACOSX_DEPLOYMENT_TARGET/LC_VERSION_MIN_MACOSX be verified? #56

Closed
letmaik opened this issue Sep 23, 2019 · 45 comments · Fixed by #198
Closed

Should MACOSX_DEPLOYMENT_TARGET/LC_VERSION_MIN_MACOSX be verified? #56

letmaik opened this issue Sep 23, 2019 · 45 comments · Fixed by #198

Comments

@letmaik
Copy link

letmaik commented Sep 23, 2019

When building wheels for macOS target version 10.9 (for example), then delocate will happily pull in dependencies that have a higher target version. In most cases this won't cause issues, but I think there should at least be a warning.

From https://stackoverflow.com/a/17462763, this is how to show the target version from a lib:

$ otool -l foo.dylib | grep -A 3 LC_VERSION_MIN_MACOSX
      cmd LC_VERSION_MIN_MACOSX
  cmdsize 16
  version 10.8
      sdk 10.8

This will typically happen when using homebrew to get dependencies. The downloaded bottles will have the same target version as the host OS. Unfortunately, brew can't be forced to build packages from source with a custom target version: https://discourse.brew.sh/t/it-is-possible-to-build-packages-that-are-compatible-with-older-macos-versions/4421. The only correct solution then would be either to build wheels on an old macOS version that becomes the target version, or to not use homebrew and build dependencies manually while setting MACOSX_DEPLOYMENT_TARGET accordingly.

@matthew-brett
Copy link
Owner

matthew-brett commented Sep 23, 2019 via email

@letmaik
Copy link
Author

letmaik commented Sep 23, 2019

Yeah, I can imagine cases where people want to ignore this, so I think a warning by default and an error if flag present makes most sense. Something like "--strict" or so.

@matthew-brett
Copy link
Owner

matthew-brett commented Sep 23, 2019 via email

@letmaik
Copy link
Author

letmaik commented Sep 23, 2019

I would but it's a bit tricky as I don't have access to a Mac.

I think these are the things to be done:

  • Write a get_target_macos_version(filename) function similar to get_rpaths(filename):

    delocate/delocate/tools.py

    Lines 290 to 325 in ed48de1

    RPATH_RE = re.compile(r"path (.*) \(offset \d+\)")
    def get_rpaths(filename):
    """ Return a tuple of rpaths from the library `filename`
    If `filename` is not a library then the returned tuple will be empty.
    Parameters
    ----------
    filaname : str
    filename of library
    Returns
    -------
    rpath : tuple
    rpath paths in `filename`
    """
    try:
    lines = _cmd_out_err(['otool', '-l', filename])
    except RuntimeError:
    return ()
    if not _line0_says_object(lines[0], filename):
    return ()
    lines = [line.strip() for line in lines]
    paths = []
    line_no = 1
    while line_no < len(lines):
    line = lines[line_no]
    line_no += 1
    if line != 'cmd LC_RPATH':
    continue
    cmdsize, path = lines[line_no:line_no+2]
    assert cmdsize.startswith('cmdsize ')
    paths.append(RPATH_RE.match(path).groups()[0])
    line_no += 2
    return tuple(paths)
  • Write a check_target_macos_version(copied_libs, require_target_macos_version=None, stop_fast=False) (and equivalent bad_reports()) function similar to check_archs(copied_libs, require_archs=(), stop_fast=False):
    def check_archs(copied_libs, require_archs=(), stop_fast=False):
    """ Check compatibility of archs in `copied_libs` dict
    Parameters
    ----------
    copied_libs : dict
    dict containing the (key, value) pairs of (``copied_lib_path``,
    ``dependings_dict``), where ``copied_lib_path`` is a library real path
    that has been copied during delocation, and ``dependings_dict`` is a
    dictionary with key, value pairs where the key is a path in the target
    being delocated (a wheel or path) depending on ``copied_lib_path``, and
    the value is the ``install_name`` of ``copied_lib_path`` in the
    depending library.
    require_archs : str or sequence, optional
    Architectures we require to be present in all library files in wheel.
    If an empty sequence, just check that depended libraries do have the
    architectures of the depending libraries, with no constraints on what
    these architectures are. If a sequence, then a set of required
    architectures e.g. ``['i386', 'x86_64']`` to specify dual Intel
    architectures. If a string, then a standard architecture name as
    returned by ``lipo -info`` or the string "intel", corresponding to the
    sequence ``['i386', 'x86_64']``
    stop_fast : bool, optional
    Whether to give up collecting errors after the first
    Returns
    -------
    bads : set
    set of length 2 or 3 tuples. A length 2 tuple is of form
    ``(depending_lib, missing_archs)`` meaning that an arch in
    `require_archs` was missing from ``depending_lib``. A length 3 tuple
    is of form ``(depended_lib, depending_lib, missing_archs)`` where
    ``depended_lib`` is the filename of the library depended on,
    ``depending_lib`` is the library depending on ``depending_lib`` and
    ``missing_archs`` is a set of missing architecture strings giving
    architectures present in ``depending_lib`` and missing in
    ``depended_lib``. An empty set means all architectures were present as
    required.
    """
    if isinstance(require_archs, string_types):
    require_archs = (['i386', 'x86_64'] if require_archs == 'intel'
    else [require_archs])
    require_archs = frozenset(require_archs)
    bads = []
    for depended_lib, dep_dict in copied_libs.items():
    depended_archs = get_archs(depended_lib)
    for depending_lib, install_name in dep_dict.items():
    depending_archs = get_archs(depending_lib)
    all_required = depending_archs | require_archs
    all_missing = all_required.difference(depended_archs)
    if len(all_missing) == 0:
    continue
    required_missing = require_archs.difference(depended_archs)
    if len(required_missing):
    bads.append((depending_lib, required_missing))
    else:
    bads.append((depended_lib, depending_lib, all_missing))
    if stop_fast:
    return set(bads)
    return set(bads)
    def bads_report(bads, path_prefix=None):
    """ Return a nice report of bad architectures in `bads`
    Parameters
    ----------
    bads : set
    set of length 2 or 3 tuples. A length 2 tuple is of form
    ``(depending_lib, missing_archs)`` meaning that an arch in
    `require_archs` was missing from ``depending_lib``. A length 3 tuple
    is of form ``(depended_lib, depending_lib, missing_archs)`` where
    ``depended_lib`` is the filename of the library depended on,
    ``depending_lib`` is the library depending on ``depending_lib`` and
    ``missing_archs`` is a set of missing architecture strings giving
    architectures present in ``depending_lib`` and missing in
    ``depended_lib``. An empty set means all architectures were present as
    required.
    path_prefix : None or str, optional
    Path prefix to strip from ``depended_lib`` and ``depending_lib``. None
    means do not strip anything.
    Returns
    -------
    report : str
    A nice report for printing
    """
    path_processor = ((lambda x : x) if path_prefix is None
    else get_rp_stripper(path_prefix))
    reports = []
    for result in bads:
    if len(result) == 3:
    depended_lib, depending_lib, missing_archs = result
    reports.append("{0} needs {1} {2} missing from {3}".format(
    path_processor(depending_lib),
    'archs' if len(missing_archs) > 1 else 'arch',
    ', '.join(sorted(missing_archs)),
    path_processor(depended_lib)))
    elif len(result) == 2:
    depending_lib, missing_archs = result
    reports.append("Required {0} {1} missing from {2}".format(
    'archs' if len(missing_archs) > 1 else 'arch',
    ', '.join(sorted(missing_archs)),
    path_processor(depending_lib)))
    else:
    raise ValueError('Report tuple should be length 2 or 3')
    return '\n'.join(sorted(reports))
  • Write code that calls check_target_macos_version() from delocate_wheel() similar to its call to check_rpaths():
    # Check architectures
    if not require_archs is None:
    stop_fast = not check_verbose
    bads = check_archs(copied_libs, require_archs, stop_fast)
    if len(bads) != 0:
    if check_verbose:
    print(bads_report(bads, pjoin(tmpdir, 'wheel')))
    raise DelocationError(
    "Some missing architectures in wheel")
  • Add two new CLI arguments --check-target-macos-version/--require-target-macos-version <version> to delocate-wheel similar to --check-archs/--require-archs <arch...>:
    Option("-k", "--check-archs",
    action="store_true",
    help="Check architectures of depended libraries"),
    Option("-d", "--dylibs-only",
    action="store_true",
    help="Only analyze files with known dynamic library "
    "extensions"),
    Option("--require-archs",
    action="store", type='string',
    help="Architectures that all wheel libraries should "
    "have (from 'intel', 'i386', 'x86_64', 'i386,x86_64')")])
  • Write tests. This will probably take the longest.

If --require-target-macos-version is not given, then it defaults to the version of the depending library.
If --check-target-macos-version is given, then an error is raised if the target macos version of the depended libraries does not exactly match the required one. If the flag is not given, nothing is checked.

I know I said that it should print a warning by default, but then it would behave differently to the existing --check-archs flag, which also does nothing if the flag is missing. And I think it's ok not to have warnings since anything that doesn't fail the CI build (etc.) will not be looked at anyway. What do you think?

@letmaik
Copy link
Author

letmaik commented Sep 23, 2019

I did a test on CI and it turns out that Homebrew libraries (bottles) generally don't have a deployment target set, and that means it can't be queried with otool. This is a bit unfortunate, but I guess you could still treat it as an error but only if your host system version is newer than your target version.

@matthew-brett
Copy link
Owner

Hmm - yes - that is unfortunate.

I hate to be cheeky - but do you have any interest in access to a Mac? There are a couple of beefy machines you'd be welcome to login to.

@letmaik
Copy link
Author

letmaik commented Sep 24, 2019

Definitely, let's exchange details via email.

@letmaik
Copy link
Author

letmaik commented Sep 24, 2019

Some new discoveries...

There's a PR up at pypa/wheel#314 (see also corresponding issue pypa/wheel#312) which adjusts the platform tag during wheel creation depending on the target version of all binaries in the wheel (Python extension plus other packaged dylibs) by taking the maximum target version of all. In there, I also stumbled upon this note (pypa/wheel#314 (comment)):

from macosx version 10.8 to 10.13 there is LC_VERSION_MIN_MACOSX section and since 10.14 (current) there is LC_BUILD_VERSION

I verified that and indeed I see LC_BUILD_VERSION for homebrew bottles on a 10.14 host:

.dylibs/libjpeg.9.dylib
       cmd LC_BUILD_VERSION
   cmdsize 32
  platform macos
       sdk 10.14
     minos 10.14

So that's great, as it means this can always be checked.

The question now is: If the pypa/wheel PR goes in, then there isn't really a need anymore to do any checks in delocate. If people want to check for a given target version of a wheel then they can just look at the wheel filename then and check for macosx_10_6_ for example.

What I don't like about the PR is that it adjusts the version silently without any diagnostic output. There will probably also be concerns about the hand-crafted mach-o parser instead of relying on an established library or tool for that.

What do you think?

@Czaki
Copy link
Contributor

Czaki commented Sep 25, 2019

Hello I`m the author of this pull request and I'm thinking about this problem with silence change of file name. The reason that I'm not put any warning is that the default python installers are for 10.6 (intel) and 10.9 (86_64). So this may produce warnings very often.

I'm not sure how to handle this.
Maybe check if MACOSX_DEPLOYMENT_TARGET is set and then create warning/fail build if needed target is higher than this variable value?

I found one python lib (https://pypi.org/project/macholib/) for parse mach-O header. It is quite big and putting it in the dependency of such basic package is not good idea.

I thnk also that Mach-O header format is more stable than otool output format. And this code is not limited to one platform (good for testing purpose)

@matthew-brett
Copy link
Owner

matthew-brett commented Sep 25, 2019 via email

@letmaik
Copy link
Author

letmaik commented Sep 25, 2019

If MACOSX_DEPLOYMENT_TARGET is not set, then distutils automatically sets it to the version that the Python installation was built when building extension modules. If there is no mismatch between that and any extra libraries, I wouldn't output a warning. You could consider when a wheel just contains an extension module without external libraries. In that case it will always be fine and there shouldn't be a warning.

So, I would warn if the version had to be bumped, and error if MACOSX_DEPLOYMENT_TARGET is set and external libs don't match. distutils doesn't allow MACOSX_DEPLOYMENT_TARGET to be lower than what Python was built with, so probably the right matching algorithm is that the target versions of libraries must be between that of Python and MACOSX_DEPLOYMENT_TARGET, both inclusive. Considering this, then also if MACOSX_DEPLOYMENT_TARGET is not set then it should error if any library has a target version below that of Python. I'm not sure to be honest why distutils has this restriction, maybe I'm missing something here. After all, Apple guarantees forward compatibility, so if I build a module targeting 10.6 but Python is 10.9, then it should still be fine, right?

@Czaki
Copy link
Contributor

Czaki commented Sep 25, 2019

I update the pull request. I add warnings when final macosx version is higher than on begin.
I also add some test for new code and more verbose comment on begin of Mach-O parser.

@matthew-brett I`m not sure what you mean by strict settings? To fail build if it Needs higher version than MACOSX_DEPLOYMENT_TARGET tells? I can try to write something like that, but I,m not sure if I'm enough experienced with python bdist process to do it properly. Maybe some suggestion?

@letmaik One problem which I see (I do not have any project to test it) is when people use external tool (like cmake, make) instead of distutils. In such case there is no guarantee that project specific code is compiled against proper version of macosx.

There is no problem when some of external libraries is compiled against eg. 10.6 and your project is 10.10. There is backward compatibility. Problem is when some of wheel files needs more modern version of macosx (like brew installed libraries).

@letmaik
Copy link
Author

letmaik commented Sep 25, 2019

@Czaki Your new tests look great, good work. We're on the same page, everything makes sense to me. I think you should extend the warning messages further to not just tell people how to silence the warning, but also print which libraries violate the version and that they should recompile them with MACOSX_DEPLOYMENT_TARGET if they want to target a lower version.

@Czaki
Copy link
Contributor

Czaki commented Sep 26, 2019

@letmaik I add this information to warning message.

@matthew-brett
Copy link
Owner

@Czaki - forgive my ignorance - but it is harder to raise an error than do a warning? Or is it just that it is difficult to pass the flag, to trigger the warning?

@Czaki
Copy link
Contributor

Czaki commented Sep 26, 2019

@matthew-brett I'm not sure how exactly looks control flow in distutils process.

I understand that i it should be added to user_options variable of class bdist_wheel. But I do not know what else I should change before i can use it as variable.

So I do not know how to pass this control variable to get_platform function.

@Czaki
Copy link
Contributor

Czaki commented Oct 23, 2019

@matthew-brett my pull request to wheel was merged.

@Czaki
Copy link
Contributor

Czaki commented Nov 5, 2019

@matthew-brett @letmaik
I realized that one thing is still not verified. If the intel wheel is created then all dependencies should be intel. Otherwise there is wrong information in platform tag. What do you thing about this?

@matthew-brett
Copy link
Owner

Well - there are now quite a few poplular wheels - like Numpy - that claim to be intel, but in fact are only 64-bit. The wheel gets installed for an intel Python, but would not work if some tried to use the intel Python in 32-bit mode. We eventually concluded this was so rare that it didn't matter.

@Czaki
Copy link
Contributor

Czaki commented Nov 19, 2019

@matthew-brett @letmaik Maybe I see one big problem. My PR to wheel package cover only libs files which are create by setup.py. But typical pypeline call dealocate after wheel is created. If dealocate somehow used wheel to recalculate platform tag? Or there should be some PR to cover checking dealocated dynamic libraries?

@letmaik
Copy link
Author

letmaik commented Nov 19, 2019

I don't think delocate renames the platform tag so this shouldn't be an issue. @matthew-brett Is that right?

@Czaki
Copy link
Contributor

Czaki commented Nov 19, 2019

Scenario. You build your code against macos 10.10. So every files has macos minimal version 10.10.
So wheel after patch will produce something like cp36_macosx_10_10_x86_64

But your code depends on external library, and you forget to compile it against old revision (or install from brew) and your system is 10.14. Then dealocate will put copy of this library inside wheel. But now the wheel need macos in version 10.14 to run and platform tag suggest 10.10.

Is this working in this way? or I'm wrong?

@matthew-brett
Copy link
Owner

Yes, that's right. Delocate does not change the wheel name, nor does it check whether the vendored libraries are compatible with the stated wheel tag.

@Czaki
Copy link
Contributor

Czaki commented Nov 19, 2019

So my PR to wheel still not solve the issue from which this talk start.
I can try prepare PR which will solve this. But if you are interested then you prefer custom code, or base on macholib package?

@matthew-brett
Copy link
Owner

If macholib is doing something complicated to implement, it's not a big deal to depend on it - but we've avoided macholib so far because it was easy enough to analyze output from command line commands such as otool.

See for example: https://github.com/matthew-brett/delocate/blob/master/delocate/tools.py#L199

@Czaki
Copy link
Contributor

Czaki commented Nov 19, 2019

Ok. If dealocate base on parsing otool then any change of output format will produce problem globally.

@letmaik
Copy link
Author

letmaik commented Nov 19, 2019

OK, I understand the issue now. This is actually a problem.
To solve this, the patch could be extended to also scan dependencies. @Czaki Maybe it's worth checking how much effort it would be to extract this from the mach headers directly instead of using otool. Then it's easier to make a good decision.

@Czaki
Copy link
Contributor

Czaki commented Nov 19, 2019

As you see in my PR code to wheel I create code to check minimal version of macos. It will be easy to extend this code to check also if intel tag could be used.

I'm not so familiar with macholib to tali how many lines of code its needs to get same information.

I do not know dealocate code enough to think how many of code is needed to replace parsing otool output.

I can try, but on begin there is need decision if use macholib or custom code (the code using otool to determine proper tag could be also put in another PR)

@matthew-brett
Copy link
Owner

I really don't mind if you'd prefer to use macholib. Why not do a PR using macholib and we can see if it's worth re-implementing the check with otool?

@letmaik
Copy link
Author

letmaik commented Nov 19, 2019

I'm slightly confused, are we talking about a PR towards wheel? Is the goal to extend wheel to also check shared library dependencies? If yes, then there are three options: use macholib, use otool, use custom mach-o parsing. Right?

@matthew-brett
Copy link
Owner

I think @Czaki is proposing to do a PR to delocate, and he is considering using macholib to detect the macOS compatibility level of the library that a given wheel depends on.

As I understand it, current wheel (thanks to pypa/wheel#314) checks the compatibility of all binaries included in the original (pre-delocation) wheel, but it does not check the libraries that these binaries depend on. Delocate will pull these into the wheel, and they can rather easily have different macOS version dependencies. The proposed PR is to deal with that problem. @Czaki - is that right?

@matthew-brett
Copy link
Owner

@Czaki - I guess you're trying to avoid copying your detection code out of the wheel package? Will macholib always give the same answer?

@Czaki
Copy link
Contributor

Czaki commented Nov 19, 2019

@matthew-brett Yes. I think about PR to dealocate.

I'm ok wit copy detection code from wheel package. Or I can try write code which will import it from wheel. This which is not implemented in wheel is checking of architecture.

Both, my code and macholib parse macho header, so if them give different answer, then one of them need to have bug.

@matthew-brett
Copy link
Owner

I wouldn't import from wheel, they don't like people importing their code, because it is more effort for them to maintain.

@Czaki
Copy link
Contributor

Czaki commented Nov 20, 2019

@matthew-brett I strongly prefer to fix pep8 warnings. It is ok for you for change formating? If it should be put in separated PR or could be part of pr with checking system version?
Czaki@7e3d8e1

@matthew-brett
Copy link
Owner

Could you make a separate PR with the pep8 changes? I promise to merge quickly.

@Czaki
Copy link
Contributor

Czaki commented Nov 20, 2019

I created PR #60

@Czaki
Copy link
Contributor

Czaki commented Nov 21, 2019

@matthew-brett Coul I ask you for review template of interface for proposed changes?
I think abut put all versions of verification in update_wheel_name function (in different PR).

Czaki@fad07c0

@Czaki
Copy link
Contributor

Czaki commented Dec 13, 2019

I have upgrade draft. Czaki@9a90044

I have one conception problem. When I read pep 427 it told that wheel may contain only one platform tag. Base on this I create current code that will calculate proper name (when filled analise_lib_file function body.

But if my assumption is true? Or there are wheels that can have multiple platform tag?

@Czaki
Copy link
Contributor

Czaki commented Dec 16, 2019

@matthew-brett in file delocate/tests/data/make_libs.shthere is line
lipo -create liba.dylib liba32.dylb -output liba_both.dylib

dylb is typos or I missed something?

@Czaki
Copy link
Contributor

Czaki commented Jan 12, 2020

@matthew-brett @letmaik Did you have access to machines with old MacOS? I have problemwith generate sample data on MacOS 10.14. During try to generate code I got such messages:

clang: warning: libstdc++ is deprecated; move to libc++ with a minimum deployment target of OS X 10.9 [-Wdeprecated]
warning: include path for stdlibc++ headers not found; pass '-stdlib=libc++' on the command line to use the libc++ standard library instead [-Wstdlibcxx-not-found]
1 warning generated.
ld: warning: The i386 architecture is deprecated for macOS (remove from the Xcode build setting: ARCHS)

I decide to firs create whole structure and test data and then implement. All base changes I put in https://github.com/Czaki/delocate/tree/weel_name_fix_base and creation test data here: https://github.com/Czaki/delocate/blob/weel_name_fix_base/delocate/tests/data_platform_tag/make_libs.sh

If I good remember on MacOs 10.13 there is possible to compile 32 bits lib.

@letmaik
Copy link
Author

letmaik commented Jan 13, 2020

Not directly, but Travis CI has many versions: https://docs.travis-ci.com/user/reference/osx/#macos-version Theoretically you could create a dummy repo with Travis CI enabled which does what you need to do and then upload the files somewhere. It may take some time to get it all working.

@Czaki
Copy link
Contributor

Czaki commented Apr 7, 2020

I create both #61 and #68. I add test for new functions but I still do not have idea how to write proper test for delocate_wheel new arguments. Can You give some ideas?

@Czaki
Copy link
Contributor

Czaki commented Apr 15, 2020

@letmaik @matthew-brett any suggestions?

@HexDecimal
Copy link
Collaborator

Sorry that this was left alone for so long @Czaki, can you tell me the current status of this issue and your PR's #61 #68? I should be able to review your PR's whenever they're ready, and answer any other questions you have.

i386 support was dropped. So you don't need to worry about that architecture anymore. The current architecture is x86_64&arm64 and the new GitHub workflows automatically upload test data artifacts. You should rebase or merge your PR's with the current master branch to access the current GitHub workflows.

If your PR's are duplicating the functionality of macholib then you should switch your PR's to import macholib instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants