Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failure to build with MACOSX_DEPLOYMENT_TARGET=11 on Big Sur #86670

Closed
fxcoudert mannequin opened this issue Nov 29, 2020 · 16 comments
Closed

Failure to build with MACOSX_DEPLOYMENT_TARGET=11 on Big Sur #86670

fxcoudert mannequin opened this issue Nov 29, 2020 · 16 comments
Labels
3.8 only security fixes 3.9 only security fixes 3.10 only security fixes build The build process and cross-build OS-mac

Comments

@fxcoudert
Copy link
Mannequin

fxcoudert mannequin commented Nov 29, 2020

BPO 42504
Nosy @ronaldoussoren, @ned-deily, @ambv, @mattip, @miss-islington, @fxcoudert, @lonnylot
PRs
  • bpo-42504: fix for MACOSX_DEPLOYMENT_TARGET=11 #23556
  • [3.9] bpo-42504: fix for MACOSX_DEPLOYMENT_TARGET=11 (GH-23556) #23622
  • bpo-42504: Ensure that get_config_var('MACOSX_DEPLOYMENT_TARGET') is a string #24341
  • [3.9] bpo-42504: Ensure that get_config_var('MACOSX_DEPLOYMENT_TARGET') is a string (GH-24341) #24410
  • [3.8] bpo-41100: Support macOS 11 Big Sur and Apple Silicon Macs #25806
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2021-05-02.09:31:51.401>
    created_at = <Date 2020-11-29.15:16:04.604>
    labels = ['OS-mac', 'build', '3.8', '3.9', '3.10']
    title = 'Failure to build with MACOSX_DEPLOYMENT_TARGET=11 on Big Sur'
    updated_at = <Date 2021-05-02.09:31:51.401>
    user = 'https://github.com/fxcoudert'

    bugs.python.org fields:

    activity = <Date 2021-05-02.09:31:51.401>
    actor = 'ned.deily'
    assignee = 'none'
    closed = True
    closed_date = <Date 2021-05-02.09:31:51.401>
    closer = 'ned.deily'
    components = ['Build', 'macOS']
    creation = <Date 2020-11-29.15:16:04.604>
    creator = 'fxcoudert'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 42504
    keywords = ['patch']
    message_count = 16.0
    messages = ['382071', '382116', '382371', '382372', '382402', '382420', '382439', '382440', '382442', '382789', '385502', '385715', '385718', '386053', '386054', '392681']
    nosy_count = 7.0
    nosy_names = ['ronaldoussoren', 'ned.deily', 'lukasz.langa', 'mattip', 'miss-islington', 'fxcoudert', 'lonnylot']
    pr_nums = ['23556', '23622', '24341', '24410', '25806']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'compile error'
    url = 'https://bugs.python.org/issue42504'
    versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

    @fxcoudert
    Copy link
    Mannequin Author

    fxcoudert mannequin commented Nov 29, 2020

    With macOS Big Sur (11.y.z), the major version is 11 (instead of the earlier 10.15, 10.14, etc). Therefore, MACOSX_DEPLOYMENT_TARGET=11 has become a valid setting. It is accepted by system tools:

    $ MACOSX_DEPLOYMENT_TARGET=11 clang a.c && echo $?
    0

    But it leads to failure in Python compilation in setup.py:

    DYLD_FRAMEWORK_PATH=/private/tmp/python@3.9-20201127-81873-q9b09s/Python-3.9.0 CC='clang' LDSHARED='clang -bundle -undefined dynamic_lookup -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk   ' OPT='-DNDEBUG -g -fwrapv -O3 -Wall'   _TCLTK_INCLUDES='' _TCLTK_LIBS=''       ./python.exe -E ./setup.py  build
    running build
    running build_ext
    Traceback (most recent call last):
      File "/private/tmp/python@3.9-20201127-81873-q9b09s/Python-3.9.0/./setup.py", line 2614, in <module>
        main()
      File "/private/tmp/python@3.9-20201127-81873-q9b09s/Python-3.9.0/./setup.py", line 2584, in main
        setup(# PyPI Metadata (PEP 301)
      File "/private/tmp/python@3.9-20201127-81873-q9b09s/Python-3.9.0/Lib/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/private/tmp/python@3.9-20201127-81873-q9b09s/Python-3.9.0/Lib/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "/private/tmp/python@3.9-20201127-81873-q9b09s/Python-3.9.0/Lib/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/private/tmp/python@3.9-20201127-81873-q9b09s/Python-3.9.0/Lib/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
      File "/private/tmp/python@3.9-20201127-81873-q9b09s/Python-3.9.0/Lib/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/private/tmp/python@3.9-20201127-81873-q9b09s/Python-3.9.0/Lib/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/private/tmp/python@3.9-20201127-81873-q9b09s/Python-3.9.0/Lib/distutils/command/build_ext.py", line 340, in run
        self.build_extensions()
      File "/private/tmp/python@3.9-20201127-81873-q9b09s/Python-3.9.0/./setup.py", line 470, in build_extensions
        self.detect_modules()
      File "/private/tmp/python@3.9-20201127-81873-q9b09s/Python-3.9.0/./setup.py", line 1807, in detect_modules
        self.detect_readline_curses()
      File "/private/tmp/python@3.9-20201127-81873-q9b09s/Python-3.9.0/./setup.py", line 1038, in detect_readline_curses
        (tuple(int(n) for n in dep_target.split('.')[0:2])
    AttributeError: 'int' object has no attribute 'split'
    make: *** [sharedmods] Error 1

    See the full report here: Homebrew/brew#9329

    The problem is here: https://github.com/python/cpython/blob/master/setup.py#L1015
    When MACOSX_DEPLOYMENT_TARGET=11, this call:
    sysconfig.get_config_var('MACOSX_DEPLOYMENT_TARGET')
    returns an int (with value 11) instead of a string (as is the case if 11.0, 10.15, etc).

    And therefore the subsequent call to: dep_target.split('.')[0:2] fails, because split() cannot be called on an int.

    @fxcoudert fxcoudert mannequin added 3.9 only security fixes build The build process and cross-build labels Nov 29, 2020
    @guillochon guillochon mannequin added OS-mac labels Nov 29, 2020
    @ronaldoussoren
    Copy link
    Contributor

    Good catch.

    @ned-deily
    Copy link
    Member

    New changeset 5291639 by FX Coudert in branch 'master':
    bpo-42504: fix for MACOSX_DEPLOYMENT_TARGET=11 (GH-23556)
    5291639

    @ned-deily ned-deily added 3.8 only security fixes 3.10 only security fixes labels Dec 3, 2020
    @miss-islington
    Copy link
    Contributor

    New changeset 09a698b by Miss Islington (bot) in branch '3.9':
    bpo-42504: fix for MACOSX_DEPLOYMENT_TARGET=11 (GH-23556)
    09a698b

    @ronaldoussoren
    Copy link
    Contributor

    FWIW I don't agree with this fix.

    sys.get_config_var('MACOSX_DEPLOYMENT_TARGET') has always resulted in a string value. There's bound to be someone relying on this.

    As I wrote in comment on the PR a nicer fix is to teach sysconfig (and distutils.sysconfig) to not convert the value of this key to an integer.

    @fxcoudert
    Copy link
    Mannequin Author

    fxcoudert mannequin commented Dec 3, 2020

    "It has always resulted in a string value": only MACOSX_DEPLOYMENT_TARGET always took the form of a non-integer. The code for sysconfig.get_config_var() has a pretty clear intent: it will try to cast its return value to an int.

    I don't have a strong opinion whether this is fixed this way or another. I would think, however, that there are surely places in existing code that expect and rely on sysconfig.get_config_var() returning an int for integer-type values.

    @ronaldoussoren
    Copy link
    Contributor

    The value *for this key* is a dot separated sequence of integer labels in a string. The limit of that is a single integer label that could be converted to a Python int.

    With the current patch get_config_var('MACOSX_DEPLOYMENT_TARGET') mostly returns a string, but sometimes an integer. And the last part is new behaviour, which means existing consumers of this won't be prepared to handle a non-string value (other than None).

    I'll probably create an alternative PR this weekend.

    sysconfig converts config values that look like integers to python ints because that's often useful, but in this case it isn't. This part of the code base is old and has not very well defined semantics.

    @ned-deily
    Copy link
    Member

    I agree with Ronald that it would be safest if the sysconfigs (all of them) always returned deployment target as a string and I would be fine with a PR that did that. That doesn't mean that the already applied PR needs to be changed, though, right? Just extra insurance.

    @ronaldoussoren
    Copy link
    Contributor

    I'd prefer to (effectively) revert the PR and treat MACOSX_DEPLOYMENT_TARGET=11 the same as MACOSX_DEPLOYMENT_TARGET=11.0.

    The same issue has crept up on the wheel and packaging repos (in the context of changing the platform tag in wheel names). As I wrote there: the ship has sailed w.r.t. those tags, there are already "macosx_11_0_univeral2" wheels on PyPI.

    @mattip
    Copy link
    Contributor

    mattip commented Dec 9, 2020

    It seems the approach that "The code for sysconfig.get_config_var() has a pretty clear intent: it will try to cast its return value to an int." has been accepted as true even though other parts of the code assumed the returned value was a string, and the CPython code was updated appropriately. This is causing problems for downstream libraries like NumPy, see numpy/numpy#17906. If CPython does not wish to reconsider this decision, it should at least be documented clearly via a type hint and/or in the documentation for sysconfig.get_config_var https://docs.python.org/3/library/sysconfig.html#sysconfig.get_config_var

    @lonnylot
    Copy link
    Mannequin

    lonnylot mannequin commented Jan 22, 2021

    I believe this is causing issues when trying to install pyjq (doloopwhile/pyjq#54).

    @ronaldoussoren are you still planning on creating a patch for this? Or should we plan on fixing it downstream?

    @ronaldoussoren
    Copy link
    Contributor

    I will create a PR later today, I've already committed the work in a branch of my clone of cpython.

    @ronaldoussoren
    Copy link
    Contributor

    see PR 24341.

    @ned-deily
    Copy link
    Member

    New changeset 49926cf by Ronald Oussoren in branch 'master':
    bpo-42504: Ensure that get_config_var('MACOSX_DEPLOYMENT_TARGET') is a string (GH-24341)
    49926cf

    @ned-deily
    Copy link
    Member

    New changeset d6675fe by Miss Islington (bot) in branch '3.9':
    bpo-42504: Ensure that get_config_var('MACOSX_DEPLOYMENT_TARGET') is a string (GH-24341) (GH-24410)
    d6675fe

    @ambv
    Copy link
    Contributor

    ambv commented May 2, 2021

    New changeset b29d0a5 by Ned Deily in branch '3.8':
    [3.8] bpo-41100: Support macOS 11 Big Sur and Apple Silicon Macs (bpo-25806)
    b29d0a5

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.8 only security fixes 3.9 only security fixes 3.10 only security fixes build The build process and cross-build OS-mac
    Projects
    None yet
    Development

    No branches or pull requests

    5 participants