classification
Title: Failure to build with MACOSX_DEPLOYMENT_TARGET=11 on Big Sur
Type: compile error Stage: resolved
Components: Build, macOS Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: fxcoudert, lonnylot, lukasz.langa, mattip, miss-islington, ned.deily, ronaldoussoren
Priority: normal Keywords: patch

Created on 2020-11-29 15:16 by fxcoudert, last changed 2021-05-02 09:31 by ned.deily. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 23556 merged fxcoudert, 2020-11-29 15:28
PR 23622 merged miss-islington, 2020-12-03 03:20
PR 24341 merged ronaldoussoren, 2021-01-26 14:39
PR 24410 merged miss-islington, 2021-02-01 03:30
PR 25806 merged ned.deily, 2021-05-02 08:24
Messages (16)
msg382071 - (view) Author: FX Coudert (fxcoudert) * Date: 2020-11-29 15:16
With macOS Big Sur (11.y.z), the major version is 11 (instead of the earlier 10.15, 10.14, etc). Therefore, MACOSX_DEPLOYMENT_TARGET=11 has become a valid setting. It is accepted by system tools:

$ MACOSX_DEPLOYMENT_TARGET=11 clang a.c && echo $?
0

But it leads to failure in Python compilation in setup.py:

DYLD_FRAMEWORK_PATH=/private/tmp/python@3.9-20201127-81873-q9b09s/Python-3.9.0 CC='clang' LDSHARED='clang -bundle -undefined dynamic_lookup -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk   ' OPT='-DNDEBUG -g -fwrapv -O3 -Wall'   _TCLTK_INCLUDES='' _TCLTK_LIBS=''       ./python.exe -E ./setup.py  build
running build
running build_ext
Traceback (most recent call last):
  File "/private/tmp/python@3.9-20201127-81873-q9b09s/Python-3.9.0/./setup.py", line 2614, in <module>
    main()
  File "/private/tmp/python@3.9-20201127-81873-q9b09s/Python-3.9.0/./setup.py", line 2584, in main
    setup(# PyPI Metadata (PEP 301)
  File "/private/tmp/python@3.9-20201127-81873-q9b09s/Python-3.9.0/Lib/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/private/tmp/python@3.9-20201127-81873-q9b09s/Python-3.9.0/Lib/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/private/tmp/python@3.9-20201127-81873-q9b09s/Python-3.9.0/Lib/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/private/tmp/python@3.9-20201127-81873-q9b09s/Python-3.9.0/Lib/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/private/tmp/python@3.9-20201127-81873-q9b09s/Python-3.9.0/Lib/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/private/tmp/python@3.9-20201127-81873-q9b09s/Python-3.9.0/Lib/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/private/tmp/python@3.9-20201127-81873-q9b09s/Python-3.9.0/Lib/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/private/tmp/python@3.9-20201127-81873-q9b09s/Python-3.9.0/./setup.py", line 470, in build_extensions
    self.detect_modules()
  File "/private/tmp/python@3.9-20201127-81873-q9b09s/Python-3.9.0/./setup.py", line 1807, in detect_modules
    self.detect_readline_curses()
  File "/private/tmp/python@3.9-20201127-81873-q9b09s/Python-3.9.0/./setup.py", line 1038, in detect_readline_curses
    (tuple(int(n) for n in dep_target.split('.')[0:2])
AttributeError: 'int' object has no attribute 'split'
make: *** [sharedmods] Error 1

See the full report here: https://github.com/Homebrew/brew/issues/9329

The problem is here: https://github.com/python/cpython/blob/master/setup.py#L1015
When MACOSX_DEPLOYMENT_TARGET=11, this call:
sysconfig.get_config_var('MACOSX_DEPLOYMENT_TARGET')
returns an int (with value 11) instead of a string (as is the case if 11.0, 10.15, etc).

And therefore the subsequent call to: dep_target.split('.')[0:2] fails, because split() cannot be called on an int.
msg382116 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2020-11-30 08:14
Good catch.
msg382371 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-12-03 03:20
New changeset 5291639e611dc3f55a34666036f2c3424648ba50 by FX Coudert in branch 'master':
bpo-42504: fix for MACOSX_DEPLOYMENT_TARGET=11 (GH-23556)
https://github.com/python/cpython/commit/5291639e611dc3f55a34666036f2c3424648ba50
msg382372 - (view) Author: miss-islington (miss-islington) Date: 2020-12-03 03:43
New changeset 09a698b4743c669983d606595a1b2daeff6c3cf8 by Miss Islington (bot) in branch '3.9':
bpo-42504: fix for MACOSX_DEPLOYMENT_TARGET=11 (GH-23556)
https://github.com/python/cpython/commit/09a698b4743c669983d606595a1b2daeff6c3cf8
msg382402 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2020-12-03 11:32
FWIW I don't agree with this fix. 

sys.get_config_var('MACOSX_DEPLOYMENT_TARGET') has always resulted in a string value. There's bound to be someone relying on this.

As I wrote in comment on the PR a nicer fix is to teach sysconfig (and distutils.sysconfig) to not convert the value of this key to an integer.
msg382420 - (view) Author: FX Coudert (fxcoudert) * Date: 2020-12-03 15:53
"It has always resulted in a string value": only MACOSX_DEPLOYMENT_TARGET always took the form of a non-integer. The code for sysconfig.get_config_var() has a pretty clear intent: it will try to cast its return value to an int.

I don't have a strong opinion whether this is fixed this way or another. I would think, however, that there are surely places in existing code that expect and rely on sysconfig.get_config_var() returning an int for integer-type values.
msg382439 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2020-12-03 18:53
The value *for this key* is a dot separated sequence of integer labels in a string. The limit of that is a single integer label that could be converted to a Python int. 

With the current patch get_config_var('MACOSX_DEPLOYMENT_TARGET') mostly returns a string, but sometimes an integer.  And the last part is new behaviour, which means existing consumers of this won't be prepared to handle a non-string value (other than None).

I'll probably create an alternative PR this weekend.


sysconfig converts config values that look like integers to python ints because that's often useful, but in this case it isn't. This part of the code base is old and has not very well defined semantics.
msg382440 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-12-03 18:57
I agree with Ronald that it would be safest if the sysconfigs (all of them) always returned deployment target as a string and I would be fine with a PR that did that. That doesn't mean that the already applied PR needs to be changed, though, right? Just extra insurance.
msg382442 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2020-12-03 19:07
I'd prefer to (effectively) revert the PR and treat MACOSX_DEPLOYMENT_TARGET=11 the same as MACOSX_DEPLOYMENT_TARGET=11.0.

The same issue has crept up on the wheel and packaging repos (in the context of changing the platform tag in wheel names).  As I wrote there: the ship has sailed w.r.t. those tags, there are already "macosx_11_0_univeral2" wheels on PyPI.
msg382789 - (view) Author: mattip (mattip) * Date: 2020-12-09 07:02
It seems the approach that "The code for sysconfig.get_config_var() has a pretty clear intent: it will try to cast its return value to an int." has been accepted as true even though other parts of the code assumed the returned value was a string, and the CPython code was updated appropriately. This is causing problems for downstream libraries like NumPy, see https://github.com/numpy/numpy/pull/17906. If CPython does not wish to reconsider this decision, it should at least be documented clearly via a type hint and/or in the documentation for sysconfig.get_config_var https://docs.python.org/3/library/sysconfig.html#sysconfig.get_config_var
msg385502 - (view) Author: Lonny Kapelushnik (lonnylot) Date: 2021-01-22 16:02
I believe this is causing issues when trying to install pyjq (https://github.com/doloopwhile/pyjq/issues/54).

@ronaldoussoren are you still planning on creating a patch for this? Or should we plan on fixing it downstream?
msg385715 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2021-01-26 13:51
I will create a PR later today, I've already committed the work in a branch of my clone of cpython.
msg385718 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2021-01-26 14:39
see PR 24341.
msg386053 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2021-02-01 03:30
New changeset 49926cf2bcc8b44d9b8f148d81979ada191dd9d5 by Ronald Oussoren in branch 'master':
bpo-42504: Ensure that get_config_var('MACOSX_DEPLOYMENT_TARGET') is a string (GH-24341)
https://github.com/python/cpython/commit/49926cf2bcc8b44d9b8f148d81979ada191dd9d5
msg386054 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2021-02-01 04:22
New changeset d6675fee1aac1af87dc31c411d04a5bfc1f39079 by Miss Islington (bot) in branch '3.9':
bpo-42504: Ensure that get_config_var('MACOSX_DEPLOYMENT_TARGET') is a string (GH-24341) (GH-24410)
https://github.com/python/cpython/commit/d6675fee1aac1af87dc31c411d04a5bfc1f39079
msg392681 - (view) Author: Ɓukasz Langa (lukasz.langa) * (Python committer) Date: 2021-05-02 09:19
New changeset b29d0a5a7811418c0a1082ca188fd4850185e290 by Ned Deily in branch '3.8':
[3.8] bpo-41100: Support macOS 11 Big Sur and Apple Silicon Macs (#25806)
https://github.com/python/cpython/commit/b29d0a5a7811418c0a1082ca188fd4850185e290
History
Date User Action Args
2021-05-02 09:31:51ned.deilysetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-05-02 09:19:05lukasz.langasetnosy: + lukasz.langa
messages: + msg392681
2021-05-02 08:24:46ned.deilysetpull_requests: + pull_request24492
2021-02-01 04:22:55ned.deilysetmessages: + msg386054
2021-02-01 03:30:30miss-islingtonsetpull_requests: + pull_request23225
2021-02-01 03:30:14ned.deilysetmessages: + msg386053
2021-01-26 14:39:49ronaldoussorensetmessages: + msg385718
2021-01-26 14:39:12ronaldoussorensetpull_requests: + pull_request23160
2021-01-26 13:51:11ronaldoussorensetmessages: + msg385715
2021-01-22 16:02:35lonnylotsetnosy: + lonnylot
messages: + msg385502
2020-12-09 07:02:32mattipsetnosy: + mattip
messages: + msg382789
2020-12-03 19:07:49ronaldoussorensetmessages: + msg382442
2020-12-03 18:57:22ned.deilysetmessages: + msg382440
2020-12-03 18:53:15ronaldoussorensetmessages: + msg382439
2020-12-03 15:53:54fxcoudertsetmessages: + msg382420
2020-12-03 11:32:11ronaldoussorensetmessages: + msg382402
2020-12-03 03:43:15miss-islingtonsetmessages: + msg382372
2020-12-03 03:23:53ned.deilysetversions: + Python 3.8, Python 3.10
2020-12-03 03:20:56miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request22490
2020-12-03 03:20:51ned.deilysetmessages: + msg382371
2020-11-30 08:14:17ronaldoussorensetmessages: + msg382116
2020-11-29 15:28:30fxcoudertsetkeywords: + patch
stage: patch review
pull_requests: + pull_request22437
2020-11-29 15:21:19guillochonsetnosy: + ned.deily, ronaldoussoren
components: + macOS
2020-11-29 15:16:04fxcoudertcreate