classification
Title: The "m" ABI flag of SOABI for pymalloc is no longer needed
Type: Stage: resolved
Components: Build Versions: Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: vstinner Nosy List: hroncok, koobs, miss-islington, ned.deily, serhiy.storchaka, vstinner
Priority: deferred blocker Keywords: patch

Created on 2019-04-24 00:27 by vstinner, last changed 2019-06-15 09:39 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 12931 merged vstinner, 2019-04-24 00:33
PR 14090 merged vstinner, 2019-06-14 16:52
PR 14097 merged miss-islington, 2019-06-15 01:25
Messages (21)
msg340748 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-04-24 00:27
In Python 2.7, the WITH_PYMALLOC define (defined by ./configure --with-pymalloc which is the default) modified the Python API. For example, PyObject_MALLOC() is a macro replaced with PyObject_Malloc() with pymalloc but replaced with PyMem_MALLOC() without pymalloc.

In Python 3.8, it's no longer the case, PyObject_MALLOC is no longer modified by WITH_PYMALLOC, it's always an alias to PyObject_Malloc() for backward compatibility.

#define PyObject_MALLOC   PyObject_Malloc

More generally, WITH_PYMALLOC has no impact on the Python headers nor on the ABI in any way. Memory allocators can be switched at runtime using PYTHONMALLOC environment variable and -X dev command line option. For example, PYTHONMALLOC=malloc disables pymalloc (forces the usage of malloc() of the C library).

I propose attached PR which removes the "m" flag from SOABI in Python 3.8 when pymalloc is enabled.
msg340749 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-04-24 00:36
I modified PyObject_MALLOC and other macros when I implemented the PEP 445 "Add new APIs to customize Python memory allocators":
https://www.python.org/dev/peps/pep-0445/

ABI tags are defined by the PEP 3149:
https://www.python.org/dev/peps/pep-3149/#proposal
"--with-pymalloc (flag: m)"

The PEP 393 already made "--with-wide-unicode (flag: u)" useless in Python 3.3.
msg340757 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-04-24 03:13
If --with-pymalloc does not impact the API, why we need this option?
msg340766 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-04-24 09:44
pymalloc is enabled by default. --without-pymalloc allows to build Python
without pymalloc and use malloc by default.
msg340773 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-04-24 14:10
New changeset 6c44fde3e03079e0c69f823dafbe04af50b5bd0d by Victor Stinner in branch 'master':
bpo-36707: Remove the "m" flag (pymalloc) from SOABI (GH-12931)
https://github.com/python/cpython/commit/6c44fde3e03079e0c69f823dafbe04af50b5bd0d
msg341097 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-04-29 16:56
This change has an impact on downstream packagers and users of Python 3.8 because the SOABI flag values are also used to construct a number of file names and directories names in an Unix or macOS framework install besides the extension module (*.so) file names.  Suggest comparing the directory/file names installed to an alternate prefix (./configure --prefix=/tmp/3x ; make install) in 3.7 versus master with this PR.  Also suggest reading PEP 3149, which introduced the ABI-specific file names, and the various issues in the bug tracker that reference it.  At the very least, these changes need to be documented in the What's New for 3.8.  But the impact of the change should probably also be discussed with at least some of the large distributors.  (The change already inadvertently broke the macOS installer build - fixing.)
msg341099 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-04-29 17:03
About the PEP, I asked Barry to review the change and he approved it.

About the doc, I didn't know that anyone rely on the exact filename. I documented all changes in a top-level section of the What's New in Python 3.8:
https://docs.python.org/dev/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build

Sorry for thr annoyance. I didn't know that the macOS installer rely on these filenames.
msg341100 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-04-29 17:07
No, you didn't document all the changed file names,  See, for instance, /usr/bin/python3* for a start :)
msg341121 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-04-29 21:26
> No, you didn't document all the changed file names,  See, for instance, /usr/bin/python3* for a start :)

Oh, I see. I was thinking at something else. First I proposed to drop the "d" from the SOABI for debug build, but then I changed my mind. So yeah, I didn't change the SOABI for:
https://docs.python.org/dev/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build

And this change should be documented elsewhere.
msg341835 - (view) Author: Miro Hrončok (hroncok) * Date: 2019-05-07 22:13
> But the impact of the change should probably also be discussed with at least some of the large distributors.

Adapting the Fedora package. Will try to mass rebuild our packages to see what breaks.
msg344478 - (view) Author: Miro Hrončok (hroncok) * Date: 2019-06-03 21:31
It's usually just custom ugly-hardcoded autotools/cmake. Nothing severe enough.  However, this change still is not documented anywhere in https://docs.python.org/dev/whatsnew/3.8.html
msg344483 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-03 22:24
> It's usually just custom ugly-hardcoded autotools/cmake. Nothing severe enough.  However, this change still is not documented anywhere in https://docs.python.org/dev/whatsnew/3.8.html

I concur that it should be documented. It's on my TODO list. But I'm not sure of what are the things impacting by ABIFLAGS="" rather than ABIFLAGS="m". Someone has to propose a PR to document that.
msg344484 - (view) Author: Miro Hrončok (hroncok) * Date: 2019-06-03 22:28
I'm not sure either. So far most affected buildscripts are confused by:

/usr/include/python3.Xm -> /usr/include/python3.X
/usr/lib64/libpython3.Xm -> /usr/lib64/libpython3.X
extension.cpython-3Xm-arch-linux-gnu.so -> extension.cpython-3X-arch-linux-gnu.so

However this is pretty much Linux specific, so I wouldn't know how to explain this in an OS agnostic way.
msg344521 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2019-06-04 04:19
For FreeBSD Python language ports, the change doesn't have a big negative impact for the Python language/interpreter ports themselves

We have the following block in lang/python3? ports:

.if ${PORT_OPTIONS:MPYMALLOC}
ABIFLAGS:=      m${ABIFLAGS}
.endif

.if ${PORT_OPTIONS:MDEBUG}
ABIFLAGS:=      d${ABIFLAGS}
.endif


We then use the variable to substitute the correct suffix into the pkg-plist items:

.if !empty(ABIFLAGS)
PLIST_FILES+=   bin/python${PYTHON_VER}${ABIFLAGS} \
                bin/python${PYTHON_VER}${ABIFLAGS}-config \
                libdata/pkgconfig/python-${PYTHON_VER}${ABIFLAGS}.pc
<snip>
${INSTALL_DATA} ${WRKSRC}/Tools/gdb/libpython.py \
                ${STAGEDIR}${PREFIX}/lib/libpython${PYTHON_VER}${ABIFLAGS}.so.1.0-gdb.py

One question I have is, can/does SOABI flag removal affect any third party package/extension builds in any way, particularly affecting the names of files they produce?

We have a ton of python packages/extensions in the ports tree, so anything that affected their builds, particularly when it comes to setuptools --record output changes, would be a blocker for supporting 3.8

For example, we had to produce custom ports framework code to account for PEP488 [1] and PEP3147 [2] filename suffixes, for these filename differences, which was painful at the time:

# PEP 0488 (https://www.python.org/dev/peps/pep-0488/)
.if ${PYTHON_REL} < 3500
PYTHON_PYOEXTENSION=    pyo
.else
PYTHON_PYOEXTENSION=    opt-1.pyc
.endif


.if ${PYTHON_REL} >= 3200 && defined(_PYTHON_FEATURE_PY3KPLIST)
# When Python version is 3.2+ we rewrite all the filenames
# of TMPPLIST that end with .py[co], so that they conform
# to PEP 3147 (see https://www.python.org/dev/peps/pep-3147/)
PYMAGICTAG=             ${PYTHON_CMD} -c 'import sys; print(sys.implementation.cache_tag)'
_USES_stage+=   935:add-plist-python
add-plist-python:
        @${AWK} '\
                /\.py[co]$$/ && !($$0 ~ "/" pc "/") {id = match($$0, /\/[^\/]+\.py[co]$$/); if (id != 0) {d = substr($$0, 1, RSTART - 1); dirs[d] = 1}; sub(/\.pyc$$/,  "." mt "&"); sub(/\.pyo$$/, "." mt "." pyo); sub(/[^\/]+\.py[co]$$/, pc "/&"); print; next} \
                /^@dirrm / {d = substr($$0, 8); if (d in dirs) {print $$0 "/" pc}; print $$0; next} \
                /^@dirrmtry / {d = substr($$0, 11); if (d in dirs) {print $$0 "/" pc}; print $$0; next} \
                {print} \
                ' \
                pc="__pycache__" mt="$$(${PYMAGICTAG})" pyo="opt-1.pyc" \
                ${TMPPLIST} > ${TMPPLIST}.pyc_tmp
        @${MV} ${TMPPLIST}.pyc_tmp ${TMPPLIST}


[1] https://www.python.org/dev/peps/pep-0488/
[2] https://www.python.org/dev/peps/pep-3147/
msg344524 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-06-04 04:47
> One question I have is, can/does SOABI flag removal affect any third party package/extension builds in any way, particularly affecting the names of files they produce?

Yep, the default file names of C extension modules differ on most/all? Unix-y platforms (certainly vanilla Linux and macOS) because the SOABI is included in the file name.

$ /tmp/py38/bin/python3.7 -m pip install --no-binary :all: psutil
$ ls /tmp/py37/lib/python3.7/site-packages/psutil/*.so
/tmp/py37/lib/python3.7/site-packages/psutil/_psutil_linux.cpython-37m-i386-linux-gnu.so
/tmp/py37/lib/python3.7/site-packages/psutil/_psutil_posix.cpython-37m-i386-linux-gnu.so

$ /tmp/py38/bin/python3.8 -m pip install --no-binary :all: psutil
$ ls /tmp/py38/lib/python3.8/site-packages/psutil/*.so
/tmp/py38/lib/python3.8/site-packages/psutil/_psutil_linux.cpython-38-i386-linux-gnu.so
/tmp/py38/lib/python3.8/site-packages/psutil/_psutil_posix.cpython-38-i386-linux-gnu.so
msg344526 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-06-04 05:00
Another user-visible difference:

$ ls /tmp/py37/bin/python*
/tmp/py37/bin/python3    /tmp/py37/bin/python3.7-config  /tmp/py37/bin/python3.7m-config
/tmp/py37/bin/python3.7  /tmp/py37/bin/python3.7m        /tmp/py37/bin/python3-config
$ ls /tmp/py38/bin/python*
/tmp/py38/bin/python3    /tmp/py38/bin/python3.8-config
/tmp/py38/bin/python3.8  /tmp/py38/bin/python3-config

Granted, these differences aren't the end of the world but:

"But I'm not sure of what are the things impacting by ABIFLAGS="" rather than ABIFLAGS="m"."

Not to beat a dead horse but - we really should not make changes like this without first understanding what the impact will be.  As it stands, this change potentially affects end-users, third-party distributors of Python, and package developers.  And, for what reason?  I don't wsee how this change provides any real value for anyone and it does cause people to have to change things, like creating conditional code for 3.7- and 3.8+.  If we were starting from scratch today, sure - "m" is not needed.  But just because we can change something doesn't mean we should.
msg344528 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2019-06-04 05:14
Those script names would already have code to handle name changes based on existing abiflags/soabi string (as FreeBSD does), and would be trivial to modify for 'm' removal.

For third party stuff: we use setuptools --record for 'everything', so as long as filenames are 'correct' for the files that are actually installed, and downstreams are clear what the scope of the changes are, 'm' removal is not an issue when it come so packaging third party packages/extensions.

A new file extension format or substantially more complex magic would be, on the other hand.

A new minor is a good time to make these changes, and there's nothing really blocking about a new python38 port/package, and nothing consumes it yet in downstream OS's
msg345650 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-15 01:24
New changeset 7efc526e5cfb929a79c192ac2dcf7eb78d3a4401 by Victor Stinner in branch 'master':
bpo-36707: Document "m" removal from sys.abiflags (GH-14090)
https://github.com/python/cpython/commit/7efc526e5cfb929a79c192ac2dcf7eb78d3a4401
msg345651 - (view) Author: miss-islington (miss-islington) Date: 2019-06-15 01:31
New changeset 3fde750cc4e4057076650a92946ec1d492464799 by Miss Islington (bot) in branch '3.8':
bpo-36707: Document "m" removal from sys.abiflags (GH-14090)
https://github.com/python/cpython/commit/3fde750cc4e4057076650a92946ec1d492464799
msg345666 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-15 09:36
Ned Deily:
> My preference remains to not make the flags change at all

Well, "m" in ABI flags became useless around Python 3.4. I don't see the point of pretending that the "m" ABI is different and then provide two names for the same thing:

    python3.8 and python3.8m
    python3.8-config and python3.8m-config
    man python3.8 and man python3.8m
    etc.

I'm not sure why, but on Fedora /usr/bin/python3.7 and /usr/bin/python3.7m are two separated files... with the same content (ex: same MD5 sum). Well, it's just 17 KiB each :-) /usr/bin/python3.7-config is a symlink to python3.7m-config.

The change itself is painful for distributors like Red Hat (Fedora, RHEL), but things should be simpler once the change is done.

My long-term goal is to get a single ABI for everything: CPython, PyPy, RustPython, MicroPython, etc. :-) Maybe it's not possible, but at least I would like to move towards this goal!

Overall project:

* https://docs.python.org/3.8/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build
* https://pythoncapi.readthedocs.io/
msg345667 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-15 09:39
Thanks everybody for your help. It is now documented.
History
Date User Action Args
2019-06-15 09:39:47vstinnersetmessages: + msg345667
2019-06-15 09:36:07vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg345666

stage: patch review -> resolved
2019-06-15 01:31:48miss-islingtonsetnosy: + miss-islington
messages: + msg345651
2019-06-15 01:25:03miss-islingtonsetpull_requests: + pull_request13953
2019-06-15 01:24:44vstinnersetmessages: + msg345650
2019-06-14 16:52:17vstinnersetstage: needs patch -> patch review
pull_requests: + pull_request13946
2019-06-04 05:14:58koobssetmessages: + msg344528
2019-06-04 05:00:31ned.deilysetmessages: + msg344526
2019-06-04 04:47:05ned.deilysetmessages: + msg344524
2019-06-04 04:19:48koobssetmessages: + msg344521
2019-06-03 22:28:34hroncoksetmessages: + msg344484
2019-06-03 22:24:18vstinnersetmessages: + msg344483
2019-06-03 21:31:37hroncoksetmessages: + msg344478
2019-05-07 22:13:30hroncoksetnosy: + hroncok
messages: + msg341835
2019-04-29 21:26:15vstinnersetmessages: + msg341121
2019-04-29 17:07:21ned.deilysetmessages: + msg341100
2019-04-29 17:03:43vstinnersetmessages: + msg341099
2019-04-29 16:58:39ned.deilysetassignee: vstinner
2019-04-29 16:56:46ned.deilysetstatus: closed -> open
priority: normal -> deferred blocker


nosy: + ned.deily
messages: + msg341097
resolution: fixed -> (no value)
stage: resolved -> needs patch
2019-04-24 14:11:15vstinnersetstatus: open -> closed
resolution: fixed
components: + Build
stage: patch review -> resolved
2019-04-24 14:10:38vstinnersetmessages: + msg340773
2019-04-24 10:50:07koobssetnosy: + koobs
2019-04-24 09:44:22vstinnersetmessages: + msg340766
2019-04-24 03:13:17serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg340757
2019-04-24 00:36:49vstinnersetmessages: + msg340749
2019-04-24 00:33:53vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request12856
2019-04-24 00:27:25vstinnercreate