classification
Title: PyIndex_Check conflicts with PEP 384
Type: compile error Stage: patch review
Components: Extension Modules Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Christian.Tismer, eric.snow, larry, ned.deily, pitrou, serhiy.storchaka, vstinner
Priority: critical Keywords: patch

Created on 2018-06-01 17:14 by Christian.Tismer, last changed 2018-06-11 12:01 by serhiy.storchaka.

Pull Requests
URL Status Linked Edit
PR 7477 closed Christian.Tismer, 2018-06-07 10:27
PR 7585 merged Christian.Tismer, 2018-06-10 12:53
PR 7581 serhiy.storchaka, 2018-06-10 22:29
PR 7627 merged serhiy.storchaka, 2018-06-11 11:15
Messages (24)
msg318436 - (view) Author: Christian Tismer (Christian.Tismer) * (Python committer) Date: 2018-06-01 17:14
The file number.rst on python 3.6 says

"""
.. c:function:: int PyIndex_Check(PyObject *o)

   Returns ``1`` if *o* is an index integer (has the nb_index slot of the
   tp_as_number structure filled in), and ``0`` otherwise.
"""

But in reality, this is a macro:

"""
#define PyIndex_Check(obj) \
   ((obj)->ob_type->tp_as_number != NULL && \
    (obj)->ob_type->tp_as_number->nb_index != NULL)
"""

But such a macro does not work with the limited API, since
non-heaptypes cannot use PyType_GetSlot.

The patch is trivial: Define the function instead of the macro
when Py_LIMITED_API is set.

I also think the documentation makes more sense when it is correct.
msg318449 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-06-01 18:25
This is not a security issue so a change would not be applicable to the 3.4 or 3.5 branches, currently in security-fix-only mode.
msg318458 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-06-01 20:34
Seems PyIter_Check and PySequence_ITEM have the same problem. And maybe more.
msg318461 - (view) Author: Christian Tismer (Christian.Tismer) * (Python committer) Date: 2018-06-01 20:45
Ok, I tried to submit a patch (not yet successful),
but as it stands, there are more locations in the code where
this will show up with similar problems.

Should I take care of these all as much as I can,
or is it better to leave it to one of you?

Whatever you prefer is ok. I just want the problem to disappear
in 3.7, because I hate to break the API any longer.

Cheers - Chris
msg318462 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-06-01 20:53
Would be nice if you fix as much similar as you can with a single PR.

For performance it makes sense to keep macros if the limited API is not used.
msg318465 - (view) Author: Christian Tismer (Christian.Tismer) * (Python committer) Date: 2018-06-01 21:12
Yes, sure, I will submit a patch that tries to reach as much
as possible locations that have a similar problem. Of course,
the code will only be turned into functions for the PEP context.

Takes a day because I need to re-learn a bit how to do this :-)
(last patch was 2010, I think... )

-- related personal note - does someone know what happened to Martin?
msg318516 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-06-02 22:06
> related personal note - does someone know what happened to Martin?

I don't think anything bad happened.  He still seems to be teaching:
http://www.beuth-hochschule.de/people/detail/1398/
msg318908 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-06-07 07:43
Christian, any progress on this?  3.7.0rc1 is planned to be tagged in 4 days, on Monday 2018-06-11.  Do you think you will be able to provide a PR before then?
msg318909 - (view) Author: Christian Tismer (Christian.Tismer) * (Python committer) Date: 2018-06-07 08:02
Hi Ned,

we had a delivery date yesterday for PySide.
The PR is almost ready and will go out today.

Ciao - Chris

On 07.06.18 09:43, Ned Deily wrote:
> 
> Ned Deily <nad@python.org> added the comment:
> 
> Christian, any progress on this?  3.7.0rc1 is planned to be tagged in 4 days, on Monday 2018-06-11.  Do you think you will be able to provide a PR before then?
> 
> ----------
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue33738>
> _______________________________________
>
msg318910 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-06-07 08:03
Thanks, Christian!
msg318925 - (view) Author: Christian Tismer (Christian.Tismer) * (Python committer) Date: 2018-06-07 10:50
I did not understand the Misc/NEWS.d thing.
What should go where, or where would a "skip news" label go?
msg318951 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2018-06-07 15:59
@Christian, you can use the "blurb" tool to create the NEWS entry.  You can use pip to install it.  See:

  https://devguide.python.org/committing/?highlight=blurb#what-s-new-and-news-entries
msg318952 - (view) Author: Christian Tismer (Christian.Tismer) * (Python committer) Date: 2018-06-07 16:09
On 07.06.18 17:59, Eric Snow wrote:
> 
> Eric Snow <ericsnowcurrently@gmail.com> added the comment:
> 
> @Christian, you can use the "blurb" tool to create the NEWS entry.  You can use pip to install it.  See:
> 
>   https://devguide.python.org/committing/?highlight=blurb#what-s-new-and-news-entries
> 
> ----------
> nosy: +eric.snow
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue33738>
> _______________________________________
> 

Thank you, Eric! Will do so.
I had something to change, anyway :-)
msg319018 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-06-08 05:18
Excluding names from limited API can break existing code that use them with defined Py_LIMITED_API. I wondering if corresponding functions should be added for PySequence_ITEM, PyObject_IS_GC, PyType_SUPPORTS_WEAKREFS, PyObject_GET_WEAKREFS_LISTPTR. Perhaps this should be discussed on Python-Dev.

Since PyList_GET_SIZE and PyList_GET_ITEM are defined only for non-limited API, it is better to wrap definitions of macros that use them (like PySequence_Fast_GET_SIZE and PySequence_Fast_GET_ITEM) in "#ifndef Py_LIMITED_API" ... "#endif".
msg319073 - (view) Author: Christian Tismer (Christian.Tismer) * (Python committer) Date: 2018-06-08 14:58
"""Excluding names from limited API can break existing code that use them with defined Py_LIMITED_API."""

How is that different? Right now, the code would break at compile time,
because the macros are accessing opaque type fields. Excluding them has
the same effect but is much cleaner.

My current first approach is conservative, because it only makes things
work that didn't work before.
Core that is clearly meant as macro is obviously not meant for the limited
API. And if it should be better included, I'm all open for it.

Right now I want to remove quickly the breakage that was a showstopper.
Maybe I misunderstood you and did exactly what you proposed?

Please let us start a discussion on python-dev. I think there is more to
do to make the limited API really usable in large projects.
msg319169 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-06-09 18:32
New changeset ea62ce7f4fefc66bc0adba16bcd7666d5bbd5b44 by Ned Deily (Christian Tismer) in branch 'master':
bpo-33738: Fix macros which contradict PEP 384 (GH-7477)
https://github.com/python/cpython/commit/ea62ce7f4fefc66bc0adba16bcd7666d5bbd5b44
msg319174 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-06-09 19:36
As I noted on PR 7477, I decided to merge the change to master so it would get some buildbot exposure before deciding whether to backport for 3.7.0rc1 which is coming up shortly.  I notice that the change introduced a compiler warning.  Should that be fixed?

..\Objects\exceptions.c(349): warning C4090: 'return': different 'const' qualifiers [D:\buildarea\3.x.ware-win81-release\build\PCbuild\pythoncore.vcxproj]

http://buildbot.python.org/all/#/builders/12/builds/960
msg319197 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-06-10 04:55
You are right Christian. I missed that PyTypeObject is opaque if Py_LIMITED_API is defined.
msg319207 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-06-10 11:29
It seems like the change broke compilation on Windows, but I'm not sure:

http://buildbot.python.org/all/#/builders/12/builds/959

       (Link target) -> 
         python3.def : error LNK2001: unresolved external symbol PyExceptionClass_Name [D:\buildarea\3.x.ware-win81-release\build\PCbuild\python3dll.vcxproj]
         python3.def : error LNK2001: unresolved external symbol PyIndex_Check [D:\buildarea\3.x.ware-win81-release\build\PCbuild\python3dll.vcxproj]
         python3.def : error LNK2001: unresolved external symbol PyIter_Check [D:\buildarea\3.x.ware-win81-release\build\PCbuild\python3dll.vcxproj]
         D:\buildarea\3.x.ware-win81-release\build\PCbuild\amd64\python3.lib : fatal error LNK1120: 3 unresolved externals [D:\buildarea\3.x.ware-win81-release\build\PCbuild\python3dll.vcxproj]

On the buildbot-status, Ned Deily wrote:

Build failure caused by known failure to rebuild files during compile phase. Worked around by clearing out stale files.  (I *think* there is still an open issue on this.)

Ned: you are thinking at bpo-33614, commit e97ba4c690613d734843db218aeedce2f0e5937f?
msg319217 - (view) Author: Christian Tismer (Christian.Tismer) * (Python committer) Date: 2018-06-10 13:10
@Victor
I cannot test on Windows because I'm in vacation.
But it is very likely similar to bpo-33614 .
The three missing symbols which are listed in python3.def
do clearly come into existence when the limited API is active.
msg319256 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-06-10 22:48
New changeset 8398713cea0eb17b013f25f86bef47c7e5e63139 by Ned Deily (Christian Tismer) in branch 'master':
bpo-33738: Address review comments in GH #7477 (GH-7585)
https://github.com/python/cpython/commit/8398713cea0eb17b013f25f86bef47c7e5e63139
msg319259 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-06-11 01:19
Sigh!  I was hoping we could get this in for 3.7.0 but I think we have run out of time and we really should not be making potential user-visible API changes at this last minute.  I did notice the new compile warning for the Windows non-debug build but I overlooked that some other non-Windows buildbots were getting them, too.  Serihy proposes a more general fix in Issue33818 but I don't think we should be making a change like that just prior to the release candidate.

And, in retrospect, I should not have considered trying to fix the stable ABI support at this late date, either. It looks like it has been broken for some time now so 3.7.0 will not be any worse.  At this point, we have Christian's two PRs in master now; if necessary, they could be reverted.  I will bow out of this discussion and let you all figure out what is best for master/3.8.  Once the changes for master are in and working, we could revisit the question of backports to 3.7 and/or 3.6 maintenance releases. I would like to both thank and apologize to Christian, in particular, and to Serhiy and everyone else who went out of their ways to try to get this in.

Lowering priority to "critical".
msg319297 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-06-11 11:35
AMD64 Windows8.1 Non-Debug 3.x buildbot is back to green.
msg319301 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-06-11 12:01
New changeset 5cbefa99198729a1d4e93d93f890c066039ee1d2 by Serhiy Storchaka in branch 'master':
Clean up after bpo-33738. (GH-7627)
https://github.com/python/cpython/commit/5cbefa99198729a1d4e93d93f890c066039ee1d2
History
Date User Action Args
2018-06-11 12:01:50serhiy.storchakasetmessages: + msg319301
2018-06-11 11:35:30vstinnersetmessages: + msg319297
2018-06-11 11:15:19serhiy.storchakasetpull_requests: + pull_request7246
2018-06-11 01:19:20ned.deilysetpriority: release blocker -> critical

messages: + msg319259
versions: - Python 3.6, Python 3.7
2018-06-10 22:48:30ned.deilysetmessages: + msg319256
2018-06-10 22:29:31serhiy.storchakasetpull_requests: + pull_request7227
2018-06-10 13:10:36Christian.Tismersetmessages: + msg319217
2018-06-10 12:53:54Christian.Tismersetpull_requests: + pull_request7207
2018-06-10 11:29:59vstinnersetnosy: + vstinner
messages: + msg319207
2018-06-10 04:55:37serhiy.storchakasetmessages: + msg319197
2018-06-09 19:36:44ned.deilysetmessages: + msg319174
2018-06-09 18:32:27ned.deilysetmessages: + msg319169
2018-06-08 14:58:15Christian.Tismersetmessages: + msg319073
2018-06-08 05:18:01serhiy.storchakasetmessages: + msg319018
2018-06-07 16:09:03Christian.Tismersetmessages: + msg318952
2018-06-07 15:59:52eric.snowsetnosy: + eric.snow
messages: + msg318951
2018-06-07 10:51:08Christian.Tismersetpull_requests: - pull_request7096
2018-06-07 10:50:56Christian.Tismersetpull_requests: - pull_request6956
2018-06-07 10:50:15Christian.Tismersetmessages: + msg318925
2018-06-07 10:27:27Christian.Tismersetpull_requests: + pull_request7100
2018-06-07 09:18:20Christian.Tismersetpull_requests: + pull_request7096
2018-06-07 08:03:40ned.deilysetmessages: + msg318910
2018-06-07 08:02:28Christian.Tismersetmessages: + msg318909
2018-06-07 07:43:20ned.deilysetmessages: + msg318908
2018-06-02 22:06:58pitrousetnosy: + pitrou
messages: + msg318516
2018-06-01 21:12:53Christian.Tismersetmessages: + msg318465
2018-06-01 20:53:57serhiy.storchakasetmessages: + msg318462
2018-06-01 20:45:57Christian.Tismersetmessages: + msg318461
2018-06-01 20:34:48serhiy.storchakasetmessages: + msg318458
2018-06-01 18:25:08ned.deilysetmessages: + msg318449
versions: + Python 3.8, - Python 3.4, Python 3.5
2018-06-01 17:25:20Christian.Tismersetkeywords: + patch
stage: patch review
pull_requests: + pull_request6956
2018-06-01 17:19:27ned.deilysetnosy: + serhiy.storchaka
2018-06-01 17:14:58Christian.Tismercreate