classification
Title: Implement PEP 590
Type: enhancement Stage: resolved
Components: Interpreter Core Versions: Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Mark.Shannon, benjamin.peterson, jdemeyer, lukasz.langa, miss-islington, pablogsal, petr.viktorin, vstinner
Priority: normal Keywords: patch

Created on 2019-05-20 18:01 by jdemeyer, last changed 2019-11-12 13:08 by miss-islington. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 13450 merged jdemeyer, 2019-05-20 18:02
PR 13185 merged jdemeyer, 2019-05-21 09:49
PR 13460 merged jdemeyer, 2019-05-21 10:44
PR 13464 merged jdemeyer, 2019-05-22 05:41
PR 13498 merged jdemeyer, 2019-05-22 14:58
PR 13653 merged jdemeyer, 2019-05-29 13:39
PR 13665 merged petr.viktorin, 2019-05-29 19:27
PR 13682 merged jdemeyer, 2019-05-30 15:15
PR 13699 merged petr.viktorin, 2019-05-31 14:20
PR 13758 merged petr.viktorin, 2019-06-02 22:39
PR 13781 merged jdemeyer, 2019-06-03 15:39
PR 13844 merged jdemeyer, 2019-06-05 15:36
PR 13858 merged jdemeyer, 2019-06-06 09:05
PR 14342 merged miss-islington, 2019-06-24 14:40
PR 14782 merged jdemeyer, 2019-07-15 12:46
Messages (26)
msg343908 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2019-05-29 18:31
New changeset aacc77fbd77640a8f03638216fa09372cc21673d by Petr Viktorin (Jeroen Demeyer) in branch 'master':
bpo-36974: implement PEP 590 (GH-13185)
https://github.com/python/cpython/commit/aacc77fbd77640a8f03638216fa09372cc21673d
msg343922 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2019-05-29 20:45
New changeset fecb75c1bb46c818e6579ba422cfa5d0d9d104d1 by Petr Viktorin in branch 'master':
bpo-36974: Fix GDB integration (GH-13665)
https://github.com/python/cpython/commit/fecb75c1bb46c818e6579ba422cfa5d0d9d104d1
msg343929 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-05-29 21:52
BUILDBOT FAILURE REPORT
=======================

Builder name: AMD64 Ubuntu Shared 3.x
Builder url: https://buildbot.python.org/all/#/builders/141/
Build url: https://buildbot.python.org/all/#/builders/141/builds/1866

Failed tests
------------

- test_pycfunction (test.test_gdb.PyBtTests)


Test leaking resources
----------------------



Build summary
-------------

== Tests result: FAILURE then FAILURE ==

405 tests OK.

10 slowest tests:
- test_multiprocessing_spawn: 6 min 27 sec
- test_tools: 5 min 34 sec
- test_concurrent_futures: 5 min 13 sec
- test_tokenize: 4 min 43 sec
- test_lib2to3: 3 min 58 sec
- test_gdb: 3 min 16 sec
- test_multiprocessing_forkserver: 2 min 25 sec
- test_asyncio: 2 min 5 sec
- test_multiprocessing_fork: 1 min 40 sec
- test_capi: 1 min 36 sec

1 test failed:
    test_gdb

17 tests skipped:
    test_devpoll test_idle test_ioctl test_kqueue test_msilib
    test_ossaudiodev test_startfile test_tcl test_tix test_tk
    test_ttk_guionly test_ttk_textonly test_turtle test_winconsoleio
    test_winreg test_winsound test_zipfile64

1 re-run test:
    test_gdb

Total duration: 42 min 45 sec


Tracebacks
----------

```traceback
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_gdb.py", line 890, in test_pycfunction
    self.assertIn('#2 <built-in method gmtime', gdb_output)
AssertionError: '#2 <built-in method gmtime' not found in 'Breakpoint 1 (time_gmtime) pending.\n[Thread debugging using libthread_db enabled]\nUsing host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".\n\nBreakpoint 1, time_gmtime (self=<module at remote 0x7ffff7f05fb0>, args=(1,)) at ./Modules/timemodule.c:446\n446\t{\n#6 Frame 0x7ffff7f11a50, for file <string>, line 3, in foo ()\n#12 Frame 0x5555557be620, for file <string>, line 5, in bar ()\n#18 Frame 0x5555557be3f0, for file <string>, line 6, in <module> ()\n'


Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_gdb.py", line 890, in test_pycfunction
    self.assertIn('#2 <built-in method gmtime', gdb_output)
AssertionError: '#2 <built-in method gmtime' not found in 'Breakpoint 1 (time_gmtime) pending.\n[Thread debugging using libthread_db enabled]\nUsing host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".\n\nBreakpoint 1, time_gmtime (self=<module at remote 0x7ffff7f05fb0>, args=(1,)) at ./Modules/timemodule.c:446\n446\t{\n#6 Frame 0x7ffff7f11a50, for file <string>, line 3, in foo ()\n#12 Frame 0x5555557be590, for file <string>, line 5, in bar ()\n#18 Frame 0x5555557be360, for file <string>, line 6, in <module> ()\n'

```

Current builder status
----------------------

The builder is failing currently

Commits
-------

- 0c2f9305640f7655ba0cd5f478948b2763b376b3

- aacc77fbd77640a8f03638216fa09372cc21673d


Other builds with similar failures
----------------------------------

-  https://buildbot.python.org/all/#/builders/21/builds/3076
-  https://buildbot.python.org/all/#/builders/21/builds/3077
-  https://buildbot.python.org/all/#/builders/21/builds/3078
-  https://buildbot.python.org/all/#/builders/21/builds/3079
-  https://buildbot.python.org/all/#/builders/21/builds/3080
-  https://buildbot.python.org/all/#/builders/13/builds/3087
-  https://buildbot.python.org/all/#/builders/13/builds/3088
-  https://buildbot.python.org/all/#/builders/13/builds/3089
-  https://buildbot.python.org/all/#/builders/13/builds/3090
-  https://buildbot.python.org/all/#/builders/85/builds/2883
-  https://buildbot.python.org/all/#/builders/85/builds/2884
-  https://buildbot.python.org/all/#/builders/85/builds/2885
-  https://buildbot.python.org/all/#/builders/85/builds/2886
-  https://buildbot.python.org/all/#/builders/141/builds/1869
-  https://buildbot.python.org/all/#/builders/176/builds/590
-  https://buildbot.python.org/all/#/builders/176/builds/591
-  https://buildbot.python.org/all/#/builders/176/builds/592
-  https://buildbot.python.org/all/#/builders/176/builds/593
-  https://buildbot.python.org/all/#/builders/176/builds/594
-  https://buildbot.python.org/all/#/builders/16/builds/3055
-  https://buildbot.python.org/all/#/builders/16/builds/3056
-  https://buildbot.python.org/all/#/builders/16/builds/3057
-  https://buildbot.python.org/all/#/builders/16/builds/3058
-  https://buildbot.python.org/all/#/builders/16/builds/3059

Common commits for all builds:

- aacc77fbd77640a8f03638216fa09372cc21673d
msg343930 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-05-29 21:56
@Petr is https://bugs.python.org/issue37090 and https://github.com/python/cpython/pull/13668 also addressing the buildbot failures?
msg343932 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2019-05-29 22:12
No, just https://github.com/python/cpython/pull/13665 is addressing the  failures. And it seems that it's successful -- only the slowest of the failed ones (AMD64 Ubuntu Shared 3.x) is still red.

bpo-37090 extends the test so it can catch more bugs in the future (but there's no rush to get it in...)
msg343934 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-05-29 22:16
Great! Thank you very much for the quick fix for the problem.

For AMD64 Ubuntu Shared 3.x, the last build was successful:

https://buildbot.python.org/all/#/builders/141/builds/1870/steps/5/logs/stdio
msg343935 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2019-05-29 22:17
All stable buildbots are back to green.
msg343967 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2019-05-30 10:43
New changeset 735e8afa9ee942367b5d0807633a2b9f662cbdbf by Petr Viktorin (Jeroen Demeyer) in branch 'master':
bpo-36974: inherit the vectorcall protocol (GH-13498)
https://github.com/python/cpython/commit/735e8afa9ee942367b5d0807633a2b9f662cbdbf
msg343968 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2019-05-30 10:44
New changeset c145f3bfbe80d498d40848450d4d33c14e2cf782 by Petr Viktorin (Jeroen Demeyer) in branch 'master':
bpo-36974: remove _PyObject_HasFastCall (GH-13460)
https://github.com/python/cpython/commit/c145f3bfbe80d498d40848450d4d33c14e2cf782
msg343975 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2019-05-30 13:11
New changeset 37788bc23f6f1ed0362b9b3b248daf296c024849 by Petr Viktorin (Jeroen Demeyer) in branch 'master':
bpo-36974: rename _FastCallKeywords -> _Vectorcall (GH-13653)
https://github.com/python/cpython/commit/37788bc23f6f1ed0362b9b3b248daf296c024849
msg344016 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2019-05-31 02:13
New changeset 530f506ac91338b55cf2be71b1cdf50cb077512f by Benjamin Peterson (Jeroen Demeyer) in branch 'master':
bpo-36974: tp_print -> tp_vectorcall_offset and tp_reserved -> tp_as_async (GH-13464)
https://github.com/python/cpython/commit/530f506ac91338b55cf2be71b1cdf50cb077512f
msg344077 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2019-05-31 14:08
I found an issue in PEP 590:

When inheriting a heap subclass from a vectorcall class that sets .tp_call=PyVectorcall_Call (as recommended), the subclass does not inherit _Py_TPFLAGS_HAVE_VECTORCALL, and thus PyVectorcall_Call does not work for it.
Possible solutions come to mind:
- Inherit tp_vectorcall_offset more normally but handle setting __call__ specially
- Inherit tp_vectorcall_offset (but not _Py_TPFLAGS_HAVE_VECTORCALL) more normally, and make PyVectorcall_Call ignore _Py_TPFLAGS_HAVE_VECTORCALL

I'll send a PR for the latter.
msg344328 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2019-06-02 21:52
New changeset fb9423fd0a85f06affb8c3a8f25dd598a649aa42 by Petr Viktorin in branch 'master':
bpo-36974: Make tp_call=PyVectorcall_Call work for inherited types (GH-13699)
https://github.com/python/cpython/commit/fb9423fd0a85f06affb8c3a8f25dd598a649aa42
msg344333 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2019-06-02 23:08
New changeset e584cbff1ea78e700cf9943d50467e3b58301ccc by Petr Viktorin in branch 'master':
bpo-36027 bpo-36974: Fix "incompatible pointer type" compiler warnings (GH-13758)
https://github.com/python/cpython/commit/e584cbff1ea78e700cf9943d50467e3b58301ccc
msg344339 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2019-06-02 23:43
New changeset 9e3e06e582accec82eb29cf665c3b4c7d84d2eb0 by Petr Viktorin (Jeroen Demeyer) in branch 'master':
bpo-36974: document PEP 590 (GH-13450)
https://github.com/python/cpython/commit/9e3e06e582accec82eb29cf665c3b4c7d84d2eb0
msg344340 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2019-06-02 23:57
New changeset be718c33f06b3496faa61142df24fb071fd5d1f1 by Petr Viktorin (Jeroen Demeyer) in branch 'master':
bpo-36974: add some assertions for PEP 590 (GH-13682)
https://github.com/python/cpython/commit/be718c33f06b3496faa61142df24fb071fd5d1f1
msg346366 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2019-06-24 10:41
New changeset a8b27e623d75377aabe50df27e97cab4e81a174a by Petr Viktorin (Jeroen Demeyer) in branch 'master':
bpo-36974: inherit tp_vectorcall_offset unconditionally (GH-13858)
https://github.com/python/cpython/commit/a8b27e623d75377aabe50df27e97cab4e81a174a
msg346492 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2019-06-25 08:19
New changeset 26fe6c35374fa32577b230b856a92a3b094e08ed by Petr Viktorin (Miss Islington (bot)) in branch '3.8':
bpo-36974: inherit tp_vectorcall_offset unconditionally (GH-13858) (GH-14342)
https://github.com/python/cpython/commit/26fe6c35374fa32577b230b856a92a3b094e08ed
msg347338 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2019-07-05 12:48
New changeset 0d722f3cd602e5f5492f9c65c8af57ea9d3743b6 by Petr Viktorin (Jeroen Demeyer) in branch 'master':
bpo-36974: separate vectorcall functions for each calling convention (GH-13781)
https://github.com/python/cpython/commit/0d722f3cd602e5f5492f9c65c8af57ea9d3743b6
msg347340 - (view) Author: Jeroen Demeyer (jdemeyer) * (Python triager) Date: 2019-07-05 12:52
Any objections to closing this?
msg347347 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2019-07-05 14:03
Buildbots are green, closing.

Thank you for the implementation!
msg348327 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2019-07-23 10:39
New changeset bf8e82f976b37856c7d35cdf88a238cb6f57fe65 by Łukasz Langa (Jeroen Demeyer) in branch '3.8':
[3.8] bpo-36974: separate vectorcall functions for each calling convention (GH-13781) (#14782)
https://github.com/python/cpython/commit/bf8e82f976b37856c7d35cdf88a238cb6f57fe65
msg348804 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-07-31 13:29
> [3.8] bpo-36974: separate vectorcall functions for each calling convention (GH-13781) (#14782)
> https://github.com/python/cpython/commit/bf8e82f976b37856c7d35cdf88a238cb6f57fe65

FYI this change caused a regression in libcomps with Python 3.8 beta3, whereas it works well with Python 3.8 beta2.

It's not a bug in Python, but it was a bug in libcomps (already fixed upstream). I just fixed libcomps:
https://github.com/rpm-software-management/libcomps/pull/50

This project used the following method descriptors (for module functions):

{"categories_match", (PyCFunction)PyCOMPS_categories_match, METH_KEYWORDS,
 PyCOMPS_validate__doc__},
{"environments_match", (PyCFunction)PyCOMPS_envs_match, METH_KEYWORDS,
 PyCOMPS_validate__doc__},

In Python 3.7, importing the module was just fine: descriptor flags are only checked at the  first call to the method.

In Python 3.8, descriptor flags are checked when the module is imported.

Fedora bug: https://bugzilla.redhat.com/show_bug.cgi?id=1734777

The fix is to use the right flags: "METH_VARARGS | METH_KEYWORDS" instead of "METH_KEYWORDS".

Should we add a note like "if you get a 'SystemError: bad call flags' on import, check the descriptor flags of your functions" in What's New in Python 3.8? Maybe with a link to this issue.
https://docs.python.org/dev/whatsnew/3.8.html#changes-in-the-c-api
msg349046 - (view) Author: Jeroen Demeyer (jdemeyer) * (Python triager) Date: 2019-08-05 10:21
> Should we add a note like "if you get a 'SystemError: bad call flags' on import, check the descriptor flags of your functions" in What's New in Python 3.8?

A better solution would be to change the error message. We could change it to something like SystemError("bad flags 0x2 for PyCOMPS_categories_match")

But maybe it's not worth it. Are there many projects that define functions/methods but never call them?
msg349469 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-08-12 13:17
'SystemError: bad call flags' error message LGTM, I don't think that it should be changed.

> Are there many projects that define functions/methods but never call them?

I have no idea.
msg356453 - (view) Author: miss-islington (miss-islington) Date: 2019-11-12 13:08
New changeset 9a13a388f202268dd7b771638adbec132449b98b by Miss Islington (bot) (Jeroen Demeyer) in branch 'master':
bpo-36974: expand call protocol documentation (GH-13844)
https://github.com/python/cpython/commit/9a13a388f202268dd7b771638adbec132449b98b
History
Date User Action Args
2019-11-12 13:08:04miss-islingtonsetnosy: + miss-islington
messages: + msg356453
2019-08-12 13:17:28vstinnersetmessages: + msg349469
2019-08-05 10:21:36jdemeyersetmessages: + msg349046
2019-07-31 13:29:07vstinnersetnosy: + vstinner
messages: + msg348804
2019-07-23 10:39:55lukasz.langasetnosy: + lukasz.langa
messages: + msg348327
2019-07-15 12:46:24jdemeyersetpull_requests: + pull_request14578
2019-07-05 14:03:23petr.viktorinsetstatus: open -> closed
resolution: fixed
messages: + msg347347

stage: patch review -> resolved
2019-07-05 12:52:13jdemeyersetmessages: + msg347340
2019-07-05 12:48:47petr.viktorinsetmessages: + msg347338
2019-06-25 08:19:39petr.viktorinsetmessages: + msg346492
2019-06-24 14:40:39miss-islingtonsetpull_requests: + pull_request14163
2019-06-24 10:41:10petr.viktorinsetmessages: + msg346366
2019-06-06 09:05:13jdemeyersetpull_requests: + pull_request13733
2019-06-05 15:36:16jdemeyersetpull_requests: + pull_request13721
2019-06-03 15:39:28jdemeyersetpull_requests: + pull_request13666
2019-06-02 23:57:25petr.viktorinsetmessages: + msg344340
2019-06-02 23:43:16petr.viktorinsetmessages: + msg344339
2019-06-02 23:08:17petr.viktorinsetmessages: + msg344333
2019-06-02 22:39:01petr.viktorinsetpull_requests: + pull_request13640
2019-06-02 21:52:22petr.viktorinsetmessages: + msg344328
2019-05-31 14:20:40petr.viktorinsetpull_requests: + pull_request13584
2019-05-31 14:08:06petr.viktorinsetmessages: + msg344077
2019-05-31 02:13:54benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg344016
2019-05-30 15:15:18jdemeyersetpull_requests: + pull_request13569
2019-05-30 13:11:43petr.viktorinsetmessages: + msg343975
2019-05-30 10:44:02petr.viktorinsetmessages: + msg343968
2019-05-30 10:43:33petr.viktorinsetmessages: + msg343967
2019-05-29 22:17:50petr.viktorinsetmessages: + msg343935
2019-05-29 22:16:52pablogsalsetmessages: + msg343934
2019-05-29 22:12:54petr.viktorinsetmessages: + msg343932
2019-05-29 21:56:23pablogsalsetmessages: + msg343930
2019-05-29 21:52:59pablogsalsetnosy: + pablogsal
messages: + msg343929
2019-05-29 20:45:44petr.viktorinsetmessages: + msg343922
2019-05-29 19:27:38petr.viktorinsetpull_requests: + pull_request13555
2019-05-29 18:31:58petr.viktorinsetmessages: + msg343908
2019-05-29 13:39:18jdemeyersetpull_requests: + pull_request13547
2019-05-22 14:58:28jdemeyersetpull_requests: + pull_request13414
2019-05-22 05:41:45jdemeyersetpull_requests: + pull_request13397
2019-05-21 10:44:05jdemeyersetpull_requests: + pull_request13370
2019-05-21 09:49:00jdemeyersetpull_requests: + pull_request13367
2019-05-20 18:02:45jdemeyersetkeywords: + patch
stage: patch review
pull_requests: + pull_request13359
2019-05-20 18:01:27jdemeyercreate