msg240409 - (view) |
Author: Zachary Ware (zach.ware) *  |
Date: 2015-04-10 03:49 |
The attached patch adds a script that can scrape header files to pick out PyAPI_DATA and PyAPI_FUNCs to generate PC/python3.def. It's not complete and is not integrated into the build at all yet. I'm planning to work on this further at the PyCon sprints.
Running this and comparing output to the current checked-in python3.def shows that we're not keeping it up-to-date well at all.
|
msg240754 - (view) |
Author: Zachary Ware (zach.ware) *  |
Date: 2015-04-13 21:11 |
Same patch, with diff to python3.def to make it available.
|
msg241262 - (view) |
Author: Martin v. Löwis (loewis) *  |
Date: 2015-04-16 19:10 |
I think the approach taken in this script is incorrect. It leads to false modifications of the stable ABI, making it in fact unstable. Four examples:
PyAST_FromNode does not belong to the stable ABI, because "struct _node" doesn't belong to the stable ABI (see PEP 384 for a list of structures that belong to the stable ABI). In fact, ast.h isn't even a header file that extension modules should regularly include.
All deletions are (e.g. PyCFunction_New) are certainly incorrect: functions must never be removed from the stable ABI. In fact, Objects/methodobject.c preserves PyCFunction_New just for the sake of the stable ABI (despite there being no longer a declaration in the header file).
PyErr_SetFromWindowsErrWithFilename could belong to the stable ABI, or not, depending on the intention of stability in its addition. Since it was there since 3.0, and had not been in the stable ABI back then, I'd claim that it doesn't belong to the stable ABI (apparently, nobody has missed it). OTOH
PyExc_BrokenPipeError might be a useful addition to the stable ABI. However, it was not there in 3.0, so the header files ought to version-guard its declaration, but currently don't.
So I prefer if the def file keeps being maintained manually; I don't mind if there is a tool that checks for inconsistencies, leaving their resolution to a maintainer.
|
msg241281 - (view) |
Author: Steve Dower (steve.dower) *  |
Date: 2015-04-16 22:00 |
That would be the preferred way to do it, but we got no traction from anyone at PyCon or on python-dev when we tried to get people to look at what's changed.
I suspect we haven't marketed the stable API well enough to have a lot of users, and the build tools still mark everything as 3.x-specific rather than 3-specific. Keeping it as a bare minimum interface has its value, but it's not unreasonable to expect that the functions available from the headers under Py_LIMITED_API are actually exported by python3.dll.
|
msg243994 - (view) |
Author: Zachary Ware (zach.ware) *  |
Date: 2015-05-24 17:54 |
What about including this in Tools/scripts/patchcheck.py? I think that might strike a good enough balance between making changes noticeable and not automatically changing the def file.
As far as all the differences this has uncovered, I'll try to make a patch to #ifdef them in/out.
|
msg243997 - (view) |
Author: Steve Dower (steve.dower) *  |
Date: 2015-05-24 18:31 |
Could the latter break people already using the stable ABI on other platforms? Does it even work for others? I don't know what the equivalent of python3.dll would be.
If it doesn't exist outside of Windows, then assuming that everything not in python3.def isn't stable is fine. Otherwise I think we need to go the other way.
|
msg244309 - (view) |
Author: Steve Dower (steve.dower) *  |
Date: 2015-05-28 15:47 |
It might be nice to have a test that tries to import all of the stable ABI at build time so that the buildbots break if someone adds a new API and doesn't think about what they've done.
As an aside, it'd be nice for the script to take the version numbers as args so we can integrate it into the MSBuild steps (which already know the version) and avoid having to update this each time we update the Python version.
|
msg264614 - (view) |
Author: Zachary Ware (zach.ware) *  |
Date: 2016-05-02 02:42 |
Unassigning since I haven't made it anywhere with this in a year; if anybody else would like to pick it up, feel free. Otherwise I'll get to it some day :)
|
msg264619 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) *  |
Date: 2016-05-02 05:14 |
This tool would be very useful, since new API often accidentally marked as stable API in header files. The script can play just deliberative role, warn if new unstable API is not wrapped with "#ifndef Py_LIMITED_API" or if stable API is not documented. May be we need separate list of public API in more regular format for comparison with header, documentation, python3.def, refcounts.dat, etc.
There is a code for parsing preprocessor instructions in Argument Clinic. May be it can be reused here.
|
msg264624 - (view) |
Author: Steve Dower (steve.dower) *  |
Date: 2016-05-02 06:09 |
As I recall, the tool is ready to go. The problem is the argument over what to do with all the APIs that currently are incorrect. Zach had a list at PyCon last year, and the consensus seemed to be that "someone" should figure out whether each API should be stable or not. Since Zach understandably doesn't want to be that someone, there's really no way to reliably tool this process when there are a number of existing violations.
(FWIW, I believe we should just add them to the stable list, as that's the only way to guarantee source compatibility. They've already shipped in 3.5 and people can use them on non-Windows OSs.)
|
msg264626 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) *  |
Date: 2016-05-02 06:22 |
See also closely related issue26900.
|
msg264652 - (view) |
Author: Zachary Ware (zach.ware) *  |
Date: 2016-05-02 15:39 |
> There is a code for parsing preprocessor instructions in Argument Clinic.
> May be it can be reused here.
It already is, actually; its availability was a major factor in my thinking this would be possible and starting to work on it.
|
msg283715 - (view) |
Author: Steve Dower (steve.dower) *  |
Date: 2016-12-21 01:41 |
I just ran into this because PyModule_AddFunctions() was added to the stable ABI in 3.5, but was not added to python3.dll.
This is blatantly a compilation error. We _need_ python3.def and the stable ABI to be in sync.
I've rerun the script against Python 3.6 and attached it. I'm also going to post to python-dev to attract some attention.
I think at this stage we ought to update python3.dll to include everything that could be used from the headers. Unfortunately, that means extensions could potentially break if built on 3.6.1 and run against 3.6.0, but it's inevitable that there will be some breakage and this way it's only temporary.
|
msg283770 - (view) |
Author: Steve Dower (steve.dower) *  |
Date: 2016-12-21 17:18 |
Guess I should at least pose the question - Ned, would you consider the change in my patch for 3.6.0? Only the one file is affected, and it only contributes to one DLL that is (a) recommended, (b) not widely used and (c) very incompatible with the included headers.
|
msg283784 - (view) |
Author: Ned Deily (ned.deily) *  |
Date: 2016-12-21 20:52 |
I don't know enough about how python3.dll is used in the Windows world to ask the right questions. One obvious one: would this change likely affect any third-party binaries already built for 3.6.0? On the other hand, if we have to have one, it would be better to have a compatibility break at 3.6.0 rather than at 3.6.1. I really don't want to take any more changes at this point for 3.6.0 but if you are convinced it is the right thing to do and won't break anything, get it in for 3.6.1 now and I'll reluctantly cherrypick it for 3.6.0.
|
msg283791 - (view) |
Author: Steve Dower (steve.dower) *  |
Date: 2016-12-21 22:16 |
> would this change likely affect any third-party binaries already built for 3.6.0? On the other hand, if we have to have one, it would be better to have a compatibility break at 3.6.0 rather than at 3.6.1.
Assuming we only make additive changes, then nothing that currently exists will break. However, extensions may then be built using 3.6.1 that will fail on earlier versions. This goes against the entire purpose of the limited API - making the change for 3.6.0 at least means they will work for the entire 3.6 series.
This is one of those "all options are bad" scenarios. Making the change today will prevent us from fixing the functions that should not have leaked into the limited API, but then again they've already leaked and people can compile against them.
Perhaps the best approach is to revise all functions in the limited API, restrict back to those that were in 3.3 and carefully add new ones from there. Then we just apologise for 3.5.[0-2] and 3.6.0 potentially producing invalid binaries and recommend getting a newer version to build with.
I've convinced myself that 3.6.0 is too soon to make the right fix here, so it'll probably always be a bad version in this respect. Hopefully by 3.6.1 we can fix up the limited API and make it reliable again :(
|
msg283872 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) *  |
Date: 2016-12-23 06:39 |
PyCmpWrapper_Type is not defined.
Some underscored names are defined only in special builds (with Py_REF_DEBUG, Py_TRACE_REFS).
PyAST_* and PyNode_* functions are not documented, ast.h and node.h are private headers.
Names like _PyArg_Parse_SizeT shouldn't be removed.
|
msg283891 - (view) |
Author: Steve Dower (steve.dower) *  |
Date: 2016-12-23 18:17 |
> ast.h and node.h are private headers.
You're right, these should be excluded (possibly from the release too?)
I've been playing with the script in a separate context and I think I've hit a few issues (though I have made some modifications, so YMMV). It's probably just as well we waited. I'll thoroughly validate it before coming back with a new patch.
|
msg283901 - (view) |
Author: Roundup Robot (python-dev)  |
Date: 2016-12-23 20:14 |
New changeset 9cb87e53e324 by Serhiy Storchaka in branch '3.5':
Sort and remove duplicates from PC/python3.def (issue #23903).
https://hg.python.org/cpython/rev/9cb87e53e324
New changeset 0927b5c80c50 by Serhiy Storchaka in branch '3.6':
Sort and remove duplicates from PC/python3.def (issue #23903).
https://hg.python.org/cpython/rev/0927b5c80c50
New changeset e64a82371d72 by Serhiy Storchaka in branch 'default':
Sort and remove duplicates from PC/python3.def (issue #23903).
https://hg.python.org/cpython/rev/e64a82371d72
|
msg283902 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) *  |
Date: 2016-12-23 20:28 |
Reordered to make diffs easier for reviewing.
|
msg283907 - (view) |
Author: Steve Dower (steve.dower) *  |
Date: 2016-12-23 23:29 |
Good call.
I think I might actually replace the script with a build step that uses the C preprocessor to get all the names, then a script to generate the file. If someone who knows the POSIX build system can help out, we could generate a list on each build for various platforms and fail when it doesn't match what's committed.
Since the preprocessor doesn't have to generate valid code, we should be able to cross-compile for other platforms, though that's less critical.
|
msg283911 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) *  |
Date: 2016-12-24 00:12 |
See also issue29058.
|
msg283972 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) *  |
Date: 2016-12-24 20:49 |
Here is a shell script that uses the C preprocessor and other Unix tools to generate PC/python3.def. It doesn't add PyAST_* and PyNode_* functions and doesn't remove PyArg_VaParse* functions. But it still removes some names that are no longer used in public interface, but should be kept in the stable ABI for compatibility. E.g. PyCFunction_New (now it is a macro expanded to PyCFunction_NewEx) or _PyTrash_deposit_object (invoked by the macros before 3.2.4). It seems to me that this is unavoidable, and the resulting file must be manually edited.
|
msg283973 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) *  |
Date: 2016-12-24 21:24 |
Following patch against 3.6 just adds new names to PC/python3.def. The only removed names are PyExc_MemoryErrorInst and PyUnicode_SetDefaultEncoding. They don't exist in 3.x.
|
msg284039 - (view) |
Author: Steve Dower (steve.dower) *  |
Date: 2016-12-26 19:49 |
Those macros like PyCFunction_New actually need to become functions again, since we need real exports to forward them to the right place. Converting stable API functions to macros breaks the ABI, since modules compiled against the old one can no longer link to newer versions, and modules compiled against newer ones cannot link to older versions.
I'll try and figure out a way we can compile those functions directly into python3.dll, since that's going to leave us with the best ability to use macros for non-stable API builds. We will, however, need to update the header files to define actual functions rather than macros when Py_LIMITED_API is set.
|
msg284043 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) *  |
Date: 2016-12-26 20:16 |
They do. Functions PyCFunction_New, _PyTrash_deposit_object etc are compiled and exported even if they don't represented in headers.
Actually we can remove the PyCFunction_New function since it was converted to to a macro before Python 3.0 (faabd9867fb8). No one Python 3 extension uses it. But this is separate issue. _PyTrash_deposit_object was used in 3.2.3 and should be kept for binary compatibility.
|
msg284047 - (view) |
Author: Steve Dower (steve.dower) *  |
Date: 2016-12-26 21:38 |
Attached a patch that integrates generation into the build of python3.dll. There are also a couple of fixes for headers to retain functions/data that were previously exported.
The patch includes an updated python3.def file, though I expect that will go away and we'll check in the python3.txt file that is generated on build (which includes the return type). That way we can fail builds that modify the file (regardless of platform) and require an explicit step when adding to the stable API.
There are still a few functions towards the end of python3.def that I need to figure out how to deal with. (Also, my sort script doesn't quite match the result from however Serhiy sorted it last, but it's as close as I could get.)
|
msg284077 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) *  |
Date: 2016-12-27 09:36 |
It is not good that the script removes existing names and adds non-existing names. It is safe to remove names that don't exist in all releases starting from 3.2.0.
1. PyCFunction_New shouldn't be removed in maintained releases. This name is exported, but is unlikely used by third-party code. It can be removed in separate issue, and only in 3.7.
2. PyCmpWrapper_Type shouldn't be added. It is not defined in 3.x.
3. _PyTrash_destroy_chain and like should not be removed for compatibility with 3.2.0-3.2.3.
4. _PyBytes_DecodeEscape shouldn't be added. It is private.
5. Exporting Py_hexdigits looks doubtful. This constant was added for internal use (like _Py_SwappedOp). Unlikely it is used in third-party code. Not attributing it as private looks an error.
6. _Py_TrueStruct shouldn't be removed. It is used in the Py_True macro.
7. *_SizeT names shouldn't be removed. They are used when PY_SSIZE_T_CLEAN is defined before including Python.h (recommended).
8. I think that rather than adding PyODict_Type to limited API, PyODict_Check and PyODict_CheckExact should be excluded from limited API (see issue29058). They never worked in limited API. Note that PyDict_Type as well as other types are not included in limited API. We can consider exposing PyODict_Check and PyODict_CheckExact as functions in limited API in 3.7, but this is other issue.
As for PyArg_VaParse and PyArg_VaParseTupleAndKeywords see issue11626. Seems they were excluded from limited API by mistake.
I suggest first fix errors in python3.def, remove non-existing names, add new names, and only after this add the ability of generating it if it doesn't break python3.def.
|
msg284079 - (view) |
Author: Roundup Robot (python-dev)  |
Date: 2016-12-27 10:01 |
New changeset 8423f86486b3 by Serhiy Storchaka in branch '3.5':
Issue #23903: Fixed errors and remove non-existing names in python3.def.
https://hg.python.org/cpython/rev/8423f86486b3
New changeset b5470d08969c by Serhiy Storchaka in branch '3.6':
Issue #23903: Fixed errors and remove non-existing names in python3.def.
https://hg.python.org/cpython/rev/b5470d08969c
New changeset 45507a5751d8 by Serhiy Storchaka in branch 'default':
Issue #23903: Fixed errors and remove non-existing names in python3.def.
https://hg.python.org/cpython/rev/45507a5751d8
|
msg284083 - (view) |
Author: Roundup Robot (python-dev)  |
Date: 2016-12-27 10:18 |
New changeset d95fee442e27 by Serhiy Storchaka in branch '3.5':
Issue #23903: Added missed names to PC/python3.def.
https://hg.python.org/cpython/rev/d95fee442e27
New changeset cb864fc4b3be by Serhiy Storchaka in branch '3.6':
Issue #23903: Added missed names to PC/python3.def.
https://hg.python.org/cpython/rev/cb864fc4b3be
New changeset 513852ad0c5c by Serhiy Storchaka in branch 'default':
Issue #23903: Added missed names to PC/python3.def.
https://hg.python.org/cpython/rev/513852ad0c5c
|
msg284105 - (view) |
Author: Steve Dower (steve.dower) *  |
Date: 2016-12-27 14:53 |
Unfortunately, there's no way too remove defined names from the DLL during Python 3 at all, except where the prototype was never provided. PyCFunction_New has always been a macro, and PySys_SetDefaultEncoding looks to have been removed before the API was committed.
Maintaining an export and a macro is going to be awkward, so I think we should just change them back to functions. Any half-decent optimizing compiler is going to inline it within core, and it's better than having two implementations in different places of the same function.
But yes I agree, fix the def first and then automate. But I don't want to fix it but adding things that were never usable.
|
msg284108 - (view) |
Author: Roundup Robot (python-dev)  |
Date: 2016-12-27 15:55 |
New changeset 2e5ad97c9c19 by Serhiy Storchaka in branch '3.5':
Issue #23903: Added missed Windows-specific names to PC/python3.def.
https://hg.python.org/cpython/rev/2e5ad97c9c19
New changeset 86a412584c02 by Serhiy Storchaka in branch '3.6':
Issue #23903: Added missed Windows-specific names to PC/python3.def.
https://hg.python.org/cpython/rev/86a412584c02
New changeset 3a2595f82447 by Serhiy Storchaka in branch 'default':
Issue #23903: Added missed Windows-specific names to PC/python3.def.
https://hg.python.org/cpython/rev/3a2595f82447
|
msg355732 - (view) |
Author: Steve Dower (steve.dower) *  |
Date: 2019-10-30 20:17 |
Reawakening this issue so we can finish it off - the stable ABI has been declared "clean" on python-dev, so there shouldn't be any more concerns about APIs leaking in now. We can assume that everything accessible from the header files should be included in python3.def.
If it has been cleaned up though, we might need to go through and restore anything that used to be in python3.def but wasn't meant to be. Those are stable.
|
msg387118 - (view) |
Author: Petr Viktorin (petr.viktorin) *  |
Date: 2021-02-16 15:43 |
I plan to do this slightly differently, see PEP 652.
|
msg393078 - (view) |
Author: Petr Viktorin (petr.viktorin) *  |
Date: 2021-05-06 09:51 |
The symbols exported by python3.dll are now generated from Misc/stable_abi.txt as per PEP 652.
See the devguide for more details: https://devguide.python.org/c-api/
This is slightly more work than generating the list fully automatically, but the extra work is negligible compared to all the things you should think about when adding API that'll be be supported until Python 4.0.
Headers are now checked on Unix (GCC) only. If anyone wants to work on a cross-platform C parser/checker, please let me know.
|
msg394859 - (view) |
Author: Petr Viktorin (petr.viktorin) *  |
Date: 2021-06-01 14:44 |
PEP 652 (bpo-43795) is now in; I'm closing this issue.
A cross-platform C parser/checker would still be an improvement, but the constraints (no external dependencies in CPython) and benefits (not many compared to the goal of strict separation of internal/public/stable headers) aren't really aligned in its favor.
|
|
Date |
User |
Action |
Args |
2022-04-11 14:58:15 | admin | set | github: 68091 |
2021-06-01 14:44:50 | petr.viktorin | set | status: open -> closed resolution: rejected messages:
+ msg394859
stage: patch review -> resolved |
2021-05-06 09:51:24 | petr.viktorin | set | messages:
+ msg393078 |
2021-02-16 15:43:36 | petr.viktorin | set | nosy:
+ petr.viktorin messages:
+ msg387118
|
2020-06-25 10:44:41 | ned.deily | set | nosy:
- ned.deily
versions:
- Python 3.7 |
2020-06-25 09:36:09 | vstinner | set | nosy:
+ paul.moore components:
+ C API
|
2019-10-30 20:17:36 | steve.dower | set | messages:
+ msg355732 versions:
+ Python 3.8, Python 3.9, - Python 3.5, Python 3.6 |
2019-01-23 18:13:59 | nascheme | set | nosy:
+ nascheme
|
2017-04-01 05:49:38 | serhiy.storchaka | set | pull_requests:
- pull_request1091 |
2017-03-31 16:36:37 | dstufft | set | pull_requests:
+ pull_request1091 |
2016-12-27 15:55:15 | python-dev | set | messages:
+ msg284108 |
2016-12-27 14:53:45 | steve.dower | set | messages:
+ msg284105 |
2016-12-27 13:42:14 | serhiy.storchaka | set | dependencies:
+ Readd PyArg_VaParse to the stable API |
2016-12-27 10:18:06 | python-dev | set | messages:
+ msg284083 |
2016-12-27 10:01:12 | python-dev | set | messages:
+ msg284079 |
2016-12-27 09:36:41 | serhiy.storchaka | set | messages:
+ msg284077 |
2016-12-26 21:38:37 | steve.dower | set | files:
+ 23903_1.patch
messages:
+ msg284047 |
2016-12-26 20:16:37 | serhiy.storchaka | set | messages:
+ msg284043 |
2016-12-26 19:49:49 | steve.dower | set | messages:
+ msg284039 |
2016-12-24 21:24:13 | serhiy.storchaka | set | files:
+ python3def-3.6.patch
messages:
+ msg283973 |
2016-12-24 20:49:59 | serhiy.storchaka | set | files:
+ genpython3def.sh
dependencies:
+ Mark new limited C API messages:
+ msg283972 |
2016-12-24 00:12:50 | serhiy.storchaka | set | messages:
+ msg283911 |
2016-12-23 23:29:46 | steve.dower | set | messages:
+ msg283907 |
2016-12-23 20:28:51 | serhiy.storchaka | set | files:
+ python3def.patch
messages:
+ msg283902 |
2016-12-23 20:14:34 | python-dev | set | nosy:
+ python-dev messages:
+ msg283901
|
2016-12-23 18:17:13 | steve.dower | set | messages:
+ msg283891 |
2016-12-23 06:39:33 | serhiy.storchaka | set | messages:
+ msg283872 |
2016-12-21 22:16:22 | steve.dower | set | messages:
+ msg283791 |
2016-12-21 20:52:43 | ned.deily | set | messages:
+ msg283784 |
2016-12-21 17:18:07 | steve.dower | set | nosy:
+ ned.deily messages:
+ msg283770
|
2016-12-21 01:42:00 | steve.dower | set | files:
+ python3def.patch
messages:
+ msg283715 versions:
+ Python 3.5, Python 3.7 |
2016-05-02 15:39:46 | zach.ware | set | messages:
+ msg264652 |
2016-05-02 06:22:17 | serhiy.storchaka | set | messages:
+ msg264626 |
2016-05-02 06:09:12 | steve.dower | set | messages:
+ msg264624 |
2016-05-02 05:14:00 | serhiy.storchaka | set | messages:
+ msg264619 |
2016-05-02 02:42:44 | zach.ware | set | assignee: zach.ware -> messages:
+ msg264614 versions:
+ Python 3.6, - Python 3.5 |
2015-05-28 15:47:36 | steve.dower | set | messages:
+ msg244309 |
2015-05-24 18:31:34 | steve.dower | set | messages:
+ msg243997 |
2015-05-24 17:54:48 | zach.ware | set | messages:
+ msg243994 |
2015-04-16 22:00:54 | steve.dower | set | messages:
+ msg241281 |
2015-04-16 19:10:36 | loewis | set | nosy:
+ loewis messages:
+ msg241262
|
2015-04-16 00:12:54 | serhiy.storchaka | set | nosy:
+ serhiy.storchaka
stage: needs patch -> patch review |
2015-04-13 21:11:10 | zach.ware | set | files:
+ 23903_with_python3.def.diff
messages:
+ msg240754 |
2015-04-10 03:49:21 | zach.ware | create | |