classification
Title: Generate PC/python3.def by scraping headers
Type: enhancement Stage: patch review
Components: Build, Windows Versions: Python 3.7, Python 3.6, Python 3.5
process
Status: open Resolution:
Dependencies: 29058 29083 Superseder:
Assigned To: Nosy List: loewis, nascheme, ned.deily, python-dev, serhiy.storchaka, steve.dower, tim.golden, zach.ware
Priority: normal Keywords: patch

Created on 2015-04-10 03:49 by zach.ware, last changed 2019-01-23 18:13 by nascheme.

Files
File name Uploaded Description Edit
add_python3defgen.diff zach.ware, 2015-04-10 03:49 review
23903_with_python3.def.diff zach.ware, 2015-04-13 21:11 review
python3def.patch steve.dower, 2016-12-21 01:41
python3def.patch serhiy.storchaka, 2016-12-23 20:28 Regenerated for review review
genpython3def.sh serhiy.storchaka, 2016-12-24 20:49
python3def-3.6.patch serhiy.storchaka, 2016-12-24 21:24 review
23903_1.patch steve.dower, 2016-12-26 21:38
Messages (32)
msg240409 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2015-04-10 03:49
The attached patch adds a script that can scrape header files to pick out PyAPI_DATA and PyAPI_FUNCs to generate PC/python3.def.  It's not complete and is not integrated into the build at all yet.  I'm planning to work on this further at the PyCon sprints.

Running this and comparing output to the current checked-in python3.def shows that we're not keeping it up-to-date well at all.
msg240754 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2015-04-13 21:11
Same patch, with diff to python3.def to make it available.
msg241262 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2015-04-16 19:10
I think the approach taken in this script is incorrect. It leads to false modifications of the stable ABI, making it in fact unstable. Four examples:

PyAST_FromNode does not belong to the stable ABI, because "struct _node" doesn't belong to the stable ABI (see PEP 384 for a list of structures that belong to the stable ABI). In fact, ast.h isn't even a header file that extension modules should regularly include.

All deletions are (e.g. PyCFunction_New) are certainly incorrect: functions must never be removed from the stable ABI. In fact, Objects/methodobject.c preserves PyCFunction_New just for the sake of the stable ABI (despite there being no longer a declaration in the header file).

PyErr_SetFromWindowsErrWithFilename could belong to the stable ABI, or not, depending on the intention of stability in its addition. Since it was there since 3.0, and had not been in the stable ABI back then, I'd claim that it doesn't belong to the stable ABI (apparently, nobody has missed it). OTOH

PyExc_BrokenPipeError might be a useful addition to the stable ABI. However, it was not there in 3.0, so the header files ought to version-guard its declaration, but currently don't.

So I prefer if the def file keeps being maintained manually; I don't mind if there is a tool that checks for inconsistencies, leaving their resolution to a maintainer.
msg241281 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-04-16 22:00
That would be the preferred way to do it, but we got no traction from anyone at PyCon or on python-dev when we tried to get people to look at what's changed.

I suspect we haven't marketed the stable API well enough to have a lot of users, and the build tools still mark everything as 3.x-specific rather than 3-specific. Keeping it as a bare minimum interface has its value, but it's not unreasonable to expect that the functions available from the headers under Py_LIMITED_API are actually exported by python3.dll.
msg243994 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2015-05-24 17:54
What about including this in Tools/scripts/patchcheck.py?  I think that might strike a good enough balance between making changes noticeable and not automatically changing the def file.

As far as all the differences this has uncovered, I'll try to make a patch to #ifdef them in/out.
msg243997 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-05-24 18:31
Could the latter break people already using the stable ABI on other platforms? Does it even work for others? I don't know what the equivalent of python3.dll would be.

If it doesn't exist outside of Windows, then assuming that everything not in python3.def isn't stable is fine. Otherwise I think we need to go the other way.
msg244309 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-05-28 15:47
It might be nice to have a test that tries to import all of the stable ABI at build time so that the buildbots break if someone adds a new API and doesn't think about what they've done.

As an aside, it'd be nice for the script to take the version numbers as args so we can integrate it into the MSBuild steps (which already know the version) and avoid having to update this each time we update the Python version.
msg264614 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2016-05-02 02:42
Unassigning since I haven't made it anywhere with this in a year; if anybody else would like to pick it up, feel free.  Otherwise I'll get to it some day :)
msg264619 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-05-02 05:14
This tool would be very useful, since new API often accidentally marked as stable API in header files. The script can play just deliberative role, warn if new unstable API is not wrapped with "#ifndef Py_LIMITED_API" or if stable API is not documented. May be we need separate list of public API in more regular format for comparison with header, documentation, python3.def, refcounts.dat, etc.

There is a code for parsing preprocessor instructions in Argument Clinic. May be it can be reused here.
msg264624 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-05-02 06:09
As I recall, the tool is ready to go. The problem is the argument over what to do with all the APIs that currently are incorrect. Zach had a list at PyCon last year, and the consensus seemed to be that "someone" should figure out whether each API should be stable or not. Since Zach understandably doesn't want to be that someone, there's really no way to reliably tool this process when there are a number of existing violations.

(FWIW, I believe we should just add them to the stable list, as that's the only way to guarantee source compatibility. They've already shipped in 3.5 and people can use them on non-Windows OSs.)
msg264626 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-05-02 06:22
See also closely related issue26900.
msg264652 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2016-05-02 15:39
> There is a code for parsing preprocessor instructions in Argument Clinic.
> May be it can be reused here.

It already is, actually; its availability was a major factor in my thinking this would be possible and starting to work on it.
msg283715 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-12-21 01:41
I just ran into this because PyModule_AddFunctions() was added to the stable ABI in 3.5, but was not added to python3.dll.

This is blatantly a compilation error. We _need_ python3.def and the stable ABI to be in sync.

I've rerun the script against Python 3.6 and attached it. I'm also going to post to python-dev to attract some attention.

I think at this stage we ought to update python3.dll to include everything that could be used from the headers. Unfortunately, that means extensions could potentially break if built on 3.6.1 and run against 3.6.0, but it's inevitable that there will be some breakage and this way it's only temporary.
msg283770 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-12-21 17:18
Guess I should at least pose the question - Ned, would you consider the change in my patch for 3.6.0? Only the one file is affected, and it only contributes to one DLL that is (a) recommended, (b) not widely used and (c) very incompatible with the included headers.
msg283784 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-12-21 20:52
I don't know enough about how python3.dll is used in the Windows world to ask the right questions.  One obvious one: would this change likely affect any third-party binaries already built for 3.6.0?  On the other hand, if we have to have one, it would be better to have a compatibility break at 3.6.0 rather than at 3.6.1.  I really don't want to take any more changes at this point for 3.6.0 but if you are convinced it is the right thing to do and won't break anything, get it in for 3.6.1 now and I'll reluctantly cherrypick it for 3.6.0.
msg283791 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-12-21 22:16
> would this change likely affect any third-party binaries already built for 3.6.0?  On the other hand, if we have to have one, it would be better to have a compatibility break at 3.6.0 rather than at 3.6.1.

Assuming we only make additive changes, then nothing that currently exists will break. However, extensions may then be built using 3.6.1 that will fail on earlier versions. This goes against the entire purpose of the limited API - making the change for 3.6.0 at least means they will work for the entire 3.6 series.

This is one of those "all options are bad" scenarios. Making the change today will prevent us from fixing the functions that should not have leaked into the limited API, but then again they've already leaked and people can compile against them.

Perhaps the best approach is to revise all functions in the limited API, restrict back to those that were in 3.3 and carefully add new ones from there. Then we just apologise for 3.5.[0-2] and 3.6.0 potentially producing invalid binaries and recommend getting a newer version to build with.

I've convinced myself that 3.6.0 is too soon to make the right fix here, so it'll probably always be a bad version in this respect. Hopefully by 3.6.1 we can fix up the limited API and make it reliable again :(
msg283872 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-12-23 06:39
PyCmpWrapper_Type is not defined.

Some underscored names are defined only in special builds (with Py_REF_DEBUG, Py_TRACE_REFS).

PyAST_* and PyNode_* functions are not documented, ast.h and node.h are private headers.

Names like _PyArg_Parse_SizeT shouldn't be removed.
msg283891 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-12-23 18:17
> ast.h and node.h are private headers.

You're right, these should be excluded (possibly from the release too?)

I've been playing with the script in a separate context and I think I've hit a few issues (though I have made some modifications, so YMMV). It's probably just as well we waited. I'll thoroughly validate it before coming back with a new patch.
msg283901 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-12-23 20:14
New changeset 9cb87e53e324 by Serhiy Storchaka in branch '3.5':
Sort and remove duplicates from PC/python3.def (issue #23903).
https://hg.python.org/cpython/rev/9cb87e53e324

New changeset 0927b5c80c50 by Serhiy Storchaka in branch '3.6':
Sort and remove duplicates from PC/python3.def (issue #23903).
https://hg.python.org/cpython/rev/0927b5c80c50

New changeset e64a82371d72 by Serhiy Storchaka in branch 'default':
Sort and remove duplicates from PC/python3.def (issue #23903).
https://hg.python.org/cpython/rev/e64a82371d72
msg283902 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-12-23 20:28
Reordered to make diffs easier for reviewing.
msg283907 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-12-23 23:29
Good call.

I think I might actually replace the script with a build step that uses the C preprocessor to get all the names, then a script to generate the file. If someone who knows the POSIX build system can help out, we could generate a list on each build for various platforms and fail when it doesn't match what's committed.

Since the preprocessor doesn't have to generate valid code, we should be able to cross-compile for other platforms, though that's less critical.
msg283911 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-12-24 00:12
See also issue29058.
msg283972 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-12-24 20:49
Here is a shell script that uses the C preprocessor and other Unix tools to generate PC/python3.def. It doesn't add PyAST_* and PyNode_* functions and doesn't remove PyArg_VaParse* functions. But it still removes some names that are no longer used in public interface, but should be kept in the stable ABI for compatibility. E.g. PyCFunction_New (now it is a macro expanded to PyCFunction_NewEx) or _PyTrash_deposit_object (invoked by the macros before 3.2.4). It seems to me that this is unavoidable, and the resulting file must be manually edited.
msg283973 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-12-24 21:24
Following patch against 3.6 just adds new names to PC/python3.def. The only removed names are PyExc_MemoryErrorInst and PyUnicode_SetDefaultEncoding. They don't exist in 3.x.
msg284039 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-12-26 19:49
Those macros like PyCFunction_New actually need to become functions again, since we need real exports to forward them to the right place. Converting stable API functions to macros breaks the ABI, since modules compiled against the old one can no longer link to newer versions, and modules compiled against newer ones cannot link to older versions.

I'll try and figure out a way we can compile those functions directly into python3.dll, since that's going to leave us with the best ability to use macros for non-stable API builds. We will, however, need to update the header files to define actual functions rather than macros when Py_LIMITED_API is set.
msg284043 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-12-26 20:16
They do. Functions PyCFunction_New, _PyTrash_deposit_object etc are compiled and exported even if they don't represented in headers.

Actually we can remove the PyCFunction_New function since it was converted to to a macro before Python 3.0 (faabd9867fb8). No one Python 3 extension uses it. But this is separate issue. _PyTrash_deposit_object was used in 3.2.3 and should be kept for binary compatibility.
msg284047 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-12-26 21:38
Attached a patch that integrates generation into the build of python3.dll. There are also a couple of fixes for headers to retain functions/data that were previously exported.

The patch includes an updated python3.def file, though I expect that will go away and we'll check in the python3.txt file that is generated on build (which includes the return type). That way we can fail builds that modify the file (regardless of platform) and require an explicit step when adding to the stable API.

There are still a few functions towards the end of python3.def that I need to figure out how to deal with. (Also, my sort script doesn't quite match the result from however Serhiy sorted it last, but it's as close as I could get.)
msg284077 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-12-27 09:36
It is not good that the script removes existing names and adds non-existing names. It is safe to remove names that don't exist in all releases starting from 3.2.0.

1. PyCFunction_New shouldn't be removed in maintained releases. This name is exported, but is unlikely used by third-party code. It can be removed in separate issue, and only in 3.7.

2. PyCmpWrapper_Type shouldn't be added. It is not defined in 3.x.

3. _PyTrash_destroy_chain and like should not be removed for compatibility with 3.2.0-3.2.3.

4. _PyBytes_DecodeEscape shouldn't be added. It is private.

5. Exporting Py_hexdigits looks doubtful. This constant was added for internal use (like _Py_SwappedOp). Unlikely it is used in third-party code. Not attributing it as private looks an error.

6. _Py_TrueStruct shouldn't be removed. It is used in the Py_True macro.

7. *_SizeT names shouldn't be removed. They are used when PY_SSIZE_T_CLEAN is defined before including Python.h (recommended).

8. I think that rather than adding PyODict_Type to limited API, PyODict_Check and PyODict_CheckExact should be excluded from limited API (see issue29058). They never worked in limited API. Note that PyDict_Type as well as other types are not included in limited API. We can consider exposing PyODict_Check and PyODict_CheckExact as functions in limited API in 3.7, but this is other issue.

As for PyArg_VaParse and PyArg_VaParseTupleAndKeywords see issue11626. Seems they were excluded from limited API by mistake.

I suggest first fix errors in python3.def, remove non-existing names, add new names, and only after this add the ability of generating it if it doesn't break python3.def.
msg284079 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-12-27 10:01
New changeset 8423f86486b3 by Serhiy Storchaka in branch '3.5':
Issue #23903: Fixed errors and remove non-existing names in python3.def.
https://hg.python.org/cpython/rev/8423f86486b3

New changeset b5470d08969c by Serhiy Storchaka in branch '3.6':
Issue #23903: Fixed errors and remove non-existing names in python3.def.
https://hg.python.org/cpython/rev/b5470d08969c

New changeset 45507a5751d8 by Serhiy Storchaka in branch 'default':
Issue #23903: Fixed errors and remove non-existing names in python3.def.
https://hg.python.org/cpython/rev/45507a5751d8
msg284083 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-12-27 10:18
New changeset d95fee442e27 by Serhiy Storchaka in branch '3.5':
Issue #23903: Added missed names to PC/python3.def.
https://hg.python.org/cpython/rev/d95fee442e27

New changeset cb864fc4b3be by Serhiy Storchaka in branch '3.6':
Issue #23903: Added missed names to PC/python3.def.
https://hg.python.org/cpython/rev/cb864fc4b3be

New changeset 513852ad0c5c by Serhiy Storchaka in branch 'default':
Issue #23903: Added missed names to PC/python3.def.
https://hg.python.org/cpython/rev/513852ad0c5c
msg284105 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-12-27 14:53
Unfortunately, there's no way too remove defined names from the DLL during Python 3 at all, except where the prototype was never provided. PyCFunction_New has always been a macro, and PySys_SetDefaultEncoding looks to have been removed before the API was committed.

Maintaining an export and a macro is going to be awkward, so I think we should just change them back to functions. Any half-decent optimizing compiler is going to inline it within core, and it's better than having two implementations in different places of the same function.

But yes I agree, fix the def first and then automate. But I don't want to fix it but adding things that were never usable.
msg284108 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-12-27 15:55
New changeset 2e5ad97c9c19 by Serhiy Storchaka in branch '3.5':
Issue #23903: Added missed Windows-specific names to PC/python3.def.
https://hg.python.org/cpython/rev/2e5ad97c9c19

New changeset 86a412584c02 by Serhiy Storchaka in branch '3.6':
Issue #23903: Added missed Windows-specific names to PC/python3.def.
https://hg.python.org/cpython/rev/86a412584c02

New changeset 3a2595f82447 by Serhiy Storchaka in branch 'default':
Issue #23903: Added missed Windows-specific names to PC/python3.def.
https://hg.python.org/cpython/rev/3a2595f82447
History
Date User Action Args
2019-01-23 18:13:59naschemesetnosy: + nascheme
2017-04-01 05:49:38serhiy.storchakasetpull_requests: - pull_request1091
2017-03-31 16:36:37dstufftsetpull_requests: + pull_request1091
2016-12-27 15:55:15python-devsetmessages: + msg284108
2016-12-27 14:53:45steve.dowersetmessages: + msg284105
2016-12-27 13:42:14serhiy.storchakasetdependencies: + Readd PyArg_VaParse to the stable API
2016-12-27 10:18:06python-devsetmessages: + msg284083
2016-12-27 10:01:12python-devsetmessages: + msg284079
2016-12-27 09:36:41serhiy.storchakasetmessages: + msg284077
2016-12-26 21:38:37steve.dowersetfiles: + 23903_1.patch

messages: + msg284047
2016-12-26 20:16:37serhiy.storchakasetmessages: + msg284043
2016-12-26 19:49:49steve.dowersetmessages: + msg284039
2016-12-24 21:24:13serhiy.storchakasetfiles: + python3def-3.6.patch

messages: + msg283973
2016-12-24 20:49:59serhiy.storchakasetfiles: + genpython3def.sh

dependencies: + Mark new limited C API
messages: + msg283972
2016-12-24 00:12:50serhiy.storchakasetmessages: + msg283911
2016-12-23 23:29:46steve.dowersetmessages: + msg283907
2016-12-23 20:28:51serhiy.storchakasetfiles: + python3def.patch

messages: + msg283902
2016-12-23 20:14:34python-devsetnosy: + python-dev
messages: + msg283901
2016-12-23 18:17:13steve.dowersetmessages: + msg283891
2016-12-23 06:39:33serhiy.storchakasetmessages: + msg283872
2016-12-21 22:16:22steve.dowersetmessages: + msg283791
2016-12-21 20:52:43ned.deilysetmessages: + msg283784
2016-12-21 17:18:07steve.dowersetnosy: + ned.deily
messages: + msg283770
2016-12-21 01:42:00steve.dowersetfiles: + python3def.patch

messages: + msg283715
versions: + Python 3.5, Python 3.7
2016-05-02 15:39:46zach.waresetmessages: + msg264652
2016-05-02 06:22:17serhiy.storchakasetmessages: + msg264626
2016-05-02 06:09:12steve.dowersetmessages: + msg264624
2016-05-02 05:14:00serhiy.storchakasetmessages: + msg264619
2016-05-02 02:42:44zach.waresetassignee: zach.ware ->
messages: + msg264614
versions: + Python 3.6, - Python 3.5
2015-05-28 15:47:36steve.dowersetmessages: + msg244309
2015-05-24 18:31:34steve.dowersetmessages: + msg243997
2015-05-24 17:54:48zach.waresetmessages: + msg243994
2015-04-16 22:00:54steve.dowersetmessages: + msg241281
2015-04-16 19:10:36loewissetnosy: + loewis
messages: + msg241262
2015-04-16 00:12:54serhiy.storchakasetnosy: + serhiy.storchaka

stage: needs patch -> patch review
2015-04-13 21:11:10zach.waresetfiles: + 23903_with_python3.def.diff

messages: + msg240754
2015-04-10 03:49:21zach.warecreate