classification
Title: Move PyInterpreterState into Include/internal/pycore_pystate.h
Type: Stage: resolved
Components: Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: eric.snow Nosy List: arigo, eric.snow, matrixise, ncoghlan, njs, vinay.sajip, vstinner
Priority: normal Keywords: patch, patch, patch

Created on 2019-02-02 01:07 by eric.snow, last changed 2019-09-27 14:45 by eric.snow.

Pull Requests
URL Status Linked Edit
PR 11731 merged eric.snow, 2019-02-02 01:24
Messages (23)
msg334733 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2019-02-02 01:07
In November Victor created the Include/cpython directory and moved a decent amount of public (but not limited) API there.  This included the PyInterpreterState struct.  I'd like to move it into the "internal" headers since it is somewhat coupled to the internal runtime implementation.  The alternative is extra complexity.  I ran into this while working on another issue.

Note that the docs indicate that all of the struct's fields are private (and I am not aware of any macros leaking fields into the stable ABI).  So moving it should not break anything (yeah, right!), and certainly not the stable ABI (Victor's move would have done that already).
msg334734 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2019-02-02 01:32
FWIW, I was hoping to the same for PyThreadState but it looks like 6 of its fields are exposed in "stable" header files via the following macros:

  # Include/object.h
  Py_TRASHCAN_SAFE_BEGIN
  Py_TRASHCAN_SAFE_END
  Include.ceval.h
  Py_EnterRecursiveCall
  Py_LeaveRecursiveCall
  _Py_MakeRecCheck
  Py_ALLOW_RECURSION
  Py_END_ALLOW_RECURSION

I'm not sure how that factors into the stable ABI (PyThreadState wasn't ever guarded by Py_LIMITED_API).  However, I didn't need to deal with it right now so I'm not going to go there. :)
msg335120 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2019-02-09 01:14
@Victor, do you see any problems with doing this?  It will help simplify other changes I'm working on.
msg335464 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-02-13 17:05
> @Victor, do you see any problems with doing this?  It will help simplify other changes I'm working on.

I'm quite sure that they are users of the PyInterpreterState structure outside CPython internals and stdlib, but I expect that the number is quite low.

Since internal headers are now installed (I modified "make install" for that) (but need to define Py_BUILD_CORE), it might be acceptable to force users of this structure to opt-in for internal headers.

Just make sure that we properly communicate on such backward incompatible changes:

* "C API Changes" section of https://docs.python.org/dev/whatsnew/3.8.html#porting-to-python-3-8
* mail to python-dev

The bpo-35810 also proposes a subtle backward incompatible change which makes me unhappy, but Stefan Behnel seems less scared than me, so maybe it will be fine.

Maybe we need to organize a collective effort to better communicate on our backward incompatible C API changes. The capi-sig mailing list may be a good channel for that. I asked to test some popular C extensions to check that they are not broken. If it's the case, we should help them to be prepared for the new C API.
msg335657 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2019-02-15 23:39
Thanks, Victor!

python-dev: https://mail.python.org/pipermail/python-dev/2019-February/156344.html

Also, my PR already has a What's New entry in the porting section. :)
msg336397 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2019-02-23 18:35
New changeset be3b295838547bba267eb08434b418ef0df87ee0 by Eric Snow in branch 'master':
bpo-35886: Make PyInterpreterState an opaque type in the public API. (GH-11731)
https://github.com/python/cpython/commit/be3b295838547bba267eb08434b418ef0df87ee0
msg336500 - (view) Author: Nathaniel Smith (njs) * (Python committer) Date: 2019-02-25 06:54
This broke cffi:

    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/opt/python/3.8-dev/include/python3.8m -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.8/c/_cffi_backend.o
    In file included from c/cffi1_module.c:20:0,
                     from c/_cffi_backend.c:7552:
    c/call_python.c: In function ‘_get_interpstate_dict’:
    c/call_python.c:20:30: error: dereferencing pointer to incomplete type ‘PyInterpreterState {aka struct _is}’
         builtins = tstate->interp->builtins;
                                  ^
    error: command 'gcc' failed with exit status 1

I haven't investigated further but heads up.
msg336501 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2019-02-25 07:49
Just so you know, when we look at the changes to CPython, the easiest fix is to add these lines in cffi:

    #if PY_VERSION_HEX >= 0x03080000
    # define Py_BUILD_CORE
    #  include "internal/pycore_pystate.h"
    # undef Py_BUILD_CORE
    #endif

But if we're looking for a cleaner way to fix this, then cffi's needs could be covered by adding a general-purpose-storage dict to PyInterpreterState (like PyThreadState->dict, but per sub-interpreter instead of per thread), and an API function to get it.
msg336506 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-02-25 10:02
> This broke cffi:

Well, that's not a surprise :-) It's a deliberate backward incompatible change.

But that's where I suggested to test "popular C extensions" with a C API change before merging it. It's ok if issues are discovered later, but the author of backward incompatible C API changes should collaborate with popular C extension maintainers to help to deal with these issues.

Can someone report the issue to cffi and post back the url to the bug here?
msg336510 - (view) Author: Nathaniel Smith (njs) * (Python committer) Date: 2019-02-25 10:41
The cffi issue is: https://bitbucket.org/cffi/cffi/issues/403/build-fails-on-38-dev-pyinterpreterstate

Armin already went ahead and committed the change to make cffi include internal/pycore_pystate.h. I guess this might not be the ideal final outcome though :-). Without some fix though it's very difficult to test things on 3.8-dev, because cffi shows up in the dependency stacks of *large* portions of the ecosystem.
msg336590 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2019-02-26 01:03
@Armin, thanks for fixing things on your end so quickly. :)

Adding PyInterpreterState.dict and a public getter function (a la PyThreadState) is probably fine.  I'm just not sure how that relates to the problem with cffi's use of the "builtins" field.  If it would be useful in general then feel free to open a separate issue.  I doubt there'd be much opposition if you presented a decent use case, since the cost would be low.

In the case of PyInterpreterState.builtins specifically, if you need it we could add a simple PyInterpreterState_GetBuiltins().  However, couldn't you already use existing public API, e.g. `PyEval_GetBuiltins()` or `PyImport_GetModule("builtins")`?  Regardless, we certainly don't want to put anyone in a position that they must define Py_BUILD_CORE just for access to PyInterpreterState fields.
msg336591 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2019-02-26 01:05
On Mon, Feb 25, 2019 at 3:02 AM STINNER Victor <report@bugs.python.org> wrote:
> But that's where I suggested to test "popular C extensions" with a C API change before merging it. It's ok if issues are discovered later, but the author of backward incompatible C API changes should collaborate with popular C extension maintainers to help to deal with these issues.

Yeah, that was my bad.  I jumped the gun a bit.
msg336659 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2019-02-26 12:53
Next incompatibility: https://github.com/python-hyper/brotlipy/issues/147 (which indirectly broke httpbin)
msg336660 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2019-02-26 12:59
(On closer inspection, that's actually be the same breakage as already mentioned above)

However, what I'm not clear on is how this would affect projects that had *already* generated their cffi code, and include that in their sdist. Are all those sdists going to fail to build on Python 3.8 now?

It's OK to require that extension modules be rebuilt for a new release, but breaking compatibility with a *code generator* that means a broad selection of projects are all going to fail to build is a much bigger problem.
msg336661 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2019-02-26 13:04
@nick which indirectly broke httpbin and this one is used by python-requests and we can't execute the tests of requests.
msg336664 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2019-02-26 13:22
@nick the C sources produced by cffi don't change.  When they are compiled, they use Py_LIMITED_API so you can continue using a single compiled module version for any recent-enough Python 3.x.  The required fix is only inside the cffi module itself.
msg336671 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2019-02-26 13:47
Oh, cool (both the fact the issue here is only with building cffi itself, and that cffi creates extension modules that build with PY_LIMITED_API).

I've filed https://bugs.python.org/issue36124 to follow up on the PyInterpreter_GetDict API idea.
msg336676 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2019-02-26 14:37
Cool.  Also, no bugfix release of cffi was planned, but I can make one if you think it might help for testing the current pre-release of CPython 3.8.
msg336679 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2019-02-26 14:40
@arigo 

Yep, I am interested because I would like to execute the tests of the major projects/libraries (django, flasks, pandas, requests, ...) and create issues for the maintainer.

the sooner we get feedback, the sooner we can fix the bugs.
msg336682 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2019-02-26 15:13
Done.
msg353093 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-24 15:04
This change broke at least two projects:

* Blender: access "interp->modules" but also "interp->builtins"
* FreeCAD: access "interp->modules"

It would be nice to document replacement solutions at:
https://docs.python.org/dev/whatsnew/3.8.html#porting-to-python-3-8

* Replace "interp->modules" with PyImport_GetModuleDict()
* Replace "interp->builtins" with PyEval_GetBuiltins()
msg353094 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-24 15:05
Note: IMHO the "The PyInterpreterState struct has been moved into the “internal” header files (...)" entry should be documented in the "Changes in the C API" section, rather than in the "Changes in the Python API" section of:
https://docs.python.org/dev/whatsnew/3.8.html#porting-to-python-3-8
msg353370 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2019-09-27 14:45
+1 to the suggested "whatsnew" updates.  If no one beats me to it I'll work on a PR soon.
History
Date User Action Args
2019-09-27 14:45:12eric.snowsetkeywords: patch, patch, patch

messages: + msg353370
2019-09-24 17:12:28vinay.sajipsetkeywords: patch, patch, patch
nosy: + vinay.sajip
2019-09-24 15:05:17vstinnersetkeywords: patch, patch, patch

messages: + msg353094
2019-09-24 15:04:25vstinnersetkeywords: patch, patch, patch
status: closed -> open
resolution: fixed ->
messages: + msg353093
2019-02-26 15:13:12arigosetkeywords: patch, patch, patch

messages: + msg336682
2019-02-26 14:40:28matrixisesetkeywords: patch, patch, patch

messages: + msg336679
2019-02-26 14:37:29arigosetkeywords: patch, patch, patch

messages: + msg336676
2019-02-26 13:47:15ncoghlansetkeywords: patch, patch, patch

messages: + msg336671
2019-02-26 13:22:53arigosetkeywords: patch, patch, patch

messages: + msg336664
2019-02-26 13:04:46matrixisesetkeywords: patch, patch, patch
nosy: + matrixise
messages: + msg336661

2019-02-26 12:59:14ncoghlansetkeywords: patch, patch, patch

messages: + msg336660
2019-02-26 12:53:05ncoghlansetkeywords: patch, patch, patch

messages: + msg336659
2019-02-26 01:05:47eric.snowsetmessages: + msg336591
2019-02-26 01:03:59eric.snowsetkeywords: patch, patch, patch

messages: + msg336590
2019-02-25 10:41:14njssetkeywords: patch, patch, patch

messages: + msg336510
2019-02-25 10:02:12vstinnersetkeywords: patch, patch, patch

messages: + msg336506
2019-02-25 07:49:01arigosetkeywords: patch, patch, patch

messages: + msg336501
2019-02-25 06:54:20njssetkeywords: patch, patch, patch
nosy: + njs, arigo
messages: + msg336500

2019-02-23 18:36:41eric.snowsetkeywords: patch, patch, patch
status: open -> closed
resolution: fixed
stage: patch review -> resolved
2019-02-23 18:35:56eric.snowsetmessages: + msg336397
2019-02-15 23:39:45eric.snowsetkeywords: patch, patch, patch

messages: + msg335657
2019-02-13 17:05:33vstinnersetkeywords: patch, patch, patch

messages: + msg335464
2019-02-09 01:45:42eric.snowsetkeywords: patch, patch, patch
nosy: + ncoghlan
2019-02-09 01:14:43eric.snowsetkeywords: patch, patch, patch

messages: + msg335120
2019-02-08 16:55:51eric.snowsetpull_requests: - pull_request11624
2019-02-08 16:55:34eric.snowsetpull_requests: - pull_request11625
2019-02-02 01:32:46eric.snowsetkeywords: patch, patch, patch

messages: + msg334734
2019-02-02 01:25:03eric.snowsetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request11625
2019-02-02 01:24:58eric.snowsetkeywords: + patch
stage: needs patch -> needs patch
pull_requests: + pull_request11624
2019-02-02 01:24:52eric.snowsetkeywords: + patch
stage: needs patch -> needs patch
pull_requests: + pull_request11623
2019-02-02 01:07:08eric.snowcreate