In November Victor created the Include/cpython directory and moved a decent amount of public (but not limited) API there.  This included the PyInterpreterState struct.  I'd like to move it into the "internal" headers since it is somewhat coupled to the internal runtime implementation.  The alternative is extra complexity.  I ran into this while working on another issue.

Note that the docs indicate that all of the struct's fields are private (and I am not aware of any macros leaking fields into the stable ABI).  So moving it should not break anything (yeah, right!), and certainly not the stable ABI (Victor's move would have done that already).

FWIW, I was hoping to the same for PyThreadState but it looks like 6 of its fields are exposed in "stable" header files via the following macros:

  # Include/object.h
  Py_TRASHCAN_SAFE_BEGIN
  Py_TRASHCAN_SAFE_END
  Include.ceval.h
  Py_EnterRecursiveCall
  Py_LeaveRecursiveCall
  _Py_MakeRecCheck
  Py_ALLOW_RECURSION
  Py_END_ALLOW_RECURSION

I'm not sure how that factors into the stable ABI (PyThreadState wasn't ever guarded by Py_LIMITED_API).  However, I didn't need to deal with it right now so I'm not going to go there. :)

@Victor, do you see any problems with doing this?  It will help simplify other changes I'm working on.

> @Victor, do you see any problems with doing this?  It will help simplify other changes I'm working on.

I'm quite sure that they are users of the PyInterpreterState structure outside CPython internals and stdlib, but I expect that the number is quite low.

Since internal headers are now installed (I modified "make install" for that) (but need to define Py_BUILD_CORE), it might be acceptable to force users of this structure to opt-in for internal headers.

Just make sure that we properly communicate on such backward incompatible changes:

* "C API Changes" section of https://docs.python.org/dev/whatsnew/3.8.html#porting-to-python-3-8
* mail to python-dev

The bpo-35810 also proposes a subtle backward incompatible change which makes me unhappy, but Stefan Behnel seems less scared than me, so maybe it will be fine.

Maybe we need to organize a collective effort to better communicate on our backward incompatible C API changes. The capi-sig mailing list may be a good channel for that. I asked to test some popular C extensions to check that they are not broken. If it's the case, we should help them to be prepared for the new C API.

Thanks, Victor!

python-dev: https://mail.python.org/pipermail/python-dev/2019-February/156344.html

Also, my PR already has a What's New entry in the porting section. :)

New changeset be3b295838547bba267eb08434b418ef0df87ee0 by Eric Snow in branch 'master':
bpo-35886: Make PyInterpreterState an opaque type in the public API. (GH-11731)
https://github.com/python/cpython/commit/be3b295838547bba267eb08434b418ef0df87ee0

This broke cffi:

    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/opt/python/3.8-dev/include/python3.8m -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.8/c/_cffi_backend.o
    In file included from c/cffi1_module.c:20:0,
                     from c/_cffi_backend.c:7552:
    c/call_python.c: In function ‘_get_interpstate_dict’:
    c/call_python.c:20:30: error: dereferencing pointer to incomplete type ‘PyInterpreterState {aka struct _is}’
         builtins = tstate->interp->builtins;
                                  ^
    error: command 'gcc' failed with exit status 1

I haven't investigated further but heads up.

Just so you know, when we look at the changes to CPython, the easiest fix is to add these lines in cffi:

    #if PY_VERSION_HEX >= 0x03080000
    # define Py_BUILD_CORE
    #  include "internal/pycore_pystate.h"
    # undef Py_BUILD_CORE
    #endif

But if we're looking for a cleaner way to fix this, then cffi's needs could be covered by adding a general-purpose-storage dict to PyInterpreterState (like PyThreadState->dict, but per sub-interpreter instead of per thread), and an API function to get it.

> This broke cffi:

Well, that's not a surprise :-) It's a deliberate backward incompatible change.

But that's where I suggested to test "popular C extensions" with a C API change before merging it. It's ok if issues are discovered later, but the author of backward incompatible C API changes should collaborate with popular C extension maintainers to help to deal with these issues.

Can someone report the issue to cffi and post back the url to the bug here?

The cffi issue is: https://bitbucket.org/cffi/cffi/issues/403/build-fails-on-38-dev-pyinterpreterstate

Armin already went ahead and committed the change to make cffi include internal/pycore_pystate.h. I guess this might not be the ideal final outcome though :-). Without some fix though it's very difficult to test things on 3.8-dev, because cffi shows up in the dependency stacks of *large* portions of the ecosystem.

@Armin, thanks for fixing things on your end so quickly. :)

Adding PyInterpreterState.dict and a public getter function (a la PyThreadState) is probably fine.  I'm just not sure how that relates to the problem with cffi's use of the "builtins" field.  If it would be useful in general then feel free to open a separate issue.  I doubt there'd be much opposition if you presented a decent use case, since the cost would be low.

In the case of PyInterpreterState.builtins specifically, if you need it we could add a simple PyInterpreterState_GetBuiltins().  However, couldn't you already use existing public API, e.g. `PyEval_GetBuiltins()` or `PyImport_GetModule("builtins")`?  Regardless, we certainly don't want to put anyone in a position that they must define Py_BUILD_CORE just for access to PyInterpreterState fields.

On Mon, Feb 25, 2019 at 3:02 AM STINNER Victor <report@bugs.python.org> wrote:
> But that's where I suggested to test "popular C extensions" with a C API change before merging it. It's ok if issues are discovered later, but the author of backward incompatible C API changes should collaborate with popular C extension maintainers to help to deal with these issues.

Yeah, that was my bad.  I jumped the gun a bit.

Next incompatibility: https://github.com/python-hyper/brotlipy/issues/147 (which indirectly broke httpbin)

(On closer inspection, that's actually be the same breakage as already mentioned above)

However, what I'm not clear on is how this would affect projects that had *already* generated their cffi code, and include that in their sdist. Are all those sdists going to fail to build on Python 3.8 now?

It's OK to require that extension modules be rebuilt for a new release, but breaking compatibility with a *code generator* that means a broad selection of projects are all going to fail to build is a much bigger problem.

@nick which indirectly broke httpbin and this one is used by python-requests and we can't execute the tests of requests.

@nick the C sources produced by cffi don't change.  When they are compiled, they use Py_LIMITED_API so you can continue using a single compiled module version for any recent-enough Python 3.x.  The required fix is only inside the cffi module itself.

Oh, cool (both the fact the issue here is only with building cffi itself, and that cffi creates extension modules that build with PY_LIMITED_API).

I've filed https://bugs.python.org/issue36124 to follow up on the PyInterpreter_GetDict API idea.

Cool.  Also, no bugfix release of cffi was planned, but I can make one if you think it might help for testing the current pre-release of CPython 3.8.

@arigo 

Yep, I am interested because I would like to execute the tests of the major projects/libraries (django, flasks, pandas, requests, ...) and create issues for the maintainer.

the sooner we get feedback, the sooner we can fix the bugs.

Done.

This change broke at least two projects:

* Blender: access "interp->modules" but also "interp->builtins"
* FreeCAD: access "interp->modules"

It would be nice to document replacement solutions at:
https://docs.python.org/dev/whatsnew/3.8.html#porting-to-python-3-8

* Replace "interp->modules" with PyImport_GetModuleDict()
* Replace "interp->builtins" with PyEval_GetBuiltins()

Note: IMHO the "The PyInterpreterState struct has been moved into the “internal” header files (...)" entry should be documented in the "Changes in the C API" section, rather than in the "Changes in the Python API" section of:
https://docs.python.org/dev/whatsnew/3.8.html#porting-to-python-3-8

+1 to the suggested "whatsnew" updates.  If no one beats me to it I'll work on a PR soon.