Currently, #include "pymem.h" may include Include/pymem.h or Include/internal/pymem.h depending where is the C file (.c) and if Include/internal/ is in the header search path or not.

I propose to:

* Rename Include/internal/ to Include/pycore/
* In this subdirectory, rename xxx.h to pycore_xxx.h to avoid any risk of confusion
* Automatically include pycore_xxx.h in xxx.h if Py_BUILD_CORE is defined: this should avoid the need of explicit #include "internal/xxx.h" in C files

For example, Include/internal/pystate.h becomes Include/pycore/pycore_pystate.h.

Attached PR implements this idea.

I chose to rename "internal" subdirectory to "pycore" to prepare the addition of other subdirectories. See:

* https://pythoncapi.readthedocs.io/split_include.html
* https://pythoncapi.readthedocs.io/

Next steps:

* Move all code surrounded by #ifdef Py_BUILD_CORE from Include/*.h into Include/pycore/*.h: see the second commit of my PR for an example
* Move core surrounded by #ifndef Py_LIMITED_API from Include/*.h into Include/limited/ (example of filename: Include/limited/limited_object.h)


This change should be backward compatible since Include/internal/ must not be used outside CPython core. If someone does that, well, be ready for breakage :-) It's not supported.

I decided to work on this issue while trying to convert the _PyObject_GC_TRACK() macro into a "static inline" function. Currently, the macro uses _PyGC_generation0 which is defined earlier by "extern PyGC_Head *_PyGC_generation0;".

Problem: _PyGC_generation0 no longer exists... Include/internal/mem.h now defines: "#define _PyGC_generation0 _PyRuntime.gc.generation0".

Include/internal/mem.h includes Include/objimpl.h, but Include/objimpl.h requires Include/internal/mem.h. The include order matters here, many header files are inter-dependent, and have two header files with the same name in Include/ and Include/internal/ causes issues depending where the #include is done...

My PR renames mem.h to pycore_objimpl.h and include pycore_objimpl.h at "the right place" in objimpl.h. Since objimpl.h controls where pycore_objimpl.h is imported, it's simpler to handle inter-dependencies simpler.

Some inter-dependencies issues are hidden by the fact the C macros don't really require functions, macros and variables in the right order. They "just work". But converting C macros to proper "static inline" functions expose many issues.

See also bpo-35059 which converts C macros to static inline functions.

Discussion on python-dev:
https://mail.python.org/pipermail/python-dev/2018-October/155587.html

New changeset 9204fb8623896cc5f68ae350784ee25e8a7b2184 by Victor Stinner in branch 'master':
bpo-35081: Cleanup pystate.c and pystate.h (GH-10240)
https://github.com/python/cpython/commit/9204fb8623896cc5f68ae350784ee25e8a7b2184

New changeset 31368a4f0e531c19affe2a1becd25fc316bc7501 by Victor Stinner in branch 'master':
bpo-35081: Move Include/pyatomic.c to Include/internal/ (GH-10239)
https://github.com/python/cpython/commit/31368a4f0e531c19affe2a1becd25fc316bc7501

Serhiy Storchaka asked: "Would not be better to move files with the content fully surrounded by #ifdef Py_BUILD_CORE out of the Include/ directory?"

See his comment and my answer in the PR 10239:
https://github.com/python/cpython/pull/10239#issuecomment-434289361

My changes move Py_BUILD_CORE to Include/internal/. I'm not sure of the effect on the backward compatibility.

Since Python 3.7, many "Py_BUILD_CORE" functions rely on Include/internal/, like PyThreadState_GET() which uses _PyRuntime.gilstate.tstate_current.

On my Fedora 28, the python3-devel package doesn't proide Include/internal/ headers, only Include/*.h in /usr/include/python3.7m/. It seems like Include/internal/ is not usable by 3rd party modules on Fedora at least.

I understand that even if a 3rd party C extension used the Py_BUILD_CORE API, Python 3.7 already broke these extensions.

I don't want C extensions to use Py_BUILD_CORE: Py_BUILD_CORE API is really designed to only be used inside Python. If this API is used outside Python, we cannot modify the API anymore since it would break extensions. But I want to make sure that we can break this API for different reasons.

In Python 3.7, pyatomic.h is included by Python.h. In Python 3.7.0, pyatomic.h content wasn't surrounded by Py_BUILD_CORE and this header file caused multiple compilation issues: see bpo-23644 and bpo-25150. The content is now restricted to Py_BUILD_CORE since Python 3.7.1. It allows us to more easily change the implementation.

Include/internal/pystate.h uses #include "pystate.h" to include Include/pystate.h, but it tries to include itself (Include/internal/pystate.h) which does nothing because of "#ifndef Py_INTERNAL_PYSTATE_H #define Py_INTERNAL_PYSTATE_H ... #endif".

Remove the #ifndef #define to see the bug:

diff --git a/Include/internal/pystate.h b/Include/internal/pystate.h
index 38845d32ec..2ef023a9a5 100644
--- a/Include/internal/pystate.h
+++ b/Include/internal/pystate.h
@@ -1,5 +1,3 @@
-#ifndef Py_INTERNAL_PYSTATE_H
-#define Py_INTERNAL_PYSTATE_H
 #ifdef __cplusplus
 extern "C" {
 #endif
@@ -222,4 +220,3 @@ PyAPI_FUNC(void) _PyInterpreterState_DeleteExceptMain(void);
 #ifdef __cplusplus
 }
 #endif
-#endif /* !Py_INTERNAL_PYSTATE_H */


Compilation fails with:

In file included from ./Include/internal/pystate.h:5,
                 from ./Include/internal/pystate.h:5,
                 from ./Include/internal/pystate.h:5,
                 from ./Include/internal/pystate.h:5,
                 from ./Include/internal/pystate.h:5,
                 from ./Include/internal/pystate.h:5,
                 from ./Include/internal/pystate.h:5,
                 ...
./Include/internal/pystate.h:5:21: error: #include nested too deeply
 #include "pystate.h"

New changeset 2be00d987d37682a55db67c298e82c405d01b868 by Victor Stinner in branch 'master':
bpo-35081: Move Py_BUILD_CORE code to internal/mem.h (GH-10249)
https://github.com/python/cpython/commit/2be00d987d37682a55db67c298e82c405d01b868

> Include/internal/pystate.h uses #include "pystate.h" to include Include/pystate.h, but it tries to include itself (Include/internal/pystate.h) which does nothing because of "#ifndef Py_INTERNAL_PYSTATE_H #define Py_INTERNAL_PYSTATE_H ... #endif".

I proposed to rename internal header files, add an "internal_" prefix, to avoid this issue: PR 10263.

New changeset 27e2d1f21975dfb8c0ddcb192fa0f45a51b7977e by Victor Stinner in branch 'master':
bpo-35081: Add pycore_ prefix to internal header files (GH-10263)
https://github.com/python/cpython/commit/27e2d1f21975dfb8c0ddcb192fa0f45a51b7977e

New changeset 50b48572d9a90c5bb36e2bef6179548ea927a35a by Victor Stinner in branch 'master':
bpo-35081: Add _PyThreadState_GET() internal macro (GH-10266)
https://github.com/python/cpython/commit/50b48572d9a90c5bb36e2bef6179548ea927a35a

New changeset e281f7d80ce2584a7e6a36acffb5a9cd796a0fe2 by Victor Stinner in branch 'master':
bpo-35081: Move accu.h to Include/internal/pycore_accu.h (GH-10271)
https://github.com/python/cpython/commit/e281f7d80ce2584a7e6a36acffb5a9cd796a0fe2

Copy of my comment on PR 10271:
https://github.com/python/cpython/pull/10271#issuecomment-434897408

I tried to enforce to require Py_BUILD_CORE in pycore_accu.h to be defined using:

#ifndef Py_BUILD_CORE
#  error "Py_BUILD_CORE must be defined to include this header"
#endif

But the compilation of the _json module failed, because it isn't compiled with Py_BUILD_CORE. Moreover, _json.c contains:

/* Core extension modules are built-in on some platforms (e.g. Windows). */
#ifdef Py_BUILD_CORE
#define Py_BUILD_CORE_BUILTIN
#undef Py_BUILD_CORE
#endif

New changeset a1c249c40517917d2e0971d55aea8d14a44b2cc8 by Victor Stinner in branch 'master':
bpo-35081: And pycore_lifecycle.h and pycore_pathconfig.h (GH-10273)
https://github.com/python/cpython/commit/a1c249c40517917d2e0971d55aea8d14a44b2cc8

> bpo-35081: And pycore_lifecycle.h and pycore_pathconfig.h (GH-10273)

I tried to add the Py_BUILD_CORE guard in pycore_pathconfig.h:

+#ifndef Py_BUILD_CORE
+#  error "Py_BUILD_CORE must be defined to include this header"
+#endif

But it breaks the compilation of _testcapimodule.c: get_coreconfig() uses _Py_wstrlist_as_pylist(), and _Py_wstrlist_as_pylist() is defined in pycore_pathconfig.h.

IMHO _testcapi should be compiled with Py_BUILD_CORE defined. I wrote PR 10274 but then _testcapi fails because of datetime.h: this bug should be fixed by PR 10238.

I created bpo-35134: Move !Py_LIMITED_API to Include/pycapi/.

Please don't change Include/datetime.h without consulting with original authors of this code (see issue876130).

> Please don't change Include/datetime.h without consulting with original authors of this code (see issue876130).

FYI I tried to be very careful on each change to never modify the *public* C API.

But I modified the Py_BUILD_CORE API. This is fine since this API can change anytime, and it must not be used outside Python.

New changeset 5ed6995675b084fe583b71f96fdde4413bb2a77b by Victor Stinner in branch 'master':
bpo-35081: Add _PyCoreConfig_AsDict() (GH-10362)
https://github.com/python/cpython/commit/5ed6995675b084fe583b71f96fdde4413bb2a77b

New changeset 9fc57a384825530635ef5ec093a31d864ea14f7c by Victor Stinner in branch 'master':
bpo-35081: Add pycore_fileutils.h (GH-10371)
https://github.com/python/cpython/commit/9fc57a384825530635ef5ec093a31d864ea14f7c

New changeset 130893debfd97c70e3a89d9ba49892f53e6b9d79 by Victor Stinner in branch 'master':
bpo-35081: Internal headers require Py_BUILD_CORE (GH-10363)
https://github.com/python/cpython/commit/130893debfd97c70e3a89d9ba49892f53e6b9d79

Victor, you moved declarations of some functions to other headers, but didn't include the new headers into files that implement the functions in some cases. For example, _PyGILState_Init was moved into Include/internal/pycore_lifecycle.h in a1c249c40517917d2e0971d55aea8d14a44b2cc8, but it's implemented in Python/pystate.c, which doesn't include the new header.

This may lead to subtle problems because the compiler can't check that signatures of the declaration and the implementation match. I suggest to use -Wmissing-prototypes and -Wmissing-declarations to detect such situations:

../../cpython/Python/pystate.c: At top level:
../../cpython/Python/pystate.c:968:1: warning: no previous prototype for ‘_PyGILState_Init’ [-Wmissing-prototypes]
 _PyGILState_Init(PyInterpreterState *i, PyThreadState *t)
 ^~~~~~~~~~~~~~~~
../../cpython/Python/pystate.c:988:1: warning: no previous prototype for ‘_PyGILState_Fini’ [-Wmissing-prototypes]
 _PyGILState_Fini(void)
 ^~~~~~~~~~~~~~~~

Sadly, there are many other similar issues in Python now, but you can at least compare the number of warnings before and after your changes.

> Python/pystate.c:968:1: warning: no previous prototype for ‘_PyGILState_Init’ [-Wmissing-prototypes]
 _PyGILState_Init(PyInterpreterState *i, PyThreadState *t)
 ^~~~~~~~~~~~~~~~

Oh, I never saw this warning before. It seems to not be included in -Wall. Would you mind to open a new issue to discuss it?

I will try to fix the regressions that I introduced, but I'm interested by a more general discussion on this issue.

New changeset 621cebe81b1e6c8de10425955bf532d31ee4df42 by Victor Stinner in branch 'master':
bpo-35081: Rename internal headers (GH-10275)
https://github.com/python/cpython/commit/621cebe81b1e6c8de10425955bf532d31ee4df42

New changeset 0d12672b30b8c6c992bef7564581117ae83e11ad by Victor Stinner (Paul Ganssle) in branch 'master':
bpo-35081: Remove Py_BUILD_CORE from datetime.h (GH-10416)
https://github.com/python/cpython/commit/0d12672b30b8c6c992bef7564581117ae83e11ad

Status: only pyport. and tupleobject.h use Py_BUILD_CORE, and only tupleobject.h uses "#ifdef Py_BUILD_CORE" (contains code specific to internals).

$ grep Py_BUILD_CORE Include/*.h
Include/pyport.h:#               if defined(Py_BUILD_CORE) || defined(Py_BUILD_CORE_BUILTIN)
Include/pyport.h:#               else /* Py_BUILD_CORE */
Include/pyport.h:#               endif /* Py_BUILD_CORE */
Include/pyport.h:#if defined(Py_BUILD_CORE) || defined(Py_BUILD_CORE_BUILTIN)
Include/pyport.h:#endif /* Py_BUILD_CORE */
Include/tupleobject.h:#ifdef Py_BUILD_CORE

tupleobject.c:

#ifdef Py_BUILD_CORE
#  define _PyTuple_ITEMS(op) ((((PyTupleObject *)(op))->ob_item))
#endif

I added this macro in bpo-35199, to prepare the C code for a new C API hiding implementation details: see bpo-35206.

Hum, this issue is much harder than what I expected.

Status:

* Include/internal/ header files have been renamed to get a new "pycore_" prefix which avoids conflicts with Include/ header files. Example: "pycore_pystate.h" vs "pystate.h".
* pyatomic.c has been moved to Include/internal/ and it is no longer 
included in Python.h.
* Except of _PyTuple_ITEMS() and _PyObject_GC_TRACK(), all internal APIs have been moved to Include/internal/. Many new header files have been created, like pycore_lifecycle.h and pycore_pathconfig.h.

TODO:

* Move _PyObject_GC_TRACK() to Include/internal/
* msg329608 describes bugs like: pystate.c:968:1: warning: no previous prototype for ‘_PyGILState_Init’ [-Wmissing-prototypes]
* bpo-35134 now manages moving unstable API to a new separated Include/ subdirectory
* bpo-35059 converts macros to static inline functions

New changeset 1a6be91e6fd65ce9cb88cbbbb193db7e92ec6076 by Victor Stinner in branch 'master':
bpo-35081: Make some _PyGC macros internal (GH-10507)
https://github.com/python/cpython/commit/1a6be91e6fd65ce9cb88cbbbb193db7e92ec6076

> Oh, I never saw this warning before. It seems to not be included in -Wall. Would you mind to open a new issue to discuss it?

Victor, I've opened #35258 as you suggested.

Making _PyGC_FINALIZED() internal broke Cython (https://github.com/cython/cython/issues/2721). It's used in the finaliser implementation (https://github.com/cython/cython/blob/da657c8e326a419cde8ae6ea91be9661b9622504/Cython/Compiler/ModuleNode.py#L1442-L1456), to determine if an object for which tp_dealloc() is called has already been finalised or whether we have to do it. I'm not sure how to deal with this on our side now. Any clue?

> Making _PyGC_FINALIZED() internal broke Cython (...)

Aha, interesting. Would you mind to open a new dedicated issue? So we can discuss how much of the GC details we want to leak into the API :-)

TODO: add a NEWS entry for all these changes.

Copy of Nick Coghlan's comment:
https://github.com/python/cpython/pull/10275#issuecomment-439339982

@vstinner For folks consuming source archives (Linux distros, anyone embedding Python in a larger application, etc) rather than prebuilt binaries, the build process is something that can break, even for Py_BUILD_CORE only changes.

For those folks, when they pull a new source archive, their builds may break, especially if they're applying additional downstream patches the way Linux distros do.

The NEWS entry for this header file rearrangement should go in the Build section (https://github.com/python/cpython/tree/master/Misc/NEWS.d/next/Build) rather than the C API section (since these headers aren't part of the public API at all), but it should still exist.

Stefan Behnel:
> Making _PyGC_FINALIZED() internal broke Cython (https://github.com/cython/cython/issues/2721). It's used in the finaliser implementation (https://github.com/cython/cython/blob/da657c8e326a419cde8ae6ea91be9661b9622504/Cython/Compiler/ModuleNode.py#L1442-L1456), to determine if an object for which tp_dealloc() is called has already been finalised or whether we have to do it. I'm not sure how to deal with this on our side now. Any clue?

I wrote PR 10626 to add _PyGC_FINALIZED() back to the C API.

My intent was only to remove _PyObject_GC_TRACK(o) and _PyObject_GC_UNTRACK(o) from the public C API.

I didn't expect that anyone would use _PyGC_FINALIZED() :-)

New changeset 3e21ad1a254cc33e8d4920ad7f026254ec728bee by Victor Stinner in branch 'master':
bpo-35081: Move _PyGC_FINALIZED() back to C API (GH-10626)
https://github.com/python/cpython/commit/3e21ad1a254cc33e8d4920ad7f026254ec728bee

Please review PR 10624 of bpo-35134: Create Include/unstable/ subdirectory. It's the second part of my plan :-)

New changeset bcda8f1d42a98d9022736dd52d855be8e220fe15 by Victor Stinner in branch 'master':
bpo-35081: Add Include/internal/pycore_object.h (GH-10640)
https://github.com/python/cpython/commit/bcda8f1d42a98d9022736dd52d855be8e220fe15

New changeset 984061eeb49c54fee901b92e5d3dde1c7a25cfa1 by Victor Stinner in branch 'master':
bpo-35081: Add new internal headers to Makefile (GH-10670)
https://github.com/python/cpython/commit/984061eeb49c54fee901b92e5d3dde1c7a25cfa1

New changeset 4ac5328affa37bdfc5847dfdb2a41bad772e7270 by Victor Stinner in branch 'master':
bpo-35081: add NEWS entry for new Include/internal/pycore_*.h files (GH-10666)
https://github.com/python/cpython/commit/4ac5328affa37bdfc5847dfdb2a41bad772e7270

I close the issue, it seems like all sub-tasks have been completed!

Summary of the change:

* Header files of Include/internal/ are now all prefixed by "pycore_" and the directory has been added to the header search path. For example, #include "pycore_pystate.h" is now used in C code, instead of #include "internal/pystate.h".
* All "#ifdef Py_BUILD_CORE" code has been moved from Include/*.h to Include/internal/*.h (except of the _PyTuple_ITEMS() macro)
* I created many new header files in Include/internal/
* _json and _locale modules are now built with Py_BUILD_CORE define
* _PyObject_GC_TRACK() and _PyObject_GC_UNTRACK() have been removed from the public C API and moved to the internal API

There are other issues to track following steps:

* bpo-35296: Install Include/internal/ header files
* bpo-35258: Fix -Wmissing-prototypes warnings
* bpo-35134: Create Include/cpython/ subdirectory
* bpo-35059: Convert macros to static inline functions

In bpo-35296, I modified "make install" to also install Include/internal/.

New changeset 8ac6539d85b481fc6b5e9145446b07e591b2caba by Victor Stinner in branch 'master':
bpo-35081: Add _PyTuple_CAST() (GH-10704)
https://github.com/python/cpython/commit/8ac6539d85b481fc6b5e9145446b07e591b2caba

New changeset ec13b9322d95a651606219469fc7b7e9c977f248 by Victor Stinner in branch 'master':
bpo-35081: Add Include/internal/pycore_tupleobject.h (GH-10705)
https://github.com/python/cpython/commit/ec13b9322d95a651606219469fc7b7e9c977f248

New changeset 45876a90e2663f12b90c2090ec3e48bd97841aae by Victor Stinner in branch 'master':
bpo-35081: Move bytes_methods.h to the internal C API (GH-18492)
https://github.com/python/cpython/commit/45876a90e2663f12b90c2090ec3e48bd97841aae

New changeset e9e7d284c434768333fdfb53a3663eae74cb995a by Victor Stinner in branch 'master':
bpo-35081: Move dtoa.h header to the internal C API (GH-18489)
https://github.com/python/cpython/commit/e9e7d284c434768333fdfb53a3663eae74cb995a

I reopen the issue since there is still activity on it.

New changeset 063abd931f064a4b6b478b0b6e9aa13ee38d2cff by Victor Stinner in branch 'main':
bpo-35081: Move interpreteridobject.h to Include/internal/ (GH-28969)
https://github.com/python/cpython/commit/063abd931f064a4b6b478b0b6e9aa13ee38d2cff

On Fri, Oct 15, 2021 at 3:56 AM STINNER Victor <report@bugs.python.org> wrote:
> New changeset 063abd931f064a4b6b478b0b6e9aa13ee38d2cff by Victor Stinner in branch 'main':
> bpo-35081: Move interpreteridobject.h to Include/internal/ (GH-28969)

FYI, _xxsubinterpretersmodule is supposed to be an extension module
(not builtin), so it's not meant to use internal APIs.  At some point
that happened and I only noticed now.  I've created bpo-45486 to
address that.