CPython's C code makes extensive use of global variables.  The globals fall into one of several categories:

* (effectively) constants (incl. static types)
* freelists, caches, and counters
* used exclusively in main or in REPL
* process-global state
* module state
* Python runtime state

Those in the last category are not explicitly organized nor easily discoverable.  Among other things, this has an impact on efforts that change the runtime (e.g. my multi-core Python project).  To improve the situation I'd like to do the following:

1. group the (stateful) runtime globals into various topical structs
2. consolidate the topical structs under a single top-level _PyRuntimeState struct
3. add a check-c-globals.py script that helps identify runtime globals

One side effect of consolidating these globals is a significant performance improvement of CPython.  Presumably this is due to the caching behavior of a single struct vs. that of dozens of separate globals.

I have a patch and will put up a PR momentarily.

One thing I'd like to also try is to use a freelist embedded in _PyRuntimeState for the PyInterpreterState and PyThreadState linked lists.

See also issue29881.

After looking at the PR, I'm a bit skeptical about this.  Suddenly a lot of things which are implementation details get moved to the public include files, which makes things more confusing from my POV.  I also don't know why all globals should be consolidated in a single place (it is not a widespread policy in other projects AFAIK, but I may be mistaken).

Perhaps it would be more reasonable to consolidate the globals in each .c file independently, without trying to join all of them together in one place nor expose them in the public include files.

The core motivation driving the original refactoring was to better understand and consolidate our runtime state in order to clarify what the GIL is actually protecting (aside from the reference counts). That then turned out to have surprising performance benefits (presumably by way of loading all of this state permanently into the CPU cache and then keeping it there).

The changes to the public include files arise more from the fact we don't currently have a common place to put internal CPython headers for declarations we want to share across compilation modules.

I agree the latter is confusing, and it could be worth having a separate "Include/internal/CPython.h" header that outright failed the build if you attempted to include it without Py_BUILD_CORE set.

(This also ties into my comments on the PR, where I believe the current approach of mixing public and private state in the same structs will cause problems with compilers generating incorrect field offsets in extension modules and embedding applications)

New changeset 76d5abc8684bac4f2fc7cccfe2cd940923357351 by Eric Snow in branch 'master':
bpo-30860: Consolidate stateful runtime globals. (#2594)
https://github.com/python/cpython/commit/76d5abc8684bac4f2fc7cccfe2cd940923357351

Sadly, Windows doesn't like much your big change.

Just one example:

     2>C:\buildbot.python.org\3.x.kloth-win64\build\Include\Python-ast.h(563): warning C4005: 'Yield': macro redefinition [C:\buildbot.python.org\3.x.kloth-win64\build\PCbuild\pythoncore.vcxproj]
         C:\Program Files (x86)\Windows Kits\8.1\Include\um\winbase.h(97): note: see previous definition of 'Yield'
         bltinmodule.c

http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x/builds/954/steps/compile/logs/stdio

I'm also a bit skeptical about the header design in this PR. We should not to have a monolithic _Python.h header, and it should not be sometimes included in Python.h. Rather, code that needs the internal headers should include them directly. In fact, this PR regresses that by moving all the internal headers into Include/...

New changeset 05351c1bd8b70d1878527762174cdaaba3572395 by Eric Snow in branch 'master':
Revert "bpo-30860: Consolidate stateful runtime globals." (#3379)
https://github.com/python/cpython/commit/05351c1bd8b70d1878527762174cdaaba3572395

Regarding the accidental exposure of _Py_CheckRecursionLimit, the main active usage of the stable ABI that we're aware of is Riverbank's C/C++ binding generator for PyQt: http://pyqt.sourceforge.net/Docs/sip4/directives.html#directive-%Module (see the use_limited_api option for the linked directive)

I checked with Phil Thomson (sip's maintainer), and the bindings sip generates don't use either of the public macros that access this nominally private value.

New changeset 2ebc5ce42a8a9e047e790aefbf9a94811569b2b6 by Eric Snow in branch 'master':
bpo-30860: Consolidate stateful runtime globals. (#3397)
https://github.com/python/cpython/commit/2ebc5ce42a8a9e047e790aefbf9a94811569b2b6

FYI, the merged code introduced a bunch of new warnings on Windows.  I'm looking into it.

The commit 2ebc5ce42a8a9e047e790aefbf9a94811569b2b6 introduced reference leaks: see bpo-31420 which tracks them.

New changeset 4866957e86594935ec2e6434b6e470ebeb0c79b4 by Victor Stinner in branch 'master':
bpo-30860: Add Include/internal/ in "make tags" (#3498)
https://github.com/python/cpython/commit/4866957e86594935ec2e6434b6e470ebeb0c79b4

New changeset ba6d5d1defd7a281c8c8804e4b4cfd7370886236 by Eric Snow in branch 'master':
bpo-30860: Always provide serialno. (#3507)
https://github.com/python/cpython/commit/ba6d5d1defd7a281c8c8804e4b4cfd7370886236

New changeset 8728018624f257c7cfe44014742ae46134047f49 by Victor Stinner (Eric Snow) in branch 'master':
bpo-30860: Fix a refleak. (#3506)
https://github.com/python/cpython/commit/8728018624f257c7cfe44014742ae46134047f49

New changeset fc1bf872e9d31f3e837f686210f94e57ad3d6582 by Eric Snow in branch 'master':
bpo-30860: Move windows.h include out of internal/*.h. (#3458)
https://github.com/python/cpython/commit/fc1bf872e9d31f3e837f686210f94e57ad3d6582

New changeset dae0276bb6bc7281d59fb0b8f1aab31634ee80dc by Eric Snow in branch 'master':
bpo-30860: Fix a refleak. (#3567)
https://github.com/python/cpython/commit/dae0276bb6bc7281d59fb0b8f1aab31634ee80dc

New changeset ccb3c7654cc3d031fb93bf443a6ef9cfb11f6b43 by Victor Stinner in branch 'master':
bpo-30860: Fix deadcode in obmalloc.c (#3499)
https://github.com/python/cpython/commit/ccb3c7654cc3d031fb93bf443a6ef9cfb11f6b43

New changeset 9e87e7776f7ace66baaf7247233afdabd00c2b44 by Victor Stinner in branch 'master':
bpo-32096: Remove obj and mem from _PyRuntime (#4532)
https://github.com/python/cpython/commit/9e87e7776f7ace66baaf7247233afdabd00c2b44