Nick, I think you are making the wrong assumption that an external threads will only ever call into the same interpreter. This is not the case. In mod_wsgi and mod_python there is a pool of external threads that for distinct HTTP requests, delegated to a specific thread, can make calls into different interpreters. This is all fine so long as you ensure that for each thread, it uses a distinct thread state for that thread for each interpreter. In other words, you cant use the same thread state instance across multiple interpreters as it is bound to a specific interpreter.

This is because autoInterpreterState is always going to be set to the main interpreter. This means that when the thread is calling into a new sub interpreter it will either inherit via current GIL state API an existing thread state bound to the main interpreter, or if one is created, will still get bound to the main interpreter. As soon as you start using a thread state bound to one interpreter against another, problems start occurring.

After thinking about this all some more I believe now what is needed is a mix of the TLS idea for current interpreter state that I am suggesting and in part the extended GIL state functions that Antoine describes.

So, the TLS records what interpreter a thread is currently running against so that GIL state APIs work for existing unmodified extension modules. At the same time though, you still need a way of switching what interpreter a thread is running against. For the latter, various of the thread state related functions that exist already could do this automatically. In some cases you will still need the extended function for acquisition that Antoine suggested.

Consider a few scenarios of usage.

First off, when an external thread calls PyInterpreter_New(), it creates a new thread state object against that new sub interpreter automatically and returns it. With this new systems, it would also automatically update the TLS for the current thread to be that new interpreter also. That way when it calls into Python which then calls back out to code which releases the GIL and then calls back in through PyGILState_Ensure(), with no arguments, it will work. This obviously implies though that PyGILState_Ensure() makes use of the TLS for the interpreter being used and isn't hard wired to the main interpreter like it is now.

Second, consider some of the other API functions such as PyThreadState_Swap(). When passing it a non NULL pointer, you are giving it a thread state object which is already bound to an interpreter. It thus can also update the TLS for the interpreter automatically. If you pass it a NULL then it clears the TLS with all functions later that rely on that TLS asserting that it is not NULL when used. Another similar case where TLS can be auto updated is functions which clear/delete an interpreter state and leave GIL unlocked at the end. These also would clear the TLS.

So, it is possible that that no new API functions may be needed to manage the TLS for what interpreter is associated with the current thread, as I thought, as existing API functions can do that management themselves transparently.

The third and final scenario, and the one where the extended GIL state functions for Ensure is still required, is where code doesn't have the GIL as yet and wants to make a call into sub interpreter rather than the main interpreter, where it already has a pointer to the sub interpreter and nothing more. In this case the new PyGILState_EnsureEx() function is used, with the sub interpreter being passed as argument.

The beauty of existing API functions of PyThreadState_Swap() etc managing the TLS for the interpreter is that the only code that needs to change is the embedded systems which are creating and using multiple interpreters in the first place. In other words, mod_wsgi would need to change, with it simply replacing all the equivalent stuff it already has for doing what PyGILState_??? functions do now but against sub interpreters. If I am right, all extension modules that don't really care about whether sub interpreters are being used should work without modification.

Oh, and I also think you probably don't need PyGILState_ReleaseEx() if all made TLS aware, just the single PyGILState_EnsureEx() is needed.