This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author grahamd
Recipients Rhamphoryncus, fdirosa, grahamd, nnorwitz, timbishop, vslavik
Date 2008-07-21.14:03:23
SpamBayes Score 3.5251363e-10
Marked as misclassified No
Message-id <1216649009.93.0.669858379886.issue1758146@psf.upfronthosting.co.za>
In-reply-to
Content
I know the discussions more or less says this, but I want to add some 
additional information.

For the record, the reason that mod_python crashes with 'Invalid thread state 
for this thread' when Py_DEBUG is defined in part relates to:

  http://issues.apache.org/jira/browse/MODPYTHON-217

Also, that Py_DEBUG check effectively says that if you use simplified GIL API 
for a particular thread against the first interpreter, you are prohibited from 
creating additional thread states for that thread. I haven't checked the 
documentation lately, but I am not sure it is really clear on that specific 
point and so in some respects the documentation may be at fault here. Someone 
might like to point to exact part of documentation which states this 
requirement.

The problem thus is that code which worked prior to Python 2.3 would still work 
with Python 2.3 and later, up to the point that some code decided to use the 
simplified GIL API. At that point Python would create its own internal thread 
state for that thread even if user code had already created one. Conversely, if 
the simplified GIL API was used against the thread first and then user code 
tried to create an additional thread state for that thread against first 
interpreter.

With Py_DEBUG defined, this scenario causes the assertion failure and the above 
error. Without Py_DEBUG defined, the code can quite happily run fine, at least 
until the point where code which left Python using a user thread state object 
attempts to reenter Python by using simplified GIL API. At that point it would 
deadlock.

Now, as I said, that one was effectively forced to use simplified GIL API for 
first interpreter with Python 2.3 probably wasn't at all clear and so 
mod_python was never updated to meet that requirement. As per the JIRA issue 
referenced above it is a known problem that code isn't meeting this 
requirement, but not much development has been done on mod_python for quite a 
while.

I have though recently made changes to personal copy of mod_python code such 
that it uses simplified GIL API for all access against first interpreter and it 
no longer suffers that assertion failure when Py_DEBUG defined. The code also 
should work for any modules which use simplified GIL API, such as SWIG 
generated bindings for Xapian. You do have to force the application using such 
modules to run under first interpreter.

The code for mod_wsgi uses simplified GIL API for first interpreter as well and 
works with SWIG generated bindings, but it is possible that it may still fail 
that assertion when Py_DEBUG is defined. This is because in order to allow 
mod_python and mod_wsgi to be used in Apache at the same time, mod_wsgi had to 
incorporate some hacks to workaround the fact that mod_python was not using 
simplified GIL API for first interpreter, but also because mod_python wasn't 
releasing the GIL for a critical section between when it was initialised and 
Apache child processes were created. It was in this section that mod_wsgi has 
to initialise itself and so it had to fiddle the thread states to be able to do 
its things. This workaround may have been enough to create additional thread 
state of a thread for first interpreter, thus later triggering the assertion.

It would have been nice to have mod_wsgi do the correct thing from the start, 
but that would have barred it being used at same time as mod_python and so 
people may have baulked at trying mod_wsgi as a result. Now that mod_wsgi has 
got some traction, in mod_wsgi version 3.0 it will be changed to remove the 
mod_python fiddle. This will mean that mod_wsgi 3.0 will not be usable at same 
time as current mod_python versions and would only be usable with the 
mod_python version (maybe 3.4) which I have made modifications for to also use 
simplified GIL APIs properly.

So that is the state of play as I see and understand it.

As to Adam's comments about use cases for multiple interpreters, we have had 
that discussion before and despite that many people rely on that feature in 
both mod_python and mod_wsgi he still continues to dismiss it outright and 
instead calls for complete removal of the feature.

Also Adam's comments that multiple interpreters were used in mod_wsgi only to 
support buggy third party software, that is untrue. Multiple interpreter 
support exists in mod_wsgi because mod_python provided a similar feature and 
mod_python existed before many of the Python web applications which are claimed 
to be the reason that sub interpreters are used in the first place. So, 
mod_python and use of sub interpreters came first, and not really the other way 
around. Where Python web applications do rely on os.environ it is historically 
because that is how things were done in CGI. Many such as Trac may still 
support that means of configuration as a fall back, but Trac now also supports 
other ways which are thread safe.

All up, use of distinct sub interpreters in mod_python and mod_wsgi is a valid 
concept in web hosting, one reason being because of the fact that it is simpler 
to use one Apache instance than many. Despite claims that sub interpreter 
support is completely broken it works fine for that use case because of the 
nature of web applications and the transient nature of individual requests and 
how sub interpreters persist for the life of the process and are not destroyed 
within the lifetime of the process.

Unfortunately some are quite blinkered to this and haven't taken the time to 
understand how it is being used. This is quite unfortunate, and rather than 
ripping out multiple interpreters, time might be better spent on improving it 
to work better as necessary if certain areas are still a problem.
History
Date User Action Args
2008-07-21 14:03:30grahamdsetspambayes_score: 3.52514e-10 -> 3.5251363e-10
recipients: + grahamd, nnorwitz, Rhamphoryncus, timbishop, vslavik, fdirosa
2008-07-21 14:03:29grahamdsetspambayes_score: 3.52514e-10 -> 3.52514e-10
messageid: <1216649009.93.0.669858379886.issue1758146@psf.upfronthosting.co.za>
2008-07-21 14:03:29grahamdlinkissue1758146 messages
2008-07-21 14:03:24grahamdcreate