Message70109
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Author | grahamd |
---|---|
Recipients | Rhamphoryncus, fdirosa, grahamd, nnorwitz, timbishop, vslavik |
Date | 2008-07-21.14:03:23 |
SpamBayes Score | 3.5251363e-10 |
Marked as misclassified | No |
Message-id | <1216649009.93.0.669858379886.issue1758146@psf.upfronthosting.co.za> |
In-reply-to |
Content | |
---|---|
I know the discussions more or less says this, but I want to add some additional information. For the record, the reason that mod_python crashes with 'Invalid thread state for this thread' when Py_DEBUG is defined in part relates to: http://issues.apache.org/jira/browse/MODPYTHON-217 Also, that Py_DEBUG check effectively says that if you use simplified GIL API for a particular thread against the first interpreter, you are prohibited from creating additional thread states for that thread. I haven't checked the documentation lately, but I am not sure it is really clear on that specific point and so in some respects the documentation may be at fault here. Someone might like to point to exact part of documentation which states this requirement. The problem thus is that code which worked prior to Python 2.3 would still work with Python 2.3 and later, up to the point that some code decided to use the simplified GIL API. At that point Python would create its own internal thread state for that thread even if user code had already created one. Conversely, if the simplified GIL API was used against the thread first and then user code tried to create an additional thread state for that thread against first interpreter. With Py_DEBUG defined, this scenario causes the assertion failure and the above error. Without Py_DEBUG defined, the code can quite happily run fine, at least until the point where code which left Python using a user thread state object attempts to reenter Python by using simplified GIL API. At that point it would deadlock. Now, as I said, that one was effectively forced to use simplified GIL API for first interpreter with Python 2.3 probably wasn't at all clear and so mod_python was never updated to meet that requirement. As per the JIRA issue referenced above it is a known problem that code isn't meeting this requirement, but not much development has been done on mod_python for quite a while. I have though recently made changes to personal copy of mod_python code such that it uses simplified GIL API for all access against first interpreter and it no longer suffers that assertion failure when Py_DEBUG defined. The code also should work for any modules which use simplified GIL API, such as SWIG generated bindings for Xapian. You do have to force the application using such modules to run under first interpreter. The code for mod_wsgi uses simplified GIL API for first interpreter as well and works with SWIG generated bindings, but it is possible that it may still fail that assertion when Py_DEBUG is defined. This is because in order to allow mod_python and mod_wsgi to be used in Apache at the same time, mod_wsgi had to incorporate some hacks to workaround the fact that mod_python was not using simplified GIL API for first interpreter, but also because mod_python wasn't releasing the GIL for a critical section between when it was initialised and Apache child processes were created. It was in this section that mod_wsgi has to initialise itself and so it had to fiddle the thread states to be able to do its things. This workaround may have been enough to create additional thread state of a thread for first interpreter, thus later triggering the assertion. It would have been nice to have mod_wsgi do the correct thing from the start, but that would have barred it being used at same time as mod_python and so people may have baulked at trying mod_wsgi as a result. Now that mod_wsgi has got some traction, in mod_wsgi version 3.0 it will be changed to remove the mod_python fiddle. This will mean that mod_wsgi 3.0 will not be usable at same time as current mod_python versions and would only be usable with the mod_python version (maybe 3.4) which I have made modifications for to also use simplified GIL APIs properly. So that is the state of play as I see and understand it. As to Adam's comments about use cases for multiple interpreters, we have had that discussion before and despite that many people rely on that feature in both mod_python and mod_wsgi he still continues to dismiss it outright and instead calls for complete removal of the feature. Also Adam's comments that multiple interpreters were used in mod_wsgi only to support buggy third party software, that is untrue. Multiple interpreter support exists in mod_wsgi because mod_python provided a similar feature and mod_python existed before many of the Python web applications which are claimed to be the reason that sub interpreters are used in the first place. So, mod_python and use of sub interpreters came first, and not really the other way around. Where Python web applications do rely on os.environ it is historically because that is how things were done in CGI. Many such as Trac may still support that means of configuration as a fall back, but Trac now also supports other ways which are thread safe. All up, use of distinct sub interpreters in mod_python and mod_wsgi is a valid concept in web hosting, one reason being because of the fact that it is simpler to use one Apache instance than many. Despite claims that sub interpreter support is completely broken it works fine for that use case because of the nature of web applications and the transient nature of individual requests and how sub interpreters persist for the life of the process and are not destroyed within the lifetime of the process. Unfortunately some are quite blinkered to this and haven't taken the time to understand how it is being used. This is quite unfortunate, and rather than ripping out multiple interpreters, time might be better spent on improving it to work better as necessary if certain areas are still a problem. |
History | |||
---|---|---|---|
Date | User | Action | Args |
2008-07-21 14:03:30 | grahamd | set | spambayes_score: 3.52514e-10 -> 3.5251363e-10 recipients: + grahamd, nnorwitz, Rhamphoryncus, timbishop, vslavik, fdirosa |
2008-07-21 14:03:29 | grahamd | set | spambayes_score: 3.52514e-10 -> 3.52514e-10 messageid: <1216649009.93.0.669858379886.issue1758146@psf.upfronthosting.co.za> |
2008-07-21 14:03:29 | grahamd | link | issue1758146 messages |
2008-07-21 14:03:24 | grahamd | create |