New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Hang in lib/test/test_threading.py #53045
Comments
The tests for the ConditionVariable a fragile. On a slow computer, or when running a DEBUG build, the main thread can issue a condition.notify() call, even though the worker threads have not settled in on a wait() call. This causes the test to stall. |
Strange, I have never noticed this (even on the buildbots). |
Well, I saw this occasionally on my dual core laptop when working on the RWLock. A notify() before a target does a wait() is ineffective and it would wait indefinitely after the notify(5) call. This isn't a "proper" fix (I had another, more complex, where we would keep a counter for the number of successful "waiters" and wait for that to reach the target value) but it is in the spirit of many of these tests, where we rely on _wait() to be sufficient to reach a certain state. |
Here is a new patch.
|
What do you mean? The implementation doesn't seem prone to spurious wakeups, and the docs don't say so either.
Why would you build a Lock out of a Condition? |
We shouldn't be testing implementation details. The Condition variables are canonically prone to "spurious wakeups" and "stolen wakeups". The fact that the current implementation doesn't have them shouldn't place that burden on future implementations of threading.Condition on this platform.
I could leave the original test in place and just add the new ones, but the old test is at least provably a subject to a race condition as per the original issue. |
No, they aren't. Just because POSIX says they are doesn't mean *our*
Ah, thanks, indeed. Except that... |
Stolen wakeups are a fact of life though, even in cpython's implementation of condition variables. And for the user of a condition variable the difference is so subtle as to not warrant special mention. This is, btw, not just a posix thing. It is same on windows, java, and in fact, comes from the original definition of a condition variable as part of the "monitor" pattern. Condition variables are defined have this caveat to allow for flexibility in their implementation. The canonical and correct way to use a condition variable is to repeatedly wait() and verify that the wakeup condition holds. The wait_for() method abstracts this into a convenient helper. There is no need to befuddle the issue by special casing the python version as being guaranteed to not have suprious wakeups, but otherwise do have the stolen thread problem. Quite the contrary, we should be careful to never overspecify our interfaces so that we may not change their ipmlementation details later. A good summary of the situation and the reason for the specification can be found here: The comment I added to the documentation clarifies the reason why other examples in the docs were using while loops and teaches shows people how to use these constructs in a robust manner. Removing it is a bad idea and I think you will find others will agree. Take it up on python-dev if you want. Unless you disagree, I'll add these test cases in addition to the broken one, which can then be fixed or removed later if it turns out to be problematic to other people. |
I agree with Antoine. However, while I think the spurious wakeups reference could be |
I don't think it's a subtle difference. Being woken up too late is not
Thanks, will take a look.
The interface is already, de facto, specified by its one and only There's probably code out there that relies on wait() not returning
So the reason appears to be:
Python is a high-level language, so making condition variable ops What follows is even less enlightening:
Teaching people to use loops is fine. But there's no need to mislead
Really, can you stop being aggressive? You haven't proved that test is If you don't agree with what is being tested, that's another matter. But |
I'm proposing the following changes to the threading docs. Most of them are cleanups and small improvements, but I also rewrote the offending paragraph, and made the wait_for example more proeminent. |
Looks good to me. |
New changeset 2f51dca92883 by Antoine Pitrou in branch '3.2': |
Ok, I've fixed the docs in 3.2 and 3.3. |
A few comments:
The ``while`` loop checking for the application's condition is necessary |
Did I forget this? Ow.
Ok.
But, once again, "the condition may not yet hold true" is false. |
This is also why we, IMHO, shouldn't rely on this behaviour in the unittests. One should never write tests that depend on unspecified behaviour, again, good engineering practice. |
Le mercredi 11 avril 2012 à 10:50 +0000, Kristján Valur Jónsson a
No, it is not "left undefined". If the documentation doesn't say
Which options?
Disagreed. Unit tests should definitely protect against the introduction If you think the condition variable specification should be changed, you |
[...]
One can imagine, for example, that another implementation (or maybe |
Le mercredi 11 avril 2012 à 11:32 +0000, Charles-François Natali a
That's right, but spurious wakeups would be a good argument against That said, we couldn't use the POSIX implementation anyway, since our |
Ok, doc improved in 9d4109af8f3b. |
LGTM. Kristján, how about updating your patch to only fix the original |
Ah, here we are some 18 months later. Let me have another go. |
Here is an updated patch that only adds the necessary safety delay to the main thread. It also explains these in the comments. |
New changeset 9e48c9538a7f by Kristjan Valur Jonsson in branch 'default': |
The race condition is not dead, at least in Python 2.7 :-) I backported the change for bpo-30727: "[2.7] test_threading.ConditionTests.test_notify() hangs randomly on Python 2.7". |
This issue discussed different things:
All discussed issues have been fixed, so I close the issue. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: