classification
Title: Join started threads in tests
Type: enhancement Stage: resolved
Components: Tests Versions: Python 3.5, Python 3.4, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: ezio.melotti, haypo, michael.foord, pitrou, python-dev, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2015-03-28 20:10 by serhiy.storchaka, last changed 2015-04-04 06:53 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
test_support_start_threads.patch serhiy.storchaka, 2015-03-28 20:10
test_support_start_threads_2.patch serhiy.storchaka, 2015-03-30 12:28 review
Messages (4)
msg239465 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-03-28 20:10
When a test starts many threads in a loop, there is a chance that the starting thread will fail due to lack of memory. In this case all started threads left dangling and make MemotyError even more probably. They also can provoke dim error messages in this and in following tests. Proposed patch adds new helper function test.support.start_threads() that starts threads and then join started threads.

Some tests even hang without this patch.
msg239583 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-03-30 08:48
> In this case all started threads left dangling and make MemotyError even more probably

Oh, I didn't know that. Good catch.

test_support_start_threads.patch looks good to me and it looks like a nice enhancement.

But would it be possible to emit a warning if a thread hangs?

For example, try to join with a timeout of 60 seconds, and then display a warning, and retry with a timeout of 120 seconds, then display a second warning, etc.

Maybe we may even give up after a timeout (ex: 5 minutes, or 15 minutes, it's up to you). An exception must be raised in this case. You can use faulthandler.dump_traceback() to display the traceback of all threads in this case.

I hope that my concern is more theorical, but it would reduce the duration of tests when the test suite is stuck (which is quite common), and so being able to test more changesets per day.

--

I noticed that under low memory condition, some tests hang because Python doesn't handle notify the "parent thread" that the "child thread" fails to start. So the parent thread is stuck and nothing happens. Well, I'm not sure that it's related to this issue. At least, test_support_start_threads.patch doesn't help.
msg239601 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-03-30 12:28
Something like this?
msg239780 - (view) Author: Roundup Robot (python-dev) Date: 2015-04-01 10:07
New changeset f8358f6e50e7 by Serhiy Storchaka in branch '2.7':
Issue #23799: Added test.test_support.start_threads() for running and
https://hg.python.org/cpython/rev/f8358f6e50e7

New changeset 8987218adc99 by Serhiy Storchaka in branch '3.4':
Issue #23799: Added test.support.start_threads() for running and cleaning up
https://hg.python.org/cpython/rev/8987218adc99

New changeset a2df4baa112b by Serhiy Storchaka in branch 'default':
Issue #23799: Added test.support.start_threads() for running and cleaning up
https://hg.python.org/cpython/rev/a2df4baa112b
History
Date User Action Args
2015-04-04 06:53:34serhiy.storchakasetstatus: open -> closed
assignee: serhiy.storchaka
resolution: fixed
stage: patch review -> resolved
2015-04-01 10:07:26python-devsetnosy: + python-dev
messages: + msg239780
2015-03-30 12:28:58serhiy.storchakasetfiles: + test_support_start_threads_2.patch

messages: + msg239601
2015-03-30 08:48:22hayposetmessages: + msg239583
2015-03-30 07:06:37serhiy.storchakasetnosy: + pitrou, haypo, ezio.melotti, michael.foord
2015-03-28 20:10:30serhiy.storchakacreate