classification
Title: test_threading_handled() and test_threading_not_handled() of test_socketserver hangs randomly on AMD64 FreeBSD 10.x Shared 3.6
Type: Stage:
Components: Tests Versions: Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: haypo, martin.panter
Priority: normal Keywords: buildbot

Created on 2017-05-17 19:07 by haypo, last changed 2017-08-18 23:25 by haypo.

Messages (7)
msg293879 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-05-17 19:07
http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.x%20Shared%203.6/builds/142/steps/test/logs/stdio

...
0:10:10 [396/405] test_pulldom passed -- running: test_socketserver (487 sec)
0:10:11 [397/405] test_dictcomps passed -- running: test_socketserver (488 sec)
0:10:11 [398/405] test_support passed -- running: test_socketserver (488 sec)
0:10:22 [399/405] test_email passed -- running: test_socketserver (499 sec)
0:10:44 [400/405] test_buffer passed -- running: test_socketserver (521 sec), test_concurrent_futures (40 sec), test_datetime (49 sec)
0:10:53 [401/405] test_ftplib passed -- running: test_socketserver (530 sec), test_concurrent_futures (49 sec), test_datetime (58 sec)
0:10:55 [402/405] test_ucn passed -- running: test_socketserver (532 sec), test_concurrent_futures (51 sec), test_datetime (60 sec)
fetching http://www.pythontest.net/unicode/9.0.0/NamedSequences.txt ...
0:11:09 [403/405] test_concurrent_futures passed (65 sec) -- running: test_socketserver (546 sec), test_datetime (74 sec)
0:11:16 [404/405] test_datetime passed (81 sec) -- running: test_socketserver (553 sec)
running: test_socketserver (583 sec)
running: test_socketserver (613 sec)
running: test_socketserver (643 sec)
running: test_socketserver (673 sec)
running: test_socketserver (703 sec)
running: test_socketserver (733 sec)
running: test_socketserver (763 sec)
running: test_socketserver (793 sec)
running: test_socketserver (823 sec)
running: test_socketserver (853 sec)
running: test_socketserver (883 sec)
0:17:04 [405/405] test_socketserver crashed
Timeout (0:15:00)!
Thread 0x0000000802006400 (most recent call first):
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/threading.py", line 295 in wait
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/threading.py", line 551 in wait
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/test_socketserver.py", line 369 in wait_done
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/test_socketserver.py", line 341 in __init__
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/test_socketserver.py", line 362 in __init__
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/test_socketserver.py", line 307 in test_threading_handled
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/unittest/case.py", line 601 in run
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/unittest/case.py", line 649 in __call__
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/unittest/suite.py", line 122 in run
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/unittest/suite.py", line 84 in __call__
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/unittest/suite.py", line 122 in run
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/unittest/suite.py", line 84 in __call__
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/unittest/suite.py", line 122 in run
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/unittest/suite.py", line 84 in __call__
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/unittest/runner.py", line 176 in run
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/support/__init__.py", line 1889 in _run_suite
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/support/__init__.py", line 1923 in run_unittest
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/runtest.py", line 164 in test_runner
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/runtest.py", line 165 in runtest_inner
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/runtest.py", line 119 in runtest
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/runtest_mp.py", line 71 in run_tests_slave
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/main.py", line 475 in _main
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/main.py", line 468 in main
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/main.py", line 532 in main
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/regrtest.py", line 46 in _main
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/regrtest.py", line 50 in <module>
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/runpy.py", line 85 in _run_code
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/runpy.py", line 193 in _run_module_as_main
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/__main__.py", line 2, in <module>
    main()
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/main.py", line 532, in main
    Regrtest().main(tests=tests, **kwargs)
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/main.py", line 468, in main
    self._main(tests, kwargs)
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/main.py", line 490, in _main
    self.run_tests()
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/main.py", line 416, in run_tests
    run_tests_multiprocess(self)
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/runtest_mp.py", line 221, in run_tests_multiprocess
    raise Exception(msg)
Exception: Child error on test_socketserver: Exit code 1
*** Error code 1

Stop.
make: stopped in /usr/home/buildbot/python/3.6.koobs-freebsd10/build
program finished with exit code 1
elapsedTime=1026.162135
msg295904 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-06-13 11:29
Similar issue: bpo-30652, test_threading_not_handled() of test_socketserver hangs randomly on AMD64 FreeBSD 10.x Shared 3.6.
msg297422 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-06-30 15:35
http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.x%20Shared%203.x/builds/525/steps/test/logs/stdio

0:16:07 load avg: 0.35 [406/406/1] test_socketserver crashed (Exit code 1)
Timeout (0:15:00)!
Thread 0x0000000802006400 (most recent call first):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/threading.py", line 296 in wait
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/threading.py", line 552 in wait
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/test_socketserver.py", line 369 in wait_done
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/test_socketserver.py", line 341 in __init__
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/test_socketserver.py", line 362 in __init__
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/test_socketserver.py", line 307 in test_threading_handled
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/unittest/case.py", line 615 in run
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/unittest/case.py", line 663 in __call__
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/unittest/suite.py", line 122 in run
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/unittest/suite.py", line 84 in __call__
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/unittest/suite.py", line 122 in run
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/unittest/suite.py", line 84 in __call__
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/unittest/suite.py", line 122 in run
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/unittest/suite.py", line 84 in __call__
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/unittest/runner.py", line 176 in run
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/support/__init__.py", line 1896 in _run_suite
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/support/__init__.py", line 1940 in run_unittest
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/libregrtest/runtest.py", line 171 in test_runner
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/libregrtest/runtest.py", line 172 in runtest_inner
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/libregrtest/runtest.py", line 130 in runtest
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/libregrtest/runtest_mp.py", line 71 in run_tests_slave
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/libregrtest/main.py", line 517 in _main
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/libregrtest/main.py", line 510 in main
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/libregrtest/main.py", line 585 in main
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/regrtest.py", line 46 in _main
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/regrtest.py", line 50 in <module>
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/runpy.py", line 85 in _run_code
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/runpy.py", line 193 in _run_module_as_main
msg297464 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2017-07-01 01:26
These tests are supposed to:

1. Create a TCP server
2. Open a TCP connection
3. Run a custom connection handler (depending on the particular test) in a new thread
4. When the handler returns, the new thread should call “shutdown_request”
5. Shutdown_request closes the server’s connection socket
6. Shutdown_request sets an event for the main thread
7. Main thread waits for the above event

The stack trace indicates a thread is stuck at step 7. My guess is that step 5 has raised an exception, killing the thread rather than continuing to step 6. I suspect it is a “socket.close” call raising an asynchronous error such as ECONNRESET; see Issue 30319. A general fix for that problem might fix these test_socketserver hangs.
msg297666 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-07-04 14:56
I marked bpo-30652 as duplicate of this issue. Copy of the first message msg295903:

http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.x%20Shared%203.6/builds/218/steps/test/logs/stdio

0:16:16 [405/405] test_socketserver crashed
Timeout (0:15:00)!
Thread 0x0000000802006400 (most recent call first):
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/threading.py", line 295 in wait
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/threading.py", line 551 in wait
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/test_socketserver.py", line 369 in wait_done
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/test_socketserver.py", line 341 in __init__
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/test_socketserver.py", line 362 in __init__
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/test_socketserver.py", line 312 in test_threading_not_handled
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/unittest/case.py", line 601 in run
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/unittest/case.py", line 649 in __call__
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/unittest/suite.py", line 122 in run
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/unittest/suite.py", line 84 in __call__
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/unittest/suite.py", line 122 in run
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/unittest/suite.py", line 84 in __call__
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/unittest/suite.py", line 122 in run
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/unittest/suite.py", line 84 in __call__
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/unittest/runner.py", line 176 in run
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/support/__init__.py", line 1889 in _run_suite
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/support/__init__.py", line 1923 in run_unittest
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/runtest.py", line 164 in test_runner
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/runtest.py", line 165 in runtest_inner
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/runtest.py", line 119 in runtest
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/runtest_mp.py", line 71 in run_tests_slave
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/main.py", line 475 in _main
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/main.py", line 468 in main
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/main.py", line 532 in main
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/regrtest.py", line 46 in _main
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/regrtest.py", line 50 in <module>
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/runpy.py", line 85 in _run_code
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/runpy.py", line 193 in _run_module_as_main
msg300142 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-08-11 00:06
I didn't see this bug recently, I hesitate to close this issue.
msg300551 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-08-18 23:25
I recently found two issues in socketserver:

* bpo-31233: socketserver.ThreadingMixIn leaks running threads after server_close()
* bpo-31151: socketserver.ForkingMixIn.server_close() leaks zombie processes

Fixing these two issues is likely to help to fix this one.
History
Date User Action Args
2017-08-18 23:25:34hayposetmessages: + msg300551
2017-08-11 00:06:11hayposetmessages: + msg300142
2017-07-04 14:56:25hayposetmessages: + msg297666
title: test_threading_handled() of test_socketserver hangs randomly on AMD64 FreeBSD 10.x Shared 3.6 -> test_threading_handled() and test_threading_not_handled() of test_socketserver hangs randomly on AMD64 FreeBSD 10.x Shared 3.6
2017-07-04 14:55:28haypolinkissue30652 superseder
2017-07-04 14:54:42haypounlinkissue30319 dependencies
2017-07-01 01:26:22martin.pantersetnosy: + martin.panter
messages: + msg297464
2017-07-01 00:49:49martin.panterlinkissue30319 dependencies
2017-06-30 15:35:13hayposetmessages: + msg297422
2017-06-13 11:30:24hayposettitle: test_socketserver killed after 15 min on AMD64 FreeBSD 10.x Shared 3.6 -> test_threading_handled() of test_socketserver hangs randomly on AMD64 FreeBSD 10.x Shared 3.6
2017-06-13 11:29:59hayposetmessages: + msg295904
2017-05-17 19:07:04haypocreate