Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test_threading_handled() and test_threading_not_handled() of test_socketserver hangs randomly on AMD64 FreeBSD 10.x Shared 3.6 #74576

Closed
vstinner opened this issue May 17, 2017 · 9 comments
Closed

test_threading_handled() and test_threading_not_handled() of test_socketserver hangs randomly on AMD64 FreeBSD 10.x Shared 3.6 #74576

vstinner opened this issue May 17, 2017 · 9 comments
Labels
tests Tests in the Lib/test dir

Comments

@vstinner
Copy link
Member

BPO 30391
Nosy @vstinner, @vadmium

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-05-29.22:03:58.026>
created_at = <Date 2017-05-17.19:07:04.292>
labels = ['tests']
title = 'test_threading_handled() and test_threading_not_handled() of test_socketserver hangs randomly on AMD64 FreeBSD 10.x Shared 3.6'
updated_at = <Date 2018-05-29.22:03:58.026>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2018-05-29.22:03:58.026>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2018-05-29.22:03:58.026>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2017-05-17.19:07:04.292>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 30391
keywords = ['buildbot']
message_count = 9.0
messages = ['293879', '295904', '297422', '297464', '297666', '300142', '300551', '302266', '318120']
nosy_count = 2.0
nosy_names = ['vstinner', 'martin.panter']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue30391'
versions = ['Python 3.6']
@vstinner
Copy link
Member Author

http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.x%20Shared%203.6/builds/142/steps/test/logs/stdio

...
0:10:10 [396/405] test_pulldom passed -- running: test_socketserver (487 sec)
0:10:11 [397/405] test_dictcomps passed -- running: test_socketserver (488 sec)
0:10:11 [398/405] test_support passed -- running: test_socketserver (488 sec)
0:10:22 [399/405] test_email passed -- running: test_socketserver (499 sec)
0:10:44 [400/405] test_buffer passed -- running: test_socketserver (521 sec), test_concurrent_futures (40 sec), test_datetime (49 sec)
0:10:53 [401/405] test_ftplib passed -- running: test_socketserver (530 sec), test_concurrent_futures (49 sec), test_datetime (58 sec)
0:10:55 [402/405] test_ucn passed -- running: test_socketserver (532 sec), test_concurrent_futures (51 sec), test_datetime (60 sec)
fetching http://www.pythontest.net/unicode/9.0.0/NamedSequences.txt ...
0:11:09 [403/405] test_concurrent_futures passed (65 sec) -- running: test_socketserver (546 sec), test_datetime (74 sec)
0:11:16 [404/405] test_datetime passed (81 sec) -- running: test_socketserver (553 sec)
running: test_socketserver (583 sec)
running: test_socketserver (613 sec)
running: test_socketserver (643 sec)
running: test_socketserver (673 sec)
running: test_socketserver (703 sec)
running: test_socketserver (733 sec)
running: test_socketserver (763 sec)
running: test_socketserver (793 sec)
running: test_socketserver (823 sec)
running: test_socketserver (853 sec)
running: test_socketserver (883 sec)
0:17:04 [405/405] test_socketserver crashed
Timeout (0:15:00)!
Thread 0x0000000802006400 (most recent call first):
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/threading.py", line 295 in wait
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/threading.py", line 551 in wait
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/test_socketserver.py", line 369 in wait_done
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/test_socketserver.py", line 341 in __init__
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/test_socketserver.py", line 362 in __init__
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/test_socketserver.py", line 307 in test_threading_handled
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/unittest/case.py", line 601 in run
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/unittest/case.py", line 649 in __call__
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/unittest/suite.py", line 122 in run
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/unittest/suite.py", line 84 in __call__
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/unittest/suite.py", line 122 in run
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/unittest/suite.py", line 84 in __call__
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/unittest/suite.py", line 122 in run
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/unittest/suite.py", line 84 in __call__
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/unittest/runner.py", line 176 in run
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/support/__init__.py", line 1889 in _run_suite
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/support/__init__.py", line 1923 in run_unittest
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/runtest.py", line 164 in test_runner
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/runtest.py", line 165 in runtest_inner
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/runtest.py", line 119 in runtest
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/runtest_mp.py", line 71 in run_tests_slave
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/main.py", line 475 in _main
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/main.py", line 468 in main
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/main.py", line 532 in main
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/regrtest.py", line 46 in _main
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/regrtest.py", line 50 in <module>
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/runpy.py", line 85 in _run_code
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/runpy.py", line 193 in _run_module_as_main
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/__main__.py", line 2, in <module>
    main()
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/main.py", line 532, in main
    Regrtest().main(tests=tests, **kwargs)
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/main.py", line 468, in main
    self._main(tests, kwargs)
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/main.py", line 490, in _main
    self.run_tests()
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/main.py", line 416, in run_tests
    run_tests_multiprocess(self)
  File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/runtest_mp.py", line 221, in run_tests_multiprocess
    raise Exception(msg)
Exception: Child error on test_socketserver: Exit code 1
*** Error code 1

Stop.
make: stopped in /usr/home/buildbot/python/3.6.koobs-freebsd10/build
program finished with exit code 1
elapsedTime=1026.162135

@vstinner vstinner added the tests Tests in the Lib/test dir label May 17, 2017
@vstinner
Copy link
Member Author

Similar issue: bpo-30652, test_threading_not_handled() of test_socketserver hangs randomly on AMD64 FreeBSD 10.x Shared 3.6.

@vstinner vstinner changed the title test_socketserver killed after 15 min on AMD64 FreeBSD 10.x Shared 3.6 test_threading_handled() of test_socketserver hangs randomly on AMD64 FreeBSD 10.x Shared 3.6 Jun 13, 2017
@vstinner
Copy link
Member Author

http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.x%20Shared%203.x/builds/525/steps/test/logs/stdio

0:16:07 load avg: 0.35 [406/406/1] test_socketserver crashed (Exit code 1)
Timeout (0:15:00)!
Thread 0x0000000802006400 (most recent call first):
File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/threading.py", line 296 in wait
File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/threading.py", line 552 in wait
File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/test_socketserver.py", line 369 in wait_done
File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/test_socketserver.py", line 341 in __init__
File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/test_socketserver.py", line 362 in __init__
File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/test_socketserver.py", line 307 in test_threading_handled
File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/unittest/case.py", line 615 in run
File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/unittest/case.py", line 663 in __call__
File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/unittest/suite.py", line 122 in run
File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/unittest/suite.py", line 84 in __call__
File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/unittest/suite.py", line 122 in run
File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/unittest/suite.py", line 84 in __call__
File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/unittest/suite.py", line 122 in run
File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/unittest/suite.py", line 84 in __call__
File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/unittest/runner.py", line 176 in run
File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/support/init.py", line 1896 in _run_suite
File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/support/init.py", line 1940 in run_unittest
File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/libregrtest/runtest.py", line 171 in test_runner
File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/libregrtest/runtest.py", line 172 in runtest_inner
File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/libregrtest/runtest.py", line 130 in runtest
File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/libregrtest/runtest_mp.py", line 71 in run_tests_slave
File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/libregrtest/main.py", line 517 in _main
File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/libregrtest/main.py", line 510 in main
File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/libregrtest/main.py", line 585 in main
File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/regrtest.py", line 46 in _main
File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/regrtest.py", line 50 in <module>
File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/runpy.py", line 85 in _run_code
File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/runpy.py", line 193 in _run_module_as_main

@vadmium
Copy link
Member

vadmium commented Jul 1, 2017

These tests are supposed to:

  1. Create a TCP server
  2. Open a TCP connection
  3. Run a custom connection handler (depending on the particular test) in a new thread
  4. When the handler returns, the new thread should call “shutdown_request”
  5. Shutdown_request closes the server’s connection socket
  6. Shutdown_request sets an event for the main thread
  7. Main thread waits for the above event

The stack trace indicates a thread is stuck at step 7. My guess is that step 5 has raised an exception, killing the thread rather than continuing to step 6. I suspect it is a “socket.close” call raising an asynchronous error such as ECONNRESET; see bpo-30319. A general fix for that problem might fix these test_socketserver hangs.

@vstinner
Copy link
Member Author

vstinner commented Jul 4, 2017

I marked bpo-30652 as duplicate of this issue. Copy of the first message msg295903:

http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.x%20Shared%203.6/builds/218/steps/test/logs/stdio

0:16:16 [405/405] test_socketserver crashed
Timeout (0:15:00)!
Thread 0x0000000802006400 (most recent call first):
File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/threading.py", line 295 in wait
File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/threading.py", line 551 in wait
File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/test_socketserver.py", line 369 in wait_done
File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/test_socketserver.py", line 341 in __init__
File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/test_socketserver.py", line 362 in __init__
File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/test_socketserver.py", line 312 in test_threading_not_handled
File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/unittest/case.py", line 601 in run
File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/unittest/case.py", line 649 in __call__
File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/unittest/suite.py", line 122 in run
File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/unittest/suite.py", line 84 in __call__
File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/unittest/suite.py", line 122 in run
File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/unittest/suite.py", line 84 in __call__
File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/unittest/suite.py", line 122 in run
File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/unittest/suite.py", line 84 in __call__
File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/unittest/runner.py", line 176 in run
File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/support/init.py", line 1889 in _run_suite
File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/support/init.py", line 1923 in run_unittest
File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/runtest.py", line 164 in test_runner
File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/runtest.py", line 165 in runtest_inner
File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/runtest.py", line 119 in runtest
File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/runtest_mp.py", line 71 in run_tests_slave
File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/main.py", line 475 in _main
File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/main.py", line 468 in main
File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/libregrtest/main.py", line 532 in main
File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/regrtest.py", line 46 in _main
File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/test/regrtest.py", line 50 in <module>
File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/runpy.py", line 85 in _run_code
File "/usr/home/buildbot/python/3.6.koobs-freebsd10/build/Lib/runpy.py", line 193 in _run_module_as_main

@vstinner vstinner changed the title test_threading_handled() of test_socketserver hangs randomly on AMD64 FreeBSD 10.x Shared 3.6 test_threading_handled() and test_threading_not_handled() of test_socketserver hangs randomly on AMD64 FreeBSD 10.x Shared 3.6 Jul 4, 2017
@vstinner
Copy link
Member Author

I didn't see this bug recently, I hesitate to close this issue.

@vstinner
Copy link
Member Author

I recently found two issues in socketserver:

  • bpo-31233: socketserver.ThreadingMixIn leaks running threads after server_close()
  • bpo-31151: socketserver.ForkingMixIn.server_close() leaks zombie processes

Fixing these two issues is likely to help to fix this one.

@vstinner
Copy link
Member Author

I modified the socketserver module so server_close() now waits until all spawned threads or processes completed. It should help to avoid such random failure.

Modifying server_close() default behaviour in Python 2.7 or 3.6 is not possible, so maybe we need to modify test_socketserver to use a private API to wait for threads/processes?

@vstinner
Copy link
Member Author

bpo-31233 and bpo-31151 have fixed the issue in Python 3.7.

I'm not sure how to fix the issue in Python 2.7 and 3.6. But since I didn't see the failure recently, I close the issue.

@ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
tests Tests in the Lib/test dir
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@vstinner @vadmium