➜

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: test_concurrent_futures failures on Windows: IOError('[Errno 232] The pipe is being closed') on _send_bytes()
Type: behavior Stage: resolved
Components: Library (Lib), Tests Versions: Python 3.2, Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: amaury.forgeotdarc, neologix, pitrou, python-dev, vstinner
Priority: normal Keywords: patch

Created on 2011-09-29 19:07 by vstinner, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
error_no_data.patch pitrou, 2011-10-04 10:55 review
Messages (9)
msg144612 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-09-29 19:07 
Re-running test 'test_concurrent_futures' in verbose mode
test_killed_child (test.test_concurrent_futures.ProcessPoolExecutorTest) ... 5.79s ok
test_map (test.test_concurrent_futures.ProcessPoolExecutorTest) ... 3.63s ok
test_map_exception (test.test_concurrent_futures.ProcessPoolExecutorTest) ... 4.05s ok
test_map_timeout (test.test_concurrent_futures.ProcessPoolExecutorTest) ... 9.64s ok
test_shutdown_race_issue12456 (test.test_concurrent_futures.ProcessPoolExecutorTest) ... 3.60s ok
test_submit (test.test_concurrent_futures.ProcessPoolExecutorTest) ... 3.63s ok
test_submit_keyword (test.test_concurrent_futures.ProcessPoolExecutorTest) ... 4.09s ok
test_map (test.test_concurrent_futures.ThreadPoolExecutorTest) ... 0.13s ok
test_map_exception (test.test_concurrent_futures.ThreadPoolExecutorTest) ... 0.12s ok
test_map_submits_without_iteration (test.test_concurrent_futures.ThreadPoolExecutorTest)
Tests verifying issue 11777. ... 0.14s ok
test_map_timeout (test.test_concurrent_futures.ThreadPoolExecutorTest) ... 6.11s ok
test_shutdown_race_issue12456 (test.test_concurrent_futures.ThreadPoolExecutorTest) ... 0.11s ok
test_submit (test.test_concurrent_futures.ThreadPoolExecutorTest) ... 0.11s ok
test_submit_keyword (test.test_concurrent_futures.ThreadPoolExecutorTest) ... 0.11s ok
test_all_completed (test.test_concurrent_futures.ProcessPoolWaitTests) ... 3.74s ok
test_first_completed (test.test_concurrent_futures.ProcessPoolWaitTests) ... 5.18s ok
test_first_completed_some_already_completed (test.test_concurrent_futures.ProcessPoolWaitTests) ... 5.02s ok
test_first_exception (test.test_concurrent_futures.ProcessPoolWaitTests) ... 6.57s ok
test_first_exception_one_already_failed (test.test_concurrent_futures.ProcessPoolWaitTests) ... 6.37s ok
test_first_exception_some_already_complete (test.test_concurrent_futures.ProcessPoolWaitTests) ... 5.15s ok
test_timeout (test.test_concurrent_futures.ProcessPoolWaitTests) ... 10.94s ok
test_all_completed (test.test_concurrent_futures.ThreadPoolWaitTests) ... 0.12s ok
test_first_completed (test.test_concurrent_futures.ThreadPoolWaitTests) ... 1.62s ok
test_first_completed_some_already_completed (test.test_concurrent_futures.ThreadPoolWaitTests) ... 1.61s ok
test_first_exception (test.test_concurrent_futures.ThreadPoolWaitTests) ... 3.12s ok
test_first_exception_one_already_failed (test.test_concurrent_futures.ThreadPoolWaitTests) ... 2.10s ok
test_first_exception_some_already_complete (test.test_concurrent_futures.ThreadPoolWaitTests) ... 1.60s ok
test_timeout (test.test_concurrent_futures.ThreadPoolWaitTests) ... 6.12s ok
test_no_timeout (test.test_concurrent_futures.ProcessPoolAsCompletedTests) ... 3.62s ok
test_zero_timeout (test.test_concurrent_futures.ProcessPoolAsCompletedTests) ... 5.65s ok
test_no_timeout (test.test_concurrent_futures.ThreadPoolAsCompletedTests) ... 0.12s ok
test_zero_timeout (test.test_concurrent_futures.ThreadPoolAsCompletedTests) ... 2.11s ok
test_cancel (test.test_concurrent_futures.FutureTests) ... ok
test_cancelled (test.test_concurrent_futures.FutureTests) ... ok
test_done (test.test_concurrent_futures.FutureTests) ... ok
test_done_callback_already_cancelled (test.test_concurrent_futures.FutureTests) ... ok
test_done_callback_already_failed (test.test_concurrent_futures.FutureTests) ... ok
test_done_callback_already_successful (test.test_concurrent_futures.FutureTests) ... ok
test_done_callback_raises (test.test_concurrent_futures.FutureTests) ... ok
test_done_callback_with_cancel (test.test_concurrent_futures.FutureTests) ... ok
test_done_callback_with_exception (test.test_concurrent_futures.FutureTests) ... ok
test_done_callback_with_result (test.test_concurrent_futures.FutureTests) ... ok
test_exception_with_success (test.test_concurrent_futures.FutureTests) ... ok
test_exception_with_timeout (test.test_concurrent_futures.FutureTests) ... ok
test_repr (test.test_concurrent_futures.FutureTests) ... ok
test_result_with_cancel (test.test_concurrent_futures.FutureTests) ... ok
test_result_with_success (test.test_concurrent_futures.FutureTests) ... ok
test_result_with_timeout (test.test_concurrent_futures.FutureTests) ... ok
test_running (test.test_concurrent_futures.FutureTests) ... ok
test_context_manager_shutdown (test.test_concurrent_futures.ProcessPoolShutdownTest) ... 3.58s ok
test_del_shutdown (test.test_concurrent_futures.ProcessPoolShutdownTest) ... 3.92s ok
test_interpreter_shutdown (test.test_concurrent_futures.ProcessPoolShutdownTest) ... 5.60s FAIL
test_processes_terminate (test.test_concurrent_futures.ProcessPoolShutdownTest) ... 3.67s ok
test_run_after_shutdown (test.test_concurrent_futures.ProcessPoolShutdownTest) ... 0.01s ok
test_context_manager_shutdown (test.test_concurrent_futures.ThreadPoolShutdownTest) ... 0.02s ok
test_del_shutdown (test.test_concurrent_futures.ThreadPoolShutdownTest) ... 0.03s ok
test_interpreter_shutdown (test.test_concurrent_futures.ThreadPoolShutdownTest) ... 1.68s ok
test_run_after_shutdown (test.test_concurrent_futures.ThreadPoolShutdownTest) ... 0.00s ok
test_threads_terminate (test.test_concurrent_futures.ThreadPoolShutdownTest) ... Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\multiprocessing\queues.py", line 267, in _feed
    send(obj)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\multiprocessing\connection.py", line 231, in send
    self._send_bytes(memoryview(buf))
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\multiprocessing\connection.py", line 299, in _send_bytes
    overlapped = win32.WriteFile(self._handle, buf, overlapped=True)
IOError: [Errno 232] The pipe is being closed
Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\multiprocessing\queues.py", line 267, in _feed
    send(obj)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\multiprocessing\connection.py", line 231, in send
    self._send_bytes(memoryview(buf))
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\multiprocessing\connection.py", line 299, in _send_bytes
    overlapped = win32.WriteFile(self._handle, buf, overlapped=True)
IOError: [Errno 232] The pipe is being closed
Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\multiprocessing\queues.py", line 267, in _feed
    send(obj)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\multiprocessing\connection.py", line 231, in send
    self._send_bytes(memoryview(buf))
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\multiprocessing\connection.py", line 299, in _send_bytes
    overlapped = win32.WriteFile(self._handle, buf, overlapped=True)
IOError: [Errno 232] The pipe is being closed
Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\multiprocessing\queues.py", line 267, in _feed
    send(obj)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\multiprocessing\connection.py", line 231, in send
    self._send_bytes(memoryview(buf))
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\multiprocessing\connection.py", line 299, in _send_bytes
    overlapped = win32.WriteFile(self._handle, buf, overlapped=True)
IOError: [Errno 232] The pipe is being closed
Warning -- threading._dangling was modified by test_concurrent_futures
test test_concurrent_futures failed
0.01s ok

======================================================================
FAIL: test_interpreter_shutdown (test.test_concurrent_futures.ProcessPoolShutdownTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\test\test_concurrent_futures.py", line 109, in test_interpreter_shutdown
    self.assertFalse(err)
AssertionError: b'Traceback (most recent call last):\n  File "D:\\cygwin\\home\\db3l\\buildarea\\3.x.bolen-windows\\build\\lib\\multiprocessing\\queues.py", line 267, in _feed\n    send(obj)\n  File "D:\\cygwin\\home\\db3l\\buildarea\\3.x.bolen-windows\\build\\lib\\multiprocessing\\connection.py", line 231, in send' is not false

----------------------------------------------------------------------
Ran 59 tests in 133.632s

http://www.python.org/dev/buildbot/all/builders/x86%20XP-4%203.x/builds/5243/steps/test/logs/stdio
msg144613 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-09-29 19:07 
See also issue #13059.
msg144874 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-10-04 10:55 
I think the solution would be to map ERROR_NO_DATA (232 - "The pipe is being closed") to EPIPE. Attached patch.
msg144887 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-10-04 12:37 
> Attached patch.

Could you please explain your change in generrmap.c in a comment? For example, just add a reference to this issue.
msg145077 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-10-07 14:22 
New changeset d8d8374ddbcc by Antoine Pitrou in branch '3.2':
Issue #13063: the Windows error ERROR_NO_DATA (numbered 232 and described
http://hg.python.org/cpython/rev/d8d8374ddbcc

New changeset 3784b6000640 by Antoine Pitrou in branch 'default':
Issue #13063: the Windows error ERROR_NO_DATA (numbered 232 and described
http://hg.python.org/cpython/rev/3784b6000640
msg145078 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-10-07 14:23 
Hopefully this will fix the random failures (EPIPE is caught somewhere down in multiprocessing.Queue._feed).
msg145125 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-10-07 19:01 
Mmh, for some reason it seems it was not enough:
http://www.python.org/dev/buildbot/all/builders/x86%20XP-4%203.x/builds/5328/steps/test/logs/stdio
msg145137 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-10-07 19:53 
Ah, this is because IOError is raised because of WindowsError, thus there's no errno translation.
Ok, I'd like to wait for PEP 3151 to be merged, then, because it should solve this.
msg145422 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-10-12 19:23 
The issue has been fixed by the merge of the PEP 3151: I now get a BrokenPipeError with errno=32 and winerror=232.
History
Date User Action Args
2022-04-11 14:57:22adminsetgithub: 57272
2011-10-12 19:23:40vstinnersetstatus: open -> closed

messages: + msg145422
2011-10-07 19:53:46pitrousetmessages: + msg145137
2011-10-07 19:01:55pitrousetstatus: pending -> open

messages: + msg145125
2011-10-07 14:23:53pitrousetstatus: open -> pending
resolution: fixed
messages: + msg145078

stage: patch review -> resolved
2011-10-07 14:22:07python-devsetnosy: + python-dev
messages: + msg145077
2011-10-04 12:37:45vstinnersetmessages: + msg144887
2011-10-04 10:56:25pitrousetversions: + Python 3.2
nosy: + amaury.forgeotdarc

components: + Tests
type: behavior
stage: patch review
2011-10-04 10:55:42pitrousetfiles: + error_no_data.patch
keywords: + patch
messages: + msg144874
2011-09-29 19:07:33vstinnersetmessages: + msg144613
2011-09-29 19:07:13vstinnercreate