msg184000 - (view) |
Author: Terry J. Reedy (terry.reedy) * |
Date: 2013-03-12 00:04 |
test_multiprocessing is giving multiple errors like
Process Process-21:
Traceback (most recent call last):
File "F:\Python\dev\py27\lib\multiprocessing\process.py", line 258, in _bootstrap
self.run()
File "F:\Python\dev\py27\lib\multiprocessing\process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "F:\Python\dev\py27\lib\test\test_multiprocessing.py", line 1703, in _test
conn.send('hello')
IOError: [Errno 10038] An operation was attempted on something that is not a socket
...
File "F:\Python\dev\py27\lib\test\test_multiprocessing.py", line 1419, in _putter
manager.connect()
File "F:\Python\dev\py27\lib\multiprocessing\managers.py", line 500, in connect
conn = Client(self._address, authkey=self._authkey)
File "F:\Python\dev\py27\lib\multiprocessing\connection.py", line 465, in XmlClient
return ConnectionWrapper(Client(*args, **kwds), _xml_dumps, _xml_loads)
File "F:\Python\dev\py27\lib\multiprocessing\connection.py", line 175, in Client
answer_challenge(c, authkey)
File "F:\Python\dev\py27\lib\multiprocessing\connection.py", line 420, in answer_challenge
message = connection.recv_bytes(256) # reject large message
IOError: [Errno 10038] An operation was attempted on something that is not a socket
After a few of these, it hangs and I have to either kill multiple processes with TaskManager (or maybe kill-python would work) or kill the console. All tracebacks seem to one of these two. The common factor is 'not a socket'. It seems that something in the test should be skipped on windows. I got similar results running test_multiprocessing by itself. The process numbers are haphazard.
|
msg184001 - (view) |
Author: Terry J. Reedy (terry.reedy) * |
Date: 2013-03-12 00:26 |
For 3.3 and 3.4, test went on for several minutes putting out [###### refs] lines but finally said '1 test OK'. I tried 3.2 again and it never outputs refs lines but fairly soon gives errors.
|
msg184013 - (view) |
Author: Terry J. Reedy (terry.reedy) * |
Date: 2013-03-12 06:49 |
running kill-python results in
Warning -- threading._dangling was modified by test_multiprocessing
Warning -- multiprocessing.process._dangling was modified by test_multiprocessing
test test_multiprocessing failed -- multiple errors occurred; run in verbose mode for details
-v added the attached details, and the test finished, though failed.
|
msg184018 - (view) |
Author: Richard Oudkerk (sbt) * |
Date: 2013-03-12 11:31 |
Does this happen every time you run the tests? (I don't see these errors.)
|
msg184030 - (view) |
Author: Terry J. Reedy (terry.reedy) * |
Date: 2013-03-12 16:31 |
All 4 or 5 times I tried on 3.2, yes.
|
msg184035 - (view) |
Author: Richard Oudkerk (sbt) * |
Date: 2013-03-12 17:09 |
Could you try the following program:
import socket
import multiprocessing
import multiprocessing.reduction
import multiprocessing.connection
def socketpair():
with socket.socket() as l:
l.bind(('localhost', 0))
l.listen(1)
s = socket.socket()
s.connect(l.getsockname())
a, _ = l.accept()
return s, a
def bar(s):
print(s)
s.sendall(b'from bar')
if __name__ == '__main__':
a, b = socketpair()
p = multiprocessing.Process(target=bar, args=(b,))
p.start()
b.close()
print(a.recv(100))
|
msg184037 - (view) |
Author: Terry J. Reedy (terry.reedy) * |
Date: 2013-03-12 17:32 |
In Command Prompt, 3.2 gave same error as before, 3.3 a different error.
multi-test.txt has full tracebacks.
|
msg184039 - (view) |
Author: Richard Oudkerk (sbt) * |
Date: 2013-03-12 18:41 |
Now could you try the attached file? (It will not work on 2.7 because a missing socket.fromfd().)
P.S. It looks like the error for 3.3 is associated with a file f:\python\mypy\traceback.py which presumably clashes with the one in the standard library.
|
msg184046 - (view) |
Author: Terry J. Reedy (terry.reedy) * |
Date: 2013-03-12 19:33 |
Both 3.2 and 3.3 give essentially the same traceback as 3.2 did before, both with installed python and yesterdays debug builds.
|
msg184050 - (view) |
Author: Richard Oudkerk (sbt) * |
Date: 2013-03-12 21:24 |
> Both 3.2 and 3.3 give essentially the same traceback as 3.2 did before,
> both with installed python and yesterdays debug builds.
It looks like on your machine socket handles are not correctly inherited by child processes -- I had assumed that they always would be.
I suppose to fix things for 3.2 and earlier it would be necessary to backport the functionality of socket.socket.share() and socket.fromshare() from 3.3.
|
msg184054 - (view) |
Author: Terry J. Reedy (terry.reedy) * |
Date: 2013-03-12 22:16 |
My original report was for 32 bit debug build on 64 bit Win 7 machine. I just re-ran test_multiprocessing with installed 64 bit python with same result. Was "I don't see these errors." on different Windows or non-Windows.
One option is to skip the failing sub-tests on Windows, like some other sub-tests:
skipped 'does not work with windows sockets'
and consider the limitation on use of multi-processing in 2.7,3.2 as "won't fix". (My view is that 3.2 users should upgrade as soon as dependencies allow.) Backporting new features requires pydev discussion.
|
msg184055 - (view) |
Author: Richard Oudkerk (sbt) * |
Date: 2013-03-12 22:24 |
> My original report was for 32 bit debug build on 64 bit Win 7 machine.
> I just re-ran test_multiprocessing with installed 64 bit python with same
> result. Was "I don't see these errors." on different Windows or non-Windows.
On 64-bit Windows 7 with both 32 and 64 bit builds.
> One option is to skip the failing sub-tests on Windows, like some other sub-tests:
> skipped 'does not work with windows sockets'
> and consider the limitation on use of multi-processing in 2.7,3.2 as "won't fix".
> (My view is that 3.2 users should upgrade as soon as dependencies allow.)
Yes, I would be inclined to do that.
> Backporting new features requires pydev discussion.
I only meant exposing that functionality in the private _multiprocessing extension.
|
msg184059 - (view) |
Author: Terry J. Reedy (terry.reedy) * |
Date: 2013-03-12 23:27 |
Same OS, different hardware, drivers, and buried settings, surprisingly different behavior. Oh, well. I will test any changes you decide on that need testing.
|
msg221683 - (view) |
Author: brian yardy (brianyardy) |
Date: 2014-06-27 12:10 |
All 4 or 5 times I tried on 3.2, yes.
In Command Prompt, 3.2 gave same error as before, 3.3 a different error.
multi-test.txt has full tracebacks.'http://www.einstantloan.co.uk/'
|
msg221695 - (view) |
Author: Terry J. Reedy (terry.reedy) * |
Date: 2014-06-27 17:21 |
This is no longer a 3.x issue. 3.2 and 3.3 get security fixes only. For 3.4, test_multiprocessing is split into 4 files and all run in multiple tries. Test_multiprocessing_spawn takes a minute, but it does 264 + 20 skipped tests, including a few 'wait' tests.
Three tries with 2.7 also passed.
|
|
Date |
User |
Action |
Args |
2022-04-11 14:57:42 | admin | set | github: 61601 |
2014-06-27 17:21:49 | terry.reedy | set | status: open -> closed resolution: out of date messages:
+ msg221695
stage: needs patch -> resolved |
2014-06-27 12:10:42 | brianyardy | set | nosy:
+ brianyardy messages:
+ msg221683
|
2013-03-12 23:27:34 | terry.reedy | set | messages:
+ msg184059 |
2013-03-12 22:24:20 | sbt | set | messages:
+ msg184055 |
2013-03-12 22:16:27 | terry.reedy | set | messages:
+ msg184054 |
2013-03-12 21:42:28 | terry.reedy | set | messages:
- msg184043 |
2013-03-12 21:42:04 | terry.reedy | set | messages:
- msg184042 |
2013-03-12 21:24:13 | sbt | set | messages:
+ msg184050 |
2013-03-12 19:33:32 | terry.reedy | set | files:
+ multi-test2.txt
messages:
+ msg184046 |
2013-03-12 19:17:52 | terry.reedy | set | messages:
+ msg184043 |
2013-03-12 19:06:15 | terry.reedy | set | messages:
+ msg184042 |
2013-03-12 18:41:46 | sbt | set | files:
+ inherit_socket.py
messages:
+ msg184039 |
2013-03-12 17:32:32 | terry.reedy | set | files:
+ mult-test.txt
messages:
+ msg184037 |
2013-03-12 17:09:36 | sbt | set | messages:
+ msg184035 |
2013-03-12 16:31:04 | terry.reedy | set | messages:
+ msg184030 |
2013-03-12 11:31:55 | sbt | set | messages:
+ msg184018 |
2013-03-12 06:49:37 | terry.reedy | set | files:
+ test_multi.txt
messages:
+ msg184013 |
2013-03-12 00:26:25 | terry.reedy | set | messages:
+ msg184001 |
2013-03-12 00:04:04 | terry.reedy | create | |