-
-
Notifications
You must be signed in to change notification settings - Fork 29.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
socket.setdefaulttimeout affecting multiprocessing Manager #50306
Comments
Terminal 1:
Python 2.6.1 (r261:67515, Apr 2 2009, 18:25:55)
[GCC 4.1.2 20071124 (Red Hat 4.1.2-42)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from multiprocessing.managers import SyncManager
>>> manager = SyncManager(authkey="mykey")
>>> manager.start()
>>> queue = manager.Queue()
>>> import pickle
>>> pickle.dump(queue, open("myqueue.pkl", "w"))
>>>
Terminal 2:
Python 2.6.1 (r261:67515, Apr 2 2009, 18:25:55)
[GCC 4.1.2 20071124 (Red Hat 4.1.2-42)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> socket.setdefaulttimeout(30)
>>> import multiprocessing, pickle
>>> multiprocessing.current_process().authkey = "mykey"
>>> queue = pickle.load(open("myqueue.pkl"))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "python2.6/pickle.py", line 1370, in load
return Unpickler(file).load()
File "python2.6/pickle.py", line 858, in load
dispatch[key](self)
File "python2.6/pickle.py", line 1133, in load_reduce
value = func(*args)
File "python2.6/multiprocessing/managers.py", line 845, in RebuildProxy
return func(token, serializer, incref=incref, **kwds)
File "python2.6/multiprocessing/managers.py", line 894, in AutoProxy
incref=incref)
File "python2.6/multiprocessing/managers.py", line 700, in __init__
self._incref()
File "python2.6/multiprocessing/managers.py", line 749, in _incref
conn = self._Client(self._token.address, authkey=self._authkey)
File "python2.6/multiprocessing/connection.py", line 140, in Client
answer_challenge(c, authkey)
File "python2.6/multiprocessing/connection.py", line 376, in
answer_challenge
response = connection.recv_bytes(256) # reject large message
IOError: [Errno 11] Resource temporarily unavailable
>>> This works as expected without socket.setdefaulttimeout(). However, the I suspect the cause is similar to http://bugs.python.org/issue976613 |
Pickling the queue and then unpickling it in a new process is something I |
Yeah, storing pickled queues in the file system makes for some easy IPC So, from what I can tell this issue is related to the mixing of standard In multiprocessing/connection.py, SocketClient() creates a familiar However, after a connection is established, SocketClient() will not use There might be a number of ways to make SocketClient() more timeout You obviously have more experience with this code base so I'll be |
Well; I'm pretty tapped out right now - I think your idea of checking to |
This should be higher priority as one of the major benefits of the multiprocessing module is remote process management in a completely transparent manner. socket timeouts are very important in this context as blocking forever waiting for a connection is not always an option. The problem of not being able to use a default socket timeout for other purposes in combination with multiprocessing managers is definitely an issue, but making multiprocessing actually use the timeout itself if set would be a huge advantage. This might not be the place to ask for it, but it would make sense for manager objects to gain a timeout attribute to be used as a timeout for local or remote communications. At the very least, the manager.connect() method should accept a timeout argument. |
I agree derek, I think that would be a fine addition, however we lack a patch and I don't have the current bandwidth to add it. |
While having multiprocessing use a timeout would be great, I didn't really have the time to fiddle with the c code. Instead of using the socket timeout, I'm modifying all the sockets created by the socket module to have no timeout (and thus to be blocking) which makes the multiprocessing module 'immune' to the socket module's default timeout. For testing, I just run the test suite twice, once with the initial default socket timeout and once with a 60 second timeout. Nothing there failed with this issue. It is worth noting, however, that when using a default socket timeout, for some reason processes that have have put data into a queue no longer block at exit waiting for the data to be consumed. I'm not sure if there is some additional cleanup code that uses sockets and might need to block? Or maybe whatever the issue was with blocking sockets is not an issue with non-blocking sockets? |
I was wrong about exit behavior of a process that has put to a queue -- it seems to behave as expected. i had been playing with a proxy to a queue rather than a queue (to which, if you put, the process can exit right away because the actual put happens in the process that owns the queue). I think this works as intended, but lmk. Also, I haven't really played with the tests that much, so that bit could use some review. It hasn't broken anything in any of my real world tests though. Also, have I mentioned that the multiprocessing module is amazing? Cause it is. I sort of feel like antigravity == multiprocessing ... |
Any chance this patch will be accepted (or at least reviewed) soon? |
The wording in 138415 suggested this patch was changing socket to not support timeouts -- which would be unacceptable. But the actual patch only seems to touch multiprocessing/connection.py -- a far more reasonable change. Unfortunately, the patch no longer applies to the development tip. I *think* the places you wanted to change are still there, and just moved. (1) Is it sufficiently clear that this is not-a-feature to justify a backport? (2) Are the problems already fixed by some of the other changes? (It doesn't look like it, but I'm not sure.) (3) Can you produce an updated patch? (The current tip is http://hg.python.org/cpython/file/fec45282dc28/Lib/multiprocessing/connection.py ) (4) If I understand the intent, then s.setblocking(True) would be slightly more clear than s.settimeout(None), though that change obviously isn't essential. |
Thanks, Jim, here is an updated patch.
My test modifications cover the entire suite twice, once without a default timeout and once with. This may be excessive? I'm not sure where non-blocking sockets might pop up as an issue since there is C code that relies on blocking sockets and I haven't dug that deep. |
New changeset 4e85e4743757 by Richard Oudkerk in branch '2.7': New changeset 290f04722be3 by Richard Oudkerk in branch '3.2': New changeset f03839401420 by Richard Oudkerk in branch 'default': |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: