New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add a fallback socketpair() implementation to the socket module #62843
Comments
socketpair() is quite useful, notably for tests. |
I was thinking about dropping the C wrapper from socketmodule.c, and |
Do you mean you want to use a pure python implementation on Unix? Then you would have to deal with AF_UNIX (which is the default family for socketpair() currently). A pure python implementation which deals with AF_UNIX would have to temporarily create a listening socket on the file system and deal with races like tempfile.mkstemp() does. I think it is simpler to only use the pure python implementation on Windows. BTW, it is possible for another process to connect to the address we temporily bind to. Neither my version nor the one in tulip handle that correctly. |
Here's a patch. Note that I'm still not sure whether it belong to the socket module or test.support. |
On Linux, many tests of test_socket are failing with the pure Python implementation. |
The only failing tests I see are due to the fact that the Python |
I applied your patch and I just replace the hasattr with: ====================================================================== Traceback (most recent call last):
File "/home/haypo/prog/python/default/Lib/test/test_socket.py", line
1259, in test_sendall_interrupted
self.check_sendall_interrupted(False)
File "/home/haypo/prog/python/default/Lib/test/test_socket.py", line
1248, in check_sendall_interrupted
c.sendall(b"x" * (1024**2))
AssertionError: ZeroDivisionError not raised ====================================================================== Traceback (most recent call last):
File "/home/haypo/prog/python/default/Lib/test/test_socket.py", line
1262, in test_sendall_interrupted_with_timeout
self.check_sendall_interrupted(True)
File "/home/haypo/prog/python/default/Lib/test/test_socket.py", line
1248, in check_sendall_interrupted
c.sendall(b"x" * (1024**2))
AssertionError: ZeroDivisionError not raised ====================================================================== Traceback (most recent call last):
File "/home/haypo/prog/python/default/Lib/test/test_socket.py", line
3640, in testDefaults
self._check_defaults(self.serv)
File "/home/haypo/prog/python/default/Lib/test/test_socket.py", line
3630, in _check_defaults
self.assertEqual(sock.family, socket.AF_UNIX)
AssertionError: 2 != 1 ====================================================================== Traceback (most recent call last):
File "/home/haypo/prog/python/default/Lib/test/test_socket.py", line
261, in _tearDown
raise exc
File "/home/haypo/prog/python/default/Lib/test/test_socket.py", line
273, in clientRun
test_func()
File "/home/haypo/prog/python/default/Lib/test/test_socket.py", line
3637, in _testDefaults
self._check_defaults(self.cli)
File "/home/haypo/prog/python/default/Lib/test/test_socket.py", line
3630, in _check_defaults
self.assertEqual(sock.family, socket.AF_UNIX)
AssertionError: 2 != 1 |
This is normal. ======================================================================
> Traceback (most recent call last):
> File "/home/haypo/prog/python/default/Lib/test/test_socket.py", line
> 1259, in test_sendall_interrupted
> self.check_sendall_interrupted(False)
> File "/home/haypo/prog/python/default/Lib/test/test_socket.py", line
> 1248, in check_sendall_interrupted
> c.sendall(b"x" * (1024**2))
> AssertionError: ZeroDivisionError not raised Not this one. $ strace -ttTf ./python -m test -v -m test_sendall_interrupted test_socket |
Alright, I think I know what's happening. The Python implementation uses a TCP socket, whereas the native but for TCP sockets, its set by: So on your machine, you probably have tcp_(r|w)mem quite larger than The solution is simply to increase the amount of data written. If it works I'll commit it, because the test isn't really reliable |
Victor, did you have a chance to test the patch? |
I tested socketpair.diff (modified to use it even on Linux) on another computer, and test ran successfully. On my main computer (Fedora 18, Linux kernel 3.9.4), the test is failing. With sendall_write.diff, the test does pass. test.support.PIPE_MAX_SIZE=4194305 (4 MB) on this computer. I don't know for the other one. |
Since I'll update socket tests to also use support.PIPE_MAX_SIZE, maybe we should rename this variable while we're at it? |
"Since I'll update socket tests to also use support.PIPE_MAX_SIZE, maybe we should rename this variable while we're at it?" Why not using a different value (constant) for sockets? |
By alternating between PIPE and SOCK, you get POPK, which is as descriptive as it gets. |
We can add a new constant (e.g. SOCK_MAX_SIZE), that would be fine, as
Thank you Antoine :-) I'll think I'll go for: |
New changeset 2de7fc69d2d4 by Charles-François Natali in branch '2.7': New changeset 498957c97c2b by Charles-François Natali in branch '3.3': New changeset 9b27cf72c79b by Charles-François Natali in branch 'default': |
Alright, I chose a 16MB size for SOCK_MAX_SIZE because nowadays, one Regarding the original issue: is it OK to add it as |
Here's a patch adding socketpair to test.support. This version has been used in test_selectors for quite some time now, |
Barring any objection, I'll commit the patch attached within a couple days. |
I have objections! You should copy the code from asyncio.windows_utils.socketpair(). It checks also type and proto parameter: raise a ValueError for unsupported values (only SOCK_STREAM and proto=0 are supported). It also supports IPv6. It handles BlockingIOError and InterruptedError on connect (I never understood why these exceptions are simply ignored). You patch checks the address received by accept() and retry. It's a little bit surprising. If you get an unexpected connection, maybe an exception should be raised, instead of just ignoring it. Maybe we should modify the code in asyncio to check also the address? Finally, please add socketpair() directly to the socket module. It is useful for applications, not only for tests. A socket.socketpair() function available on Windows would help me to write tests for the issue bpo-22018! |
I could use it then, although all those checks are unnecessary since
Why?
Well, that's what I suggested initially, but never got any reply, so I So here it is. |
2014-07-24 0:21 GMT+02:00 Charles-François Natali <report@bugs.python.org>:
socket.socket() supports SOCK_DGRAM on Windows, but asyncio I don't remember why I added a specific check on the proto parameter. We might support more protocols in the future, but I'm not sure that
Your version is safer. You should reuse your while dropping unknown connections. By the way, we should reuse socket.socketpair() in Something like: if hasattr(socket, 'socketpair'):
socketpair = socket.socketpair
else:
def socketpair(...): ... Please also fix socketpair() in asyncio to add the while/drop unknown
On Windows, asyncio uses a socket pair in its default event loop Oh, and you forgot to modify the documentation to update |
I tested on Windows: socket.socket(proto=1) raises an OSError(WSAEPROTONOSUPPORT): """ The requested protocol has not been configured into the system, or no implementation for it exists. For example, a socket call requests a SOCK_DGRAM socket, but specifies a stream protocol. Since the error comes directly at socket.socket(), we drop drop the explicit test in socketpair(). |
That's a separate issue.
Did you look at the patch? 363 .. versionchanged:: 3.5 |
2014-07-24 10:11 GMT+02:00 Charles-François Natali <report@bugs.python.org>:
Ok.
Ok, I missed this part. In this case, socketpair-4.diff looks good to me. You can commit your I will open another issue to synchronize asyncio, maybe fix accept() |
Hey Charles-François, can you commit your patch? I forgot that you did commited it yet, and I expected socket.socketpair() to be available on all platforms. |
New changeset 6098141155f9 by Charles-François Natali in branch 'default': |
New changeset 03d3f2664930 by Victor Stinner in branch '3.4': |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: