classification
Title: PEP 475: handle EINTR in the socket module (connect)
Type: Stage:
Components: Extension Modules Versions: Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: haypo, neologix, pitrou, python-dev, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2015-03-09 09:08 by haypo, last changed 2015-04-09 08:32 by python-dev. This issue is now closed.

Files
File name Uploaded Description Edit
connect_eintr.patch haypo, 2015-03-09 09:08 review
connect_eintr.py haypo, 2015-03-09 15:05
connect_eintr-2.patch haypo, 2015-03-09 15:08
connect_test.c haypo, 2015-03-09 15:43
connect_eintr-3.patch haypo, 2015-03-09 15:59
connect_eintr-4.patch haypo, 2015-03-31 20:15 review
connect_eintr-4.patch haypo, 2015-03-31 20:24 review
test_connect_eintr.py haypo, 2015-04-02 10:33
test_connect_eintr2.py haypo, 2015-04-02 11:19
test_connect_eintr3.py haypo, 2015-04-02 11:31
Messages (31)
msg237609 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-03-09 09:08
The PEP 475 has been accepted and is partialy implemented. The socket module is not fully patched to handle EINTR. For example, socket.socket.connect() doesn't handle EINTR yet.

Attached patch fixes socket.connect() to handle EINTR. It should fix issue #11266 and #20611 in Python 3.5 (Python 2.7 and 3.4 will need to be patched to handle explicitly InterruptedError/OSError(EINTR) in Python).

By the way, some socket functions handle EINTR but don't recompute the timeout yet. _PyTime_monotonic() should be used.
msg237610 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2015-03-09 09:10
If EINTR is received during connect, the socket is unusable, that's why i
didn't implement it.
msg237616 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-03-09 09:20
> If EINTR is received during connect, the socket is unusable,
> that's why i didn't implement it.

Can you elaborate?

socket.connect() should be modified according to the PEP 475:
https://www.python.org/dev/peps/pep-0475/#modified-functions

What do you mean by "unusable"? Is it possible to retry connect()? Is it safe to call getsockopt(fd, SOL_SOCKET, SO_ERROR) until it returns EISCONN?

For high-level functions for socket.create_connection(), how should we handle InterruptedError?

See also this change in asyncio to handle InterruptedError in asyncio:
https://hg.python.org/cpython/rev/ad67f66a5f3c
(I wrote it and I didn't test my change, I didn't know how to test it.)
Tulip issue: https://code.google.com/p/tulip/issues/detail?id=205
msg237619 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-03-09 09:29
About EINTR and connect(), I've found the following insightful page:
http://www.madore.org/~david/computers/connect-intr.html

Official POSIX wording is this:

"""If connect() is interrupted by a signal that is caught while blocked waiting to establish a connection, connect() shall fail and set errno to [EINTR], but the connection request shall not be aborted, and the connection shall be established asynchronously."""
msg237656 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-03-09 15:05
connect_eintr.py: script calling socket.connect() in a loop and sending SIGARLM signal every millisecond.
msg237657 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-03-09 15:08
Oops, connect_eintr.py noticed me (thanks to my recent change of the issue #23571 !) that connect_eintr.patch is wrong: socket.connect() returned None with an exception sent, send connect() was interrupted by SIGINT (CTRL+c).

Fixed patch.
msg237666 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-03-09 15:43
> http://www.madore.org/~david/computers/connect-intr.html

This article contains a program connect_test.c to test how connect() behaves on EINTR. Since it's in the public domain, I attached a copy.

The program contains the comment: "All systems function as expected when TEST_TWO is set."

If TEST_TWO is defined, poll() is used to wait until the socket is writable, and then getsockopt(SO_ERROR) is used to check if the error is now zero, when connect() fails with EINTR.
msg237669 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-03-09 15:59
connect_eintr-3.patch: Different patch, don't retry connect() if it returns EINTR, but poll using poll/select. The patch changes also the asyncio module.
msg238624 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-03-20 09:09
I tested polling+getsockopt(SO_ERROR) with connect_test.c on Linux, it works:

haypo@smithers$ gcc -D TEST_TWO=1 connect_test.c -o connect_test
haypo@smithers$ ./connect_test 
Will try to connect to 127.0.0.1 on port 80
(connect has been interrupted and now completed successfully)
msg238720 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2015-03-20 20:49
Could you regenerate your latest patch?
It doesn't show properly in the review tool.

Also, what's with the assert?
msg239444 - (view) Author: Roundup Robot (python-dev) Date: 2015-03-28 00:29
New changeset f841d3bc30ee by Victor Stinner in branch 'default':
Issue #23618, #22117: refactor socketmodule.c
https://hg.python.org/cpython/rev/f841d3bc30ee
msg239640 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-03-30 20:34
test_selectors.patch: Enhance test_selector to test the two kinds of signal handlers: one raises an exception, the other one doesn't. I wait until kqueue & devpoll retry on EINTR to push test_selectors.patch.
msg239690 - (view) Author: Roundup Robot (python-dev) Date: 2015-03-31 12:11
New changeset e8246baad0f6 by Victor Stinner in branch 'default':
Issue #23618: Refactor the _socket module
https://hg.python.org/cpython/rev/e8246baad0f6

New changeset fa5542660b17 by Victor Stinner in branch 'default':
Issue #23618: Refactor internal_select() to prepare socket.connect() for EINTR
https://hg.python.org/cpython/rev/fa5542660b17

New changeset c027d6468667 by Victor Stinner in branch 'default':
Issue #23618: internal_connect_select() now waits also for error events
https://hg.python.org/cpython/rev/c027d6468667
msg239693 - (view) Author: Roundup Robot (python-dev) Date: 2015-03-31 12:26
New changeset 47b2d1ff9743 by Victor Stinner in branch 'default':
Issue #23618: Fix internal_connect_select()
https://hg.python.org/cpython/rev/47b2d1ff9743
msg239719 - (view) Author: Roundup Robot (python-dev) Date: 2015-03-31 14:39
New changeset daf3d2a717e5 by Victor Stinner in branch 'default':
Issue #23618: Refactor internal_connect()
https://hg.python.org/cpython/rev/daf3d2a717e5

New changeset c59d81b802f8 by Victor Stinner in branch 'default':
Issue #23618: Refactor internal_connect()
https://hg.python.org/cpython/rev/c59d81b802f8
msg239741 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-03-31 17:39
Le mardi 31 mars 2015, Roundup Robot <report@bugs.python.org> a écrit :
>
> New changeset c59d81b802f8 by Victor Stinner in branch 'default':
> Issue #23618: Refactor internal_connect()
> https://hg.python.org/cpython/rev/c59d81b802f8
>

Oh I forgot to add an #ifdef for socklen_t. I didn't get a warning, so
socklen_t is probably an int. Getsockopt() uses a int* for the value length.

I also forgot to drop the comment before SetLastError(), it's no more
revelant.
msg239744 - (view) Author: Roundup Robot (python-dev) Date: 2015-03-31 19:32
New changeset d9374864d4a9 by Victor Stinner in branch 'default':
Issue #23618: Cleanup internal_connect() in socketmodule.c
https://hg.python.org/cpython/rev/d9374864d4a9

New changeset 4fad2b9fc4e6 by Victor Stinner in branch 'default':
Issue #23618: Fix EINTR handling in socket.connect()
https://hg.python.org/cpython/rev/4fad2b9fc4e6
msg239746 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-03-31 20:15
connect_eintr-4.patch: Updated patch after all my changes to refactor socket.connect() (which is now almost a rewrite from scratch!).
msg239748 - (view) Author: Roundup Robot (python-dev) Date: 2015-03-31 20:24
New changeset 7ed567ad8b4c by Victor Stinner in branch 'default':
Issue #23618: Enhance EINTR handling in socket.connect()
https://hg.python.org/cpython/rev/7ed567ad8b4c
msg239775 - (view) Author: Roundup Robot (python-dev) Date: 2015-04-01 09:10
New changeset 8ec4acfdb851 by Victor Stinner in branch 'default':
Issue #23618: Fix EINTR handling on Windows
https://hg.python.org/cpython/rev/8ec4acfdb851
msg239896 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-04-02 10:33
test_connnect_eintr.py: program to interrupt socket.connect() with signals. It looks like socket.connect() cannot be interrupted by signals: connect() only fails with WSAEINTR when WSACancelBlockingCall() is called, but WSACancelBlockingCall() "has been removed in compliance with the Windows Sockets 2 specification, revision 2.2.0":

https://msdn.microsoft.com/en-us/library/windows/desktop/ms741547%28v=vs.85%29.aspx

"Blocking hooks are generally used to keep a single-threaded GUI application responsive during calls to blocking functions. Instead of using blocking hooks, an applications should use a separate thread (separate from the main GUI thread) for network activity."
msg239899 - (view) Author: Roundup Robot (python-dev) Date: 2015-04-02 10:57
New changeset aad52bfc816f by Victor Stinner in branch 'default':
Issue #23618: Don't declare recvmsg/sendmsg helper functions on Windows
https://hg.python.org/cpython/rev/aad52bfc816f

New changeset f22188acc77d by Victor Stinner in branch 'default':
Issue #23618: Document EINTR changes in socket documentation
https://hg.python.org/cpython/rev/f22188acc77d
msg239902 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-04-02 11:19
Hum, connect() does not always block with test_connect_eintr.py, and this program sometimes fail with ConnectionResetError on connect() on FreeBSD.

New program which works on Linux and FreeBSD. It now ensures that connect() will block.
msg239903 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-04-02 11:31
test_connect_eintr3.py: even better:

- block signals in the server thread
- count signals during connect()
- display progress: "*" for signal received during connect(), "_" for signal received before/after connect(), "[" and "]" for the beginning and end of a connection, "#" for client connection reset

Example of output on FreeBSD:

Register SIGINT
Register SIGALRM
Register SIGWINCH
Register SIGTERM
Register SIGCHLD
Send SIGALRM every 200.0 ms
Run func() during 5.0 seconds
Type CTRL+c, resize the window, etc.

___[]______[]_____[_*#]_____[#]__[#]________[#]____[#]_____[**]____[*#]______[__#]___[#]_____[#]____[*#]______[#]______[#]_____[*#]_____[#]_____[#]______[#]______[#]______[#]______[#]______[#]____[#]_______[#]_____[#]_______[#]_____[#]_____[#]______[#]_______[#]____[#]______[#]______[*#]_____[#]________[#]__

Test completed in 5.1 sec
func() has been called 36 times
Got 204 signals
Got 7 signals during connect()
msg239904 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-04-02 11:33
Example of test_connect_eintr3.py output on Linux (3.18):

Register SIGINT
Register SIGALRM
Register SIGWINCH
Register SIGTERM
Register SIGCHLD
Send SIGALRM every 200.0 ms
Run func() during 5.0 seconds
Type CTRL+c, resize the window, etc.

[]_____[]_____[]_____[]_____[**********][**********][]______[**********][]_____[]____[**********][**********]

Test completed in 5.4 sec
func() has been called 12 times
Got 85 signals
Got 50 signals during connect()


Example of test_connect_eintr3.py output on Mac OS X Yosemite (10.10):

Register SIGINT
Register SIGALRM
Register SIGWINCH
Register SIGTERM
Register SIGCHLD
Send SIGALRM every 200.0 ms
Run func() during 5.0 seconds
Type CTRL+c, resize the window, etc.

[]______[*********][***********][**********]_[********][***********]

Test completed in 5.3 sec
func() has been called 6 times
Got 55 signals
Got 49 signals during connect()
msg239905 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-04-02 11:37
Example of test_connect_eintr3.py output on OpenIndiana:

Register SIGINT
Register SIGALRM
Register SIGWINCH
Register SIGTERM
Register SIGCHLD
Send SIGALRM every 200.0 ms
Run func() during 5.0 seconds
Type CTRL+c, resize the window, etc.

______[]____[]_____[******]______[]_______[*****]______[*****]______[]

Test completed in 5.2 sec
func() has been called 7 times
Got 56 signals
Got 16 signals during connect()


Oh, and obviously, test_connect_eintr3.py fails with InterruptedError without the patch. Example on Linux:

Register SIGINT
Register SIGALRM
Register SIGWINCH
Register SIGTERM
Register SIGCHLD
Send SIGALRM every 200.0 ms
Run func() during 5.0 seconds
Type CTRL+c, resize the window, etc.

[]____[]_____[]_____[]____[******Traceback (most recent call last):
  File "test_connect_eintr.py", line 97, in <module>
    func()
  File "test_connect_eintr.py", line 51, in func
    client.connect(server_addr)
InterruptedError: [Errno 4] Interrupted system call
msg239906 - (view) Author: Roundup Robot (python-dev) Date: 2015-04-02 11:42
New changeset 75fcc7a3738a by Victor Stinner in branch 'default':
Issue #23618: socket.socket.connect() now waits until the connection completes
https://hg.python.org/cpython/rev/75fcc7a3738a
msg239911 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-04-02 12:25
http://buildbot.python.org/all/builders/AMD64%20Snow%20Leop%203.x/builds/2904/steps/test/logs/stdio

======================================================================
FAIL: test_connect_ex_error (test.test_ssl.NetworkedTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/test/test_ssl.py", line 1441, in test_connect_ex_error
    self.assertIn(rc, (errno.ECONNREFUSED, errno.EWOULDBLOCK))
AssertionError: 36 not found in (61, 35)

where 36 is errno.EINPROGRESS
msg239912 - (view) Author: Roundup Robot (python-dev) Date: 2015-04-02 12:38
New changeset 44adbb5eeb4b by Victor Stinner in branch 'default':
Issue #23618: Fix sock_connect_impl(), set the socket error code
https://hg.python.org/cpython/rev/44adbb5eeb4b
msg239913 - (view) Author: Roundup Robot (python-dev) Date: 2015-04-02 13:20
New changeset 09a4d5cc6afd by Victor Stinner in branch 'default':
Issue #23618: Ooops, remove abort() added for debug purpose
https://hg.python.org/cpython/rev/09a4d5cc6afd
msg240312 - (view) Author: Roundup Robot (python-dev) Date: 2015-04-09 08:32
New changeset bff88c866886 by Victor Stinner in branch 'default':
Issue #23618: Fix internal_select() for negative timeout (blocking socket) when
https://hg.python.org/cpython/rev/bff88c866886
History
Date User Action Args
2015-04-09 08:32:58python-devsetmessages: + msg240312
2015-04-02 14:17:55hayposetstatus: open -> closed
resolution: fixed
2015-04-02 13:20:22python-devsetmessages: + msg239913
2015-04-02 12:38:14python-devsetmessages: + msg239912
2015-04-02 12:25:36hayposetmessages: + msg239911
2015-04-02 11:42:47python-devsetmessages: + msg239906
2015-04-02 11:37:04hayposetmessages: + msg239905
2015-04-02 11:33:52hayposetmessages: + msg239904
2015-04-02 11:31:21hayposetfiles: + test_connect_eintr3.py

messages: + msg239903
2015-04-02 11:19:16hayposetfiles: + test_connect_eintr2.py

messages: + msg239902
2015-04-02 10:57:54python-devsetmessages: + msg239899
2015-04-02 10:33:51hayposetfiles: + test_connect_eintr.py

messages: + msg239896
2015-04-01 09:10:38python-devsetmessages: + msg239775
2015-03-31 20:24:17hayposetfiles: + connect_eintr-4.patch
2015-03-31 20:24:05python-devsetmessages: + msg239748
2015-03-31 20:15:57hayposetfiles: + connect_eintr-4.patch

messages: + msg239746
2015-03-31 19:32:02python-devsetmessages: + msg239744
2015-03-31 17:39:56hayposetmessages: + msg239741
2015-03-31 14:39:29python-devsetmessages: + msg239719
2015-03-31 12:26:10python-devsetmessages: + msg239693
2015-03-31 12:11:39python-devsetmessages: + msg239690
2015-03-31 10:21:42hayposetfiles: - test_selectors.patch
2015-03-30 20:34:16hayposetfiles: + test_selectors.patch

messages: + msg239640
2015-03-30 02:14:06hayposettitle: PEP 475: handle EINTR in the socket module -> PEP 475: handle EINTR in the socket module (connect)
2015-03-28 00:29:19python-devsetnosy: + python-dev
messages: + msg239444
2015-03-20 20:49:46neologixsetmessages: + msg238720
2015-03-20 09:09:25hayposetmessages: + msg238624
2015-03-12 15:51:50haypolinkissue23648 dependencies
2015-03-09 15:59:27hayposetfiles: + connect_eintr-3.patch

messages: + msg237669
2015-03-09 15:43:24hayposetfiles: + connect_test.c

messages: + msg237666
2015-03-09 15:08:37hayposetfiles: + connect_eintr-2.patch

messages: + msg237657
2015-03-09 15:05:38hayposetfiles: + connect_eintr.py

messages: + msg237656
2015-03-09 09:29:31pitrousetnosy: + pitrou
messages: + msg237619
2015-03-09 09:20:02hayposetmessages: + msg237616
2015-03-09 09:10:48neologixsetmessages: + msg237610
2015-03-09 09:08:33haypocreate