classification
Title: PEP 475 meta issue
Type: Stage:
Components: Versions: Python 3.5
process
Status: closed Resolution: fixed
Dependencies: 23285 23485 23618 23646 23707 23708 23709 23715 23836 23851 23853 Superseder:
Assigned To: Nosy List: Arfrever, neologix, python-dev, vstinner
Priority: normal Keywords: patch

Created on 2015-03-12 15:51 by vstinner, last changed 2015-05-26 23:33 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
test_eintr.py vstinner, 2015-03-18 09:23
unittest_setitimer.patch vstinner, 2015-03-19 23:33 review
fstat_not_eintr.py vstinner, 2015-03-23 21:51
inject_signals.patch vstinner, 2015-03-30 20:36 review
Messages (26)
msg237954 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-12 15:51
I created this issue to group all issues related to the implementation of the PEP 475 "Retry system calls failing with EINTR".
https://www.python.org/dev/peps/pep-0475/

- Issue #23646: the time module, retry sleep()
- Issue #23618: the socket module
- Issue #23285: the io and the os modules, and open()
- Issue #23485: the select and selectors module
msg238355 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-17 22:39
In msg196555, Charles-François Natali wrote:

"""
From a cursory look, the main files affected would be:
Modules/fcntlmodule.c
Modules/ossaudiodev.c
Modules/posixmodule.c
Modules/selectmodule.c
Modules/selectmodule.c
Modules/signalmodule.c
Modules/socketmodule.c
Modules/syslogmodule.c
"""

For syslog, I tested the following code and I don't get any InterruptedError. The return type of the C functions openlog(), syslog() and closelog() is void: no result. So I don't see how they could fail because of a signal. Code of my test:
---
import syslog
import signal

hit = 0

def noop(*args):
    global hit
    hit += 1

signal.signal(signal.SIGALRM, noop)
t = 1e-6
nlog = 10**2
signal.setitimer(signal.ITIMER_REAL, t, t)
for i in range(nlog):
    syslog.openlog()
    syslog.syslog("test %s")
    syslog.closelog()
signal.signal(signal.SIGALRM, signal.SIG_IGN)

print(nlog, "logs")
print(hit, "signals")
---
msg238391 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-18 09:23
test_eintr.py: program to check if a Python function can fail with InterruptedError. Modify func() to test a function. (The program checks select.select(), currently it fails with InterruptedError.)
msg238393 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-18 09:47
I found a way to get a list of functions which can fail with EINTR: search for TEMP_FAILURE_RETRY in the source code of the glibc. TEMP_FAILURE_RETRY:
https://www.gnu.org/software/libc/manual/html_node/Interrupted-Primitives.html#Interrupted-Primitives

Following functions are called with TEMP_FAILURE_RETRY (in glibc code, doc and examples):

- accept, send, sendfile, sendmsg, accept4, connect, sendto, recvmsg
- write, writev, read, readv, pwrite
- fdatasync, fsync
- waitpid
- select, poll
- mq_send, mq_timedsend, mq_receive, mq_timedreceive
- clock_nanosleep, timer_settime, nanosleep
- posix_fallocate
- sem_wait, sem_timedwait, sem_trywait
- aio_suspend
- fcntl(F_SETLK)
msg238394 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-18 09:48
stat manual page of SCO documentation says that stat() can fail with EINTR, but I don't see this error in latest Linux manual page. I tested manually, and I failed to see stat() failing with InterruptedError.
http://uw714doc.sco.com/en/man/html.2/stat.2.html

So only a few system calls on the filesystem (open, read, write, close, fsync, ...) can fail with EINTR?
msg238521 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-19 13:41
More issues:

- Issue #23707: os.urandom() doesn't handle EINTR correctly
- Issue #23708: Add _Py_read() and _Py_write() functions
- Issue #23709: Refactor ossaudiodev: use _Py_read and _Py_write with the Py_buffer
msg238529 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-19 14:39
signal.sigwaitinfo() raises InterruptedError if the wait was interrupted by a signal  handler (this handler was for a signal other than one of those in set).

IMO we should also retry sigwaitinfo() if it fails with EINTR.
msg238541 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-19 16:49
I searched for "EINTR" in all manual pages of my Fedora 21. I found a long list of functions (filename of manual pages). But I didn't check each manual page.

There are false positive. For example, "stat" is listed, but in fact EINTR is only mentionned in commented text (not visible using the "man" command): "SVr4 documents additional stat() error conditions EINTR, ...".

Another example of false positive with system:
"""
Early  drafts  required, or allowed, system() to return with errno set to
[EINTR] if it was interrupted with a signal. This error return was removed, and
a requirement that system() not return until the child has terminated was
added. This  means  that  if  a waitpid()  call  in  system() exits with errno
set to [EINTR], system() must reissue the waitpid().
"""

Shell command to search for all EINTR:
find /usr/share/man -name "*.gz"|xargs zgrep -H EINTR > EINTR.txt

Shell command to extract file names from the output:

sed 's!.*/!!g' EINTR.txt|sed 's!\.[0-9][a-z]\?m\?\.gz:.*!!g'|sort -u > EINTR_files.txt

Filename of manual pages:

accept
aio_suspend
catclose
catgets
chmod
chown
chroot
clock_nanosleep
close
closedir
connect
curs_getch
dup
endgrent
epoll_wait
exec
execve
fallocate
fchdir
fchmod
fchown
fclose
fcntl
fflush
fgetc
fgetwc
flock
fopen
fork
fprintf
fputc
fputwc
freopen
fseek
fsetpos
fstatvfs
fsync
ftruncate
futex
getch
getgrent
getgrgid
getgrnam
getmsg
getpwent
getpwnam
getpwuid
has_key
inotify
ioctl
io_getevents
lchown
lio_listio
lockf
mknod
mq_open
mq_receive
mq_send
msgop
msgrcv
msgsnd
mvgetch
mvwgetch
nanosleep
open
pause
pclose
poll
posix_fallocate
posix_mem_offset
posix_trace_close
posix_trace_create
posix_trace_get_filter
posix_trace_getnext_event
posix_trace_start
posix_typed_mem_get_info
posix_typed_mem_open
pselect
pthread_atfork
pthread_attr_destroy
pthread_attr_getdetachstate
pthread_attr_getguardsize
pthread_attr_getinheritsched
pthread_attr_getschedparam
pthread_attr_getschedpolicy
pthread_attr_getscope
pthread_attr_getstack
pthread_attr_getstacksize
pthread_barrierattr_destroy
pthread_barrierattr_getpshared
pthread_barrier_destroy
pthread_barrier_wait
pthread_cancel
pthread_cleanup_pop
pthread_condattr_destroy
pthread_condattr_getclock
pthread_condattr_getpshared
pthread_cond_broadcast
pthread_cond_destroy
pthread_cond_init
pthread_cond_timedwait
pthread_create
pthread_detach
pthread_equal
pthread_getconcurrency
pthread_getschedparam
pthread_getspecific
pthread_join
pthread_key_create
pthread_key_delete
pthread_kill
pthread_mutexattr_destroy
pthread_mutexattr_getprioceiling
pthread_mutexattr_getprotocol
pthread_mutexattr_getpshared
pthread_mutexattr_getrobust
pthread_mutexattr_gettype
pthread_mutex_consistent
pthread_mutex_destroy
pthread_mutex_getprioceiling
pthread_mutex_lock
pthread_mutex_timedlock
pthread_once
pthread_rwlockattr_destroy
pthread_rwlockattr_getpshared
pthread_rwlock_destroy
pthread_rwlock_rdlock
pthread_rwlock_timedrdlock
pthread_rwlock_timedwrlock
pthread_rwlock_trywrlock
pthread_rwlock_unlock
pthreads
pthread_setcancelstate
pthread_setschedprio
pthread_sigmask
pthread_spin_destroy
pthread_spin_lock
pthread_spin_unlock
pthread_tryjoin_np
ptrace
putmsg
read
recv
recvfrom
recvmsg
request_key
restart_syscall
scanf
select
select_tut
semop
sem_open
sem_timedwait
sem_trywait
sem_wait
send
sendmsg
sendto
shm_open
sigaction
sighold
siginterrupt
signal
sigpause
sigsuspend
sigtimedwait
sigvec
sigwaitinfo
spu_run
stat
statfs
statvfs
strace
system
tcdrain
tcsetattr
tmpfile
truncate
ualarm
ungetch
unlink
usleep
ustat
wait
wait4
waitid
wgetch
write

Not interesting for Python:

Expect
HTTP::Tiny
Net::HTTP
POSIX
Tcl_AppendAllObjTypes
Tcl_ConvertToType
Tcl_GetObjType
Tcl_RegisterObjType
nfs
perlapio
perlfunc
perlipc
zmq_ctx_destroy
zmq_getsockopt
zmq_msg_recv
zmq_msg_send
zmq_poll
zmq_recv
zmq_recvmsg
zmq_send
zmq_sendmsg
zmq_setsockopt
zmq_term
zshmodules
msg238586 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-19 23:33
unittest_setitimer.patch: hack unittest.TestCase.run() to inject signals every 1 ms, send the first signal in 100 ms. It should help to detect functions which doesn't retry on EINTR.
msg238597 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-20 01:19
New issue #23715: handle EINTR in the signal module.
msg238644 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-20 11:12
pthread_* are false positive, they cannot fail with EINTR:

"""
Pthreads function return values
       Most pthreads functions return 0 on success, and an error number of failure.  Note that the pthreads functions do not set errno.  For each of the pthreads functions that can return an error, POSIX.1-2001 specifies that the function can never fail with the error EINTR.
"""

Example of pthread_create:
"""
The pthread_create() function shall not return an error code of [EINTR].
"""

pthread_atfork
pthread_attr_destroy
pthread_attr_getdetachstate
pthread_attr_getguardsize
pthread_attr_getinheritsched
pthread_attr_getschedparam
pthread_attr_getschedpolicy
pthread_attr_getscope
pthread_attr_getstack
pthread_attr_getstacksize
pthread_barrierattr_destroy
pthread_barrierattr_getpshared
pthread_barrier_destroy
pthread_barrier_wait
pthread_cancel
pthread_cleanup_pop
pthread_condattr_destroy
pthread_condattr_getclock
pthread_condattr_getpshared
pthread_cond_broadcast
pthread_cond_destroy
pthread_cond_init
pthread_cond_timedwait
pthread_create
pthread_detach
pthread_equal
pthread_getconcurrency
pthread_getschedparam
pthread_getspecific
pthread_join
pthread_key_create
pthread_key_delete
pthread_kill
pthread_mutexattr_destroy
pthread_mutexattr_getprioceiling
pthread_mutexattr_getprotocol
pthread_mutexattr_getpshared
pthread_mutexattr_getrobust
pthread_mutexattr_gettype
pthread_mutex_consistent
pthread_mutex_destroy
pthread_mutex_getprioceiling
pthread_mutex_lock
pthread_mutex_timedlock
pthread_once
pthread_rwlockattr_destroy
pthread_rwlockattr_getpshared
pthread_rwlock_destroy
pthread_rwlock_rdlock
pthread_rwlock_timedrdlock
pthread_rwlock_timedwrlock
pthread_rwlock_trywrlock
pthread_rwlock_unlock
pthreads
pthread_setcancelstate
pthread_setschedprio
pthread_sigmask
pthread_spin_destroy
pthread_spin_lock
pthread_spin_unlock
pthread_tryjoin_np
msg238653 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-20 12:10
New issue #23719: "port test_eintr to Windows".
msg238722 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2015-03-20 20:52
Well, all the syscalls which can blocki can fail with EINTR, so all
I:O related one.
msg239073 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-23 21:28
> Well, all the syscalls which can blocki can fail with EINTR, so all
> I:O related one.

This is also what I expect, but how do you explain that I'm unable to see os.stat() failing with EINTR? I'm testing on ext4 file system. May it occur with other file systems (ex: NFS)? Or maybe not on Linux, but on other platforms?
msg239077 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-23 21:51
fstat_not_eintr.py: run this script from a NFS share and unplug the network cable, wait, replug. Spoiler: fstat() hangs until the network is back, CTRL+c or setitimer() don't interrupt it.

By the way, it looks like the itimer is interrupted during fstat!

Extract of strace output:
---
<cable unplugged>

rt_sigreturn()                          = -1 EINTR (Interrupted system call)
select(0, NULL, NULL, NULL, {0, 100000}) = ? ERESTARTNOHAND (To be restarted if no handler)
--- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} ---
rt_sigreturn()                          = -1 EINTR (Interrupted system call)
write(1, "fstat...\n", 9)               = 9

fstat(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0

<cable replugged>

--- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} ---
rt_sigreturn()                          = 0
---

The fstat() took 3 seconds, I expected 30 SIGALRM, but none occurred during fstat().
msg239169 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2015-03-24 19:13
> fstat_not_eintr.py: run this script from a NFS share and unplug the network cable, wait, replug. Spoiler: fstat() hangs until the network is back, CTRL+c or setitimer() don't interrupt it.

You have to mount the share with the eintr option.

Getting EINTR on a local FS is probably more challenging.
msg239184 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-24 21:38
Charles-François Natali added the comment:
> You have to mount the share with the eintr option.

Oh, I didn't know this option. Unlikely, it looks like the option was
deprecated something like 7 years ago, in Linux kernel 2.6.25.

Related commits:
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/fs/nfs?id=d33e4dfeab5eb0c3c1359cc1a399f2f8b49e18de
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/fs/nfs?id=c381ad2cf2d5dcd3991bcc8a18fddd9d5c66ccaa

The patch on the NFS mailing list:
http://comments.gmane.org/gmane.linux.nfs/20011

> Getting EINTR on a local FS is probably more challenging.

Another option would be to play with FUSE. But I'm not convinced that
it's useful if fstat() never fails EINTR on filesystems used in
production.
msg239359 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-03-26 22:51
New changeset b7c0137cccbe by Victor Stinner in branch 'default':
Issue #23648: Document the PEP 475 in the "Porting to Python 3.5" section and
https://hg.python.org/cpython/rev/b7c0137cccbe
msg239641 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-30 20:36
inject_signals.patch: Quick-and-dirty patch to send signals (SIGALRM) every milliseconds when running unit tests.

On Linux, results are very good. Only multiprocessing fails because of the issue #23618 (socket connect) which has a pending patch. I hope that we will quickly reach a point where the full test suite completes without failure while sending millions of signals to it!
msg239777 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-04-01 09:44
New issue #23836: Enhance faulthandler to handle better reentrant calls.
msg239908 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-04-02 11:54
New issue #23851: _posixsubprocess retries close() on EINTR.
msg239909 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-04-02 11:56
New changeset 66e4ef9a7467 by Victor Stinner in branch 'default':
Issue #23648: Complete the list of modified functions for the PEP 475
https://hg.python.org/cpython/rev/66e4ef9a7467
msg239925 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-04-02 15:30
New issue #23853: handle EINTR in the ssl module.
msg240169 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * (Python triager) Date: 2015-04-06 18:36
> New changeset 66e4ef9a7467 by Victor Stinner in branch 'default':
> Issue #23648: Complete the list of modified functions for the PEP 475
> https://hg.python.org/cpython/rev/66e4ef9a7467
>
> +    * special cases: :func:`os.close` and :func:`os.dup2` now ignore
> +      :py:data:`~errno.EINTR` error, the syscall is not retried (see the PEP
> +      for the rationale)

But os.dup2 is not mentioned in PEP 475.
msg240184 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-04-06 21:30
> But os.dup2 is not mentioned in PEP 475.

Currently, What's in Python 3.5 is more complete than the PEP 475. Another example, the PEP doesn't mention signal.sigtimedwait(). The PEP should be updated when the implementation will be finished.
msg244063 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-05-25 22:55
I updated the list of modified functions in the PEP 475.

Except of the issue #23719, all other issues related to the PEP 475 have been fixed. It's time to close this meta issue.
History
Date User Action Args
2015-05-26 23:33:13vstinnersetstatus: open -> closed
2015-05-25 22:55:42vstinnersetresolution: fixed
dependencies: - PEP 475: port test_eintr to Windows
messages: + msg244063
2015-04-06 21:30:18vstinnersetmessages: + msg240184
2015-04-06 18:36:12Arfreversetnosy: + Arfrever
messages: + msg240169
2015-04-02 15:30:56vstinnersetdependencies: + PEP 475: handle EINTR in the ssl module
messages: + msg239925
2015-04-02 11:56:49python-devsetmessages: + msg239909
2015-04-02 11:54:58vstinnersetdependencies: + PEP 475: _posixsubprocess retries close() on EINTR
messages: + msg239908
2015-04-01 09:45:05vstinnersetdependencies: + PEP 475: Enhance faulthandler to handle better reentrant calls
2015-04-01 09:44:40vstinnersetmessages: + msg239777
2015-03-30 20:36:51vstinnersetfiles: + inject_signals.patch

messages: + msg239641
2015-03-26 22:51:08python-devsetnosy: + python-dev
messages: + msg239359
2015-03-24 21:38:18vstinnersetmessages: + msg239184
2015-03-24 19:13:44neologixsetmessages: + msg239169
2015-03-23 21:51:24vstinnersetfiles: + fstat_not_eintr.py

messages: + msg239077
2015-03-23 21:28:41vstinnersetmessages: + msg239073
2015-03-20 20:52:43neologixsetmessages: + msg238722
2015-03-20 12:10:19vstinnersetdependencies: + PEP 475: port test_eintr to Windows
messages: + msg238653
2015-03-20 11:12:08vstinnersetmessages: + msg238644
2015-03-20 01:19:05vstinnersetdependencies: + PEP 475: handle EINTR in the signal module
messages: + msg238597
2015-03-19 23:33:43vstinnersetfiles: + unittest_setitimer.patch
keywords: + patch
messages: + msg238586
2015-03-19 16:49:51vstinnersetmessages: + msg238541
2015-03-19 14:39:52vstinnersetmessages: + msg238529
2015-03-19 13:41:20vstinnersetdependencies: + PEP 475: os.urandom() doesn't handle EINTR correctly, PEP 475: Add _Py_read() and _Py_write() functions, Refactor ossaudiodev: use _Py_read and _Py_write with the Py_buffer
messages: + msg238521
2015-03-18 09:48:09vstinnersetmessages: + msg238394
2015-03-18 09:47:19vstinnersetmessages: + msg238393
2015-03-18 09:23:07vstinnersetfiles: + test_eintr.py

messages: + msg238391
2015-03-17 22:39:59vstinnersetmessages: + msg238355
2015-03-12 15:51:59vstinnersetnosy: + neologix
2015-03-12 15:51:50vstinnersetdependencies: + PEP 475 - EINTR handling, PEP 475: handle EINTR in the select and selectors module, PEP 475: handle EINTR in the socket module (connect), PEP 475: handle EINTR in the time module, retry sleep()
2015-03-12 15:51:26vstinnercreate