classification
Title: platform.uname() not EINTR safe
Type: behavior Stage: resolved
Components: Versions: Python 3.1, Python 3.2, Python 3.3, Python 3.4, Python 3.5, Python 2.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder: handle EINTR in the stdlib
View: 18885
Assigned To: Nosy List: Tor.Colvin, lemburg, neologix, stefanoborini, vstinner
Priority: normal Keywords: patch

Created on 2014-06-15 17:32 by Tor.Colvin, last changed 2014-07-22 20:16 by neologix. This issue is now closed.

Files
File name Uploaded Description Edit
platform.py.fix Tor.Colvin, 2014-06-15 17:32 review
platform_syscmd_uname.patch vstinner, 2014-06-16 16:29
Messages (12)
msg220654 - (view) Author: Tor Colvin (Tor.Colvin) Date: 2014-06-15 17:32
platform.uname() periodically spews EINTR errors on mac - so use subproces.communicate() which is EINTR safe after http://bugs.python.org/issue1068268

Calling f.read() on naked os.open() output can produce "IOError: [Errno 4] Interrupted system call". Attached is a suggested fix.
msg220738 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-06-16 16:29
Here is a different but similar patch for _syscmd_uname():
- use os.fsdecode() to use the locale encoding instead of latin-1
- use subprocess.DEVNULL to redirect errors to /dev/null
- use "with proc:" to ensure that the subprocesss is cleaned in case of error
msg220769 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-06-16 21:30
> Calling f.read() on naked os.open() output can produce "IOError: [Errno 4] Interrupted system call". Attached is a suggested fix.

I cannot reproduce your issue. What is your exact Python version and OS?

I tried to run 100 threads calling 100 times platform._syscmd_uname('-p'). I didn't reproduce the issue on Python 2.7 or 3.5.

---
import platform
import threading

def test():
    for x in range(100):
        p = platform._syscmd_uname('-p')

threads = [threading.Thread(target=test) for n in range(100)]
for thread in threads:
    thread.start()
for thread in threads:
    thread.join()
---
msg220774 - (view) Author: Tor Colvin (Tor.Colvin) Date: 2014-06-16 21:53
2.7.6 or 3.4.1 with OS X 10.8. I can only reproduce it an automated test environment when the load is high on the machine. I've also seen it with 10.6/10.7. It's definitely inconsistently reproducible.
msg220811 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2014-06-17 09:18
I'm not sure whether using os.fsencoding() is a good idea. The encoding used by uname is not defined anywhere and it's possible that Python using a wrong encoding may cause the call to fail (e.g. in case the host name includes non-ASCII chars).

Then again: _syscmd_uname() is currently only used to determine the processor, which will most likely always be ASCII.

BTW: It would be good to replace all other calls to os.popen() by subprocess as well. platform.py in Python 3 no longer has the requirement to stay compatible with earlier Python 2 releases.
msg220812 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-06-17 09:21
"I'm not sure whether using os.fsencoding() is a good idea. The encoding used by uname is not defined anywhere and it's possible that Python using a wrong encoding may cause the call to fail (e.g. in case the host name includes non-ASCII chars)."

I also expect ASCII, but I prefer os.fsdecode() because it's the encoding used almost everywhere in Python 3. It's for example the encoding (and error handler) currently used by _syscmd_uname() indirectly by TextIOWrapper.
msg220813 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2014-06-17 09:22
On 17.06.2014 11:21, STINNER Victor wrote:
> 
> STINNER Victor added the comment:
> 
> "I'm not sure whether using os.fsencoding() is a good idea. The encoding used by uname is not defined anywhere and it's possible that Python using a wrong encoding may cause the call to fail (e.g. in case the host name includes non-ASCII chars)."
> 
> I also expect ASCII, but I prefer os.fsdecode() because it's the encoding used almost everywhere in Python 3. It's for example the encoding (and error handler) currently used by _syscmd_uname() indirectly by TextIOWrapper.

Ok.
msg220814 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-06-17 09:25
"2.7.6 or 3.4.1 with OS X 10.8. I can only reproduce it an automated test environment when the load is high on the machine. I've also seen it with 10.6/10.7. It's definitely inconsistently reproducible."

I'm surprised that the Python read() method doesn't handle EINTR internally.

I'm in favor of handling EINTR internally almost everywhere, I mean in the Python modules implemented in the C, not in each call using these C methods. "handling EINTR" means calling PyErr_CheckSignals() which may raises a Python exception (ex: KeyboardInterrupt).
msg220923 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2014-06-18 07:03
> I'm surprised that the Python read() method doesn't handle EINTR internally.
>
> I'm in favor of handling EINTR internally almost everywhere, I mean in the Python modules implemented in the C, not in each call using these C methods. "handling EINTR" means calling PyErr_CheckSignals() which may raises a Python exception (ex: KeyboardInterrupt).

It's good, we all agree on that.
I think the different EINTR-related reports should be closed as #18885
duplicates.
Then we need a patch for #18885: my idea was to write a pair of macros
similar to Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS that would
handle the retry logic, instead of having to check for EINTR and
calling Py_CheckPendingSignals() everywhere.
msg222551 - (view) Author: Stefano Borini (stefanoborini) Date: 2014-07-08 09:40
You can't use subprocess. platform is used during build. subprocess needs select, but select is a compiled module and at that specific time in the build process is not compiled yet.
msg222552 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2014-07-08 09:50
On 08.07.2014 11:40, Stefano Borini wrote:
> 
> You can't use subprocess. platform is used during build. subprocess needs select, but select is a compiled module and at that specific time in the build process is not compiled yet.

Good point :-)
msg222558 - (view) Author: Stefano Borini (stefanoborini) Date: 2014-07-08 13:10
Wouldn't it make sense to use the same strategy used in the subprocess fix (that is, retry?). See http://hg.python.org/cpython/rev/6e664bcc958d/
History
Date User Action Args
2014-07-22 20:16:28neologixsetstatus: open -> closed
superseder: handle EINTR in the stdlib
resolution: duplicate
stage: resolved
2014-07-08 13:10:01stefanoborinisetmessages: + msg222558
2014-07-08 09:50:26lemburgsetmessages: + msg222552
2014-07-08 09:40:57stefanoborinisetnosy: + stefanoborini
messages: + msg222551
2014-06-18 07:03:01neologixsetmessages: + msg220923
2014-06-17 09:25:18vstinnersetmessages: + msg220814
2014-06-17 09:22:42lemburgsetmessages: + msg220813
2014-06-17 09:21:27vstinnersetmessages: + msg220812
2014-06-17 09:18:50lemburgsetmessages: + msg220811
2014-06-17 00:24:47pitrousetnosy: + neologix
2014-06-16 21:53:35Tor.Colvinsetmessages: + msg220774
2014-06-16 21:30:05vstinnersetmessages: + msg220769
2014-06-16 16:29:33vstinnersetfiles: + platform_syscmd_uname.patch

nosy: + vstinner
messages: + msg220738

keywords: + patch
2014-06-15 20:43:29ned.deilysetnosy: + lemburg
2014-06-15 17:32:30Tor.Colvincreate