Title:'/dev/stdout', 'a') raises OSError with errno=ESPIPE
Type: behavior Stage: patch review
Components: IO Versions: Python 3.8, Python 3.7, Python 2.7
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: ZackerySpytz, hathawsh, javabrett, martin.panter, r.david.murray, skip.montanaro, vstinner, ztane
Priority: normal Keywords: patch

Created on 2016-08-19 20:11 by hathawsh, last changed 2019-05-11 21:51 by ZackerySpytz.

Pull Requests
URL Status Linked Edit
PR 13254 open ZackerySpytz, 2019-05-11 21:50
Messages (11)
msg273158 - (view) Author: Shane Hathaway (hathawsh) Date: 2016-08-19 20:11
With Python 2, the following call worked:

open('/dev/stdout', 'a')

Users of Supervisor have been depending on that for a long time. 

With Python 3.5, this is what happens:

>>> open('/dev/stdout', 'a')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 29] Illegal seek

Presumably, this happens because Python 3 opens the file in 'w' mode and seeks to the end, while Python 2 passed the 'a' flag directly to the underlying C library; the underlying library is apparently smart enough to treat 'a' and 'w' identically when opening character device files and FIFOs.

It's a nasty little surprise for those upgrading from Python 2.
msg273169 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-08-20 00:15
The origin of this seems to be r68835 and Issue 5008. Victor mentioned imitating the Gnu C library. Maybe there is a better way that also supports non-seekable files better, perhaps handle ESPIPE without failing.

This also affects Python 2, if you consider or io.FileIO directly.
msg273171 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-08-20 00:31
Syscalls made by open("/dev/stdout", "a") in Python 2:

  open("/dev/stdout", O_WRONLY|O_CREAT|O_APPEND, 0666) = 3
  lseek(3, 0, SEEK_END)                   = -1 ESPIPE (Illegal seek)
  fstat(3, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0

I used this comand:

  $ strace -o trace python2 -c 'import os; os.uname();os.uname();os.uname(); f=open("/dev/stdout", "a"); os.uname(); f.close()'

It looks like the C library simply ignores ESPIPE on lseek(fd, 0, SEEK_END) when opening a file in append mode:;a=blob;f=libio/fileops.c;h=13157354272ff9ab1832d4a619a81f05898fcd69;hb=HEAD#l242

  if ((read_write & _IO_IS_APPENDING) && (read_write & _IO_NO_READS))
    if (_IO_SEEKOFF (fp, (_IO_off64_t)0, _IO_seek_end, _IOS_INPUT|_IOS_OUTPUT)
	== _IO_pos_BAD && errno != ESPIPE)
	close_not_cancel (fdesc);
	return NULL;

from _IO_file_open() file operation.
msg273172 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-08-20 00:32
> Users of Supervisor have been depending on that for a long time. 

Hum, the workaround is very simple no? Just open /dev/stdout with mode "w", no?
msg273204 - (view) Author: Antti Haapala (ztane) * Date: 2016-08-20 11:51
Presumably the case was that a *named* log file is opened with 'a' mode, and one could pass '/dev/stdout' just like any other name of a file, and it did work, but not in Python 3.5.
msg273207 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-08-20 12:36
Antti Haapala added the comment:
> Presumably the case was that a *named* log file is opened with 'a' mode, and one could pass '/dev/stdout' just like any other name of a file, and it did work, but not in Python 3.5.

Oh ok, in this case, you need something smarter like:

mode = 'a' if not stat.S_ISCHR(os.stat(filename).st_mode) else 'w'
fp = open(filename, mode)

or something like (emulate append mode, but catch ESPIPE):

fp = open(filename, 'w')
try:, os.SEEK_END)
except OSError as exc:
  if exc.errno != errno.ESPIPE: raise
msg273223 - (view) Author: Shane Hathaway (hathawsh) Date: 2016-08-20 16:35
Thanks for the analysis. I have already started a pull request to fix this in Supervisor, but I also thought this change to Python might be gratuitous and accidental. It seems like open('/dev/stdout', 'a') ought to work the same as Python 2. If not, the Python documentation should warn people that using 'a' with character devices and FIFOs will cause an OSError.
msg273225 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-08-20 16:57
And the fact that python deviates from posix in this regard seems like a bug to me, since the posix behavior is, as noted, very useful.  Can't we handle ESPIPE like the C library does, instead of forcing users to learn that arcane incantation?
msg273262 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-08-21 03:13
Handling ESPIPE for append mode seems reasonable to me, even as a bug fix for existing versions.

But there is a similar problem with "r+" and "w+" modes and unseekable files (unless buffering=0). See Issue 20074. So we can’t say in general that Python 3 faithfully implements all aspects of Python 2’s / C’s / Posix’s file modes.
msg273274 - (view) Author: Antti Haapala (ztane) * Date: 2016-08-21 07:23
Yeah, it definitely is a bug in CPython. open(mode='a') should always append to the end of the given file. 

If you're writing an append-only text log to some file-like object, that's the mode you use, not some version/platform/filesystem specific voodoo to find out what's the least incorrect way to work around Python implementation deficiencies.
msg315005 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2018-04-05 19:40
I was bitten by this porting a system from Python 2.7 to 3.6. "/dev/stderr" is a very nice default for logfiles. Users will frequently override the default, so you really want to open the logfile in append mode. Having to jump through hoops to avoid blasting a user's logfile is kinda dumb, and as others have pointed out, error-prone. works just fine with O_WRONLY|O_APPEND as the flags. /dev/null can be opened in append mode, but not /dev/stderr or /dev/stdout.
Date User Action Args
2019-05-11 21:51:52ZackerySpytzsetnosy: + ZackerySpytz

versions: + Python 3.7, Python 3.8, - Python 3.5, Python 3.6
2019-05-11 21:50:42ZackerySpytzsetkeywords: + patch
stage: test needed -> patch review
pull_requests: + pull_request13166
2018-07-21 12:56:54javabrettsetnosy: + javabrett
2018-04-05 19:40:33skip.montanarosetnosy: + skip.montanaro
messages: + msg315005
2016-08-26 22:30:05martin.pantersettitle:'/dev/stdout', 'a') raises OSError with errno=ESPIPE ->'/dev/stdout', 'a') raises OSError with errno=ESPIPE
2016-08-26 18:03:12terry.reedysettitle: In Python 3, open('/dev/stdout', 'a') raises OSError with errno=ESPIPE ->'/dev/stdout', 'a') raises OSError with errno=ESPIPE
type: behavior
stage: test needed
2016-08-21 07:23:29ztanesetmessages: + msg273274
2016-08-21 03:13:01martin.pantersetmessages: + msg273262
2016-08-20 16:57:54r.david.murraysetversions: + Python 3.6
2016-08-20 16:57:45r.david.murraysetnosy: + r.david.murray
messages: + msg273225
2016-08-20 16:35:13hathawshsetmessages: + msg273223
2016-08-20 12:36:06vstinnersetmessages: + msg273207
2016-08-20 11:51:49ztanesetnosy: + ztane
messages: + msg273204
2016-08-20 00:32:24vstinnersetmessages: + msg273172
2016-08-20 00:31:52vstinnersetmessages: + msg273171
2016-08-20 00:15:46martin.pantersetnosy: + vstinner, martin.panter

messages: + msg273169
versions: + Python 2.7
2016-08-19 20:11:12hathawshcreate