classification
Title: io.open('/dev/stdout', 'a') raises OSError with errno=ESPIPE
Type: behavior Stage: resolved
Components: IO Versions: Python 3.8, Python 3.7, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: ZackerySpytz, benjamin.peterson, hathawsh, javabrett, martin.panter, r.david.murray, skip.montanaro, vstinner, ztane
Priority: normal Keywords: patch

Created on 2016-08-19 20:11 by hathawsh, last changed 2019-11-12 23:54 by benjamin.peterson. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 13254 closed ZackerySpytz, 2019-05-11 21:50
PR 17112 merged benjamin.peterson, 2019-11-12 05:45
PR 17136 merged benjamin.peterson, 2019-11-12 22:56
PR 17137 merged benjamin.peterson, 2019-11-12 22:57
Messages (15)
msg273158 - (view) Author: Shane Hathaway (hathawsh) Date: 2016-08-19 20:11
With Python 2, the following call worked:

open('/dev/stdout', 'a')

Users of Supervisor have been depending on that for a long time. 

With Python 3.5, this is what happens:

>>> open('/dev/stdout', 'a')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 29] Illegal seek

Presumably, this happens because Python 3 opens the file in 'w' mode and seeks to the end, while Python 2 passed the 'a' flag directly to the underlying C library; the underlying library is apparently smart enough to treat 'a' and 'w' identically when opening character device files and FIFOs.

It's a nasty little surprise for those upgrading from Python 2.
msg273169 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-08-20 00:15
The origin of this seems to be r68835 and Issue 5008. Victor mentioned imitating the Gnu C library. Maybe there is a better way that also supports non-seekable files better, perhaps handle ESPIPE without failing.

This also affects Python 2, if you consider io.open() or io.FileIO directly.
msg273171 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-08-20 00:31
Syscalls made by open("/dev/stdout", "a") in Python 2:

  open("/dev/stdout", O_WRONLY|O_CREAT|O_APPEND, 0666) = 3
  lseek(3, 0, SEEK_END)                   = -1 ESPIPE (Illegal seek)
  fstat(3, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0

I used this comand:

  $ strace -o trace python2 -c 'import os; os.uname();os.uname();os.uname(); f=open("/dev/stdout", "a"); os.uname(); f.close()'

It looks like the C library simply ignores ESPIPE on lseek(fd, 0, SEEK_END) when opening a file in append mode:
https://sourceware.org/git/?p=glibc.git;a=blob;f=libio/fileops.c;h=13157354272ff9ab1832d4a619a81f05898fcd69;hb=HEAD#l242

  if ((read_write & _IO_IS_APPENDING) && (read_write & _IO_NO_READS))
    if (_IO_SEEKOFF (fp, (_IO_off64_t)0, _IO_seek_end, _IOS_INPUT|_IOS_OUTPUT)
	== _IO_pos_BAD && errno != ESPIPE)
      {
	close_not_cancel (fdesc);
	return NULL;
      }

from _IO_file_open() file operation.
msg273172 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-08-20 00:32
> Users of Supervisor have been depending on that for a long time. 

Hum, the workaround is very simple no? Just open /dev/stdout with mode "w", no?
msg273204 - (view) Author: Antti Haapala (ztane) * Date: 2016-08-20 11:51
Presumably the case was that a *named* log file is opened with 'a' mode, and one could pass '/dev/stdout' just like any other name of a file, and it did work, but not in Python 3.5.
msg273207 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-08-20 12:36
Antti Haapala added the comment:
> Presumably the case was that a *named* log file is opened with 'a' mode, and one could pass '/dev/stdout' just like any other name of a file, and it did work, but not in Python 3.5.

Oh ok, in this case, you need something smarter like:

mode = 'a' if not stat.S_ISCHR(os.stat(filename).st_mode) else 'w'
fp = open(filename, mode)

or something like (emulate append mode, but catch ESPIPE):

fp = open(filename, 'w')
try:
  fp.seek(0, os.SEEK_END)
except OSError as exc:
  if exc.errno != errno.ESPIPE: raise
msg273223 - (view) Author: Shane Hathaway (hathawsh) Date: 2016-08-20 16:35
Thanks for the analysis. I have already started a pull request to fix this in Supervisor, but I also thought this change to Python might be gratuitous and accidental. It seems like open('/dev/stdout', 'a') ought to work the same as Python 2. If not, the Python documentation should warn people that using 'a' with character devices and FIFOs will cause an OSError.
msg273225 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-08-20 16:57
And the fact that python deviates from posix in this regard seems like a bug to me, since the posix behavior is, as noted, very useful.  Can't we handle ESPIPE like the C library does, instead of forcing users to learn that arcane incantation?
msg273262 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-08-21 03:13
Handling ESPIPE for append mode seems reasonable to me, even as a bug fix for existing versions.

But there is a similar problem with "r+" and "w+" modes and unseekable files (unless buffering=0). See Issue 20074. So we can’t say in general that Python 3 faithfully implements all aspects of Python 2’s / C’s / Posix’s file modes.
msg273274 - (view) Author: Antti Haapala (ztane) * Date: 2016-08-21 07:23
Yeah, it definitely is a bug in CPython. open(mode='a') should always append to the end of the given file. 

If you're writing an append-only text log to some file-like object, that's the mode you use, not some version/platform/filesystem specific voodoo to find out what's the least incorrect way to work around Python implementation deficiencies.
msg315005 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2018-04-05 19:40
I was bitten by this porting a system from Python 2.7 to 3.6. "/dev/stderr" is a very nice default for logfiles. Users will frequently override the default, so you really want to open the logfile in append mode. Having to jump through hoops to avoid blasting a user's logfile is kinda dumb, and as others have pointed out, error-prone.

os.open works just fine with O_WRONLY|O_APPEND as the flags. /dev/null can be opened in append mode, but not /dev/stderr or /dev/stdout.
msg353343 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2019-09-27 03:41
If we were starting from zero, I would suggest omitting the lseek() after open. Sure .tell() might report 0 on newly opened files, but avoiding unnecessary io operations is nicer.
msg356503 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2019-11-12 22:51
New changeset 74fa9f723f700a342e582b5ad4b51a2c4801cd1c by Benjamin Peterson in branch 'master':
closes bpo-27805: Ignore ESPIPE in initializing seek of append-mode files. (GH-17112)
https://github.com/python/cpython/commit/74fa9f723f700a342e582b5ad4b51a2c4801cd1c
msg356506 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2019-11-12 23:34
New changeset b8b3e4377ec38c7d64570afabbd0923a51f0666c by Benjamin Peterson in branch '3.7':
[3.7] closes bpo-27805: Ignore ESPIPE in initializing seek of append-mode files. (GH-17137)
https://github.com/python/cpython/commit/b8b3e4377ec38c7d64570afabbd0923a51f0666c
msg356508 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2019-11-12 23:54
New changeset 9788f97bf69230ec6b38a483c90e88828eba9a1b by Benjamin Peterson in branch '3.8':
[3.8] closes bpo-27805: Ignore ESPIPE in initializing seek of append-mode files. (GH-17136)
https://github.com/python/cpython/commit/9788f97bf69230ec6b38a483c90e88828eba9a1b
History
Date User Action Args
2019-11-12 23:54:27benjamin.petersonsetmessages: + msg356508
2019-11-12 23:34:50benjamin.petersonsetmessages: + msg356506
2019-11-12 22:57:39benjamin.petersonsetpull_requests: + pull_request16647
2019-11-12 22:56:17benjamin.petersonsetpull_requests: + pull_request16646
2019-11-12 22:51:45benjamin.petersonsetstatus: open -> closed
resolution: fixed
messages: + msg356503

stage: patch review -> resolved
2019-11-12 05:45:38benjamin.petersonsetpull_requests: + pull_request16619
2019-09-27 03:41:57benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg353343
2019-05-11 21:51:52ZackerySpytzsetnosy: + ZackerySpytz

versions: + Python 3.7, Python 3.8, - Python 3.5, Python 3.6
2019-05-11 21:50:42ZackerySpytzsetkeywords: + patch
stage: test needed -> patch review
pull_requests: + pull_request13166
2018-07-21 12:56:54javabrettsetnosy: + javabrett
2018-04-05 19:40:33skip.montanarosetnosy: + skip.montanaro
messages: + msg315005
2016-08-26 22:30:05martin.pantersettitle: os.open('/dev/stdout', 'a') raises OSError with errno=ESPIPE -> io.open('/dev/stdout', 'a') raises OSError with errno=ESPIPE
2016-08-26 18:03:12terry.reedysettitle: In Python 3, open('/dev/stdout', 'a') raises OSError with errno=ESPIPE -> os.open('/dev/stdout', 'a') raises OSError with errno=ESPIPE
type: behavior
stage: test needed
2016-08-21 07:23:29ztanesetmessages: + msg273274
2016-08-21 03:13:01martin.pantersetmessages: + msg273262
2016-08-20 16:57:54r.david.murraysetversions: + Python 3.6
2016-08-20 16:57:45r.david.murraysetnosy: + r.david.murray
messages: + msg273225
2016-08-20 16:35:13hathawshsetmessages: + msg273223
2016-08-20 12:36:06vstinnersetmessages: + msg273207
2016-08-20 11:51:49ztanesetnosy: + ztane
messages: + msg273204
2016-08-20 00:32:24vstinnersetmessages: + msg273172
2016-08-20 00:31:52vstinnersetmessages: + msg273171
2016-08-20 00:15:46martin.pantersetnosy: + vstinner, martin.panter

messages: + msg273169
versions: + Python 2.7
2016-08-19 20:11:12hathawshcreate