msg273158 - (view) |
Author: Shane Hathaway (hathawsh) |
Date: 2016-08-19 20:11 |
With Python 2, the following call worked:
open('/dev/stdout', 'a')
Users of Supervisor have been depending on that for a long time.
With Python 3.5, this is what happens:
>>> open('/dev/stdout', 'a')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OSError: [Errno 29] Illegal seek
Presumably, this happens because Python 3 opens the file in 'w' mode and seeks to the end, while Python 2 passed the 'a' flag directly to the underlying C library; the underlying library is apparently smart enough to treat 'a' and 'w' identically when opening character device files and FIFOs.
It's a nasty little surprise for those upgrading from Python 2.
|
msg273169 - (view) |
Author: Martin Panter (martin.panter) *  |
Date: 2016-08-20 00:15 |
The origin of this seems to be r68835 and Issue 5008. Victor mentioned imitating the Gnu C library. Maybe there is a better way that also supports non-seekable files better, perhaps handle ESPIPE without failing.
This also affects Python 2, if you consider io.open() or io.FileIO directly.
|
msg273171 - (view) |
Author: STINNER Victor (vstinner) *  |
Date: 2016-08-20 00:31 |
Syscalls made by open("/dev/stdout", "a") in Python 2:
open("/dev/stdout", O_WRONLY|O_CREAT|O_APPEND, 0666) = 3
lseek(3, 0, SEEK_END) = -1 ESPIPE (Illegal seek)
fstat(3, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0
I used this comand:
$ strace -o trace python2 -c 'import os; os.uname();os.uname();os.uname(); f=open("/dev/stdout", "a"); os.uname(); f.close()'
It looks like the C library simply ignores ESPIPE on lseek(fd, 0, SEEK_END) when opening a file in append mode:
https://sourceware.org/git/?p=glibc.git;a=blob;f=libio/fileops.c;h=13157354272ff9ab1832d4a619a81f05898fcd69;hb=HEAD#l242
if ((read_write & _IO_IS_APPENDING) && (read_write & _IO_NO_READS))
if (_IO_SEEKOFF (fp, (_IO_off64_t)0, _IO_seek_end, _IOS_INPUT|_IOS_OUTPUT)
== _IO_pos_BAD && errno != ESPIPE)
{
close_not_cancel (fdesc);
return NULL;
}
from _IO_file_open() file operation.
|
msg273172 - (view) |
Author: STINNER Victor (vstinner) *  |
Date: 2016-08-20 00:32 |
> Users of Supervisor have been depending on that for a long time.
Hum, the workaround is very simple no? Just open /dev/stdout with mode "w", no?
|
msg273204 - (view) |
Author: Antti Haapala (ztane) * |
Date: 2016-08-20 11:51 |
Presumably the case was that a *named* log file is opened with 'a' mode, and one could pass '/dev/stdout' just like any other name of a file, and it did work, but not in Python 3.5.
|
msg273207 - (view) |
Author: STINNER Victor (vstinner) *  |
Date: 2016-08-20 12:36 |
Antti Haapala added the comment:
> Presumably the case was that a *named* log file is opened with 'a' mode, and one could pass '/dev/stdout' just like any other name of a file, and it did work, but not in Python 3.5.
Oh ok, in this case, you need something smarter like:
mode = 'a' if not stat.S_ISCHR(os.stat(filename).st_mode) else 'w'
fp = open(filename, mode)
or something like (emulate append mode, but catch ESPIPE):
fp = open(filename, 'w')
try:
fp.seek(0, os.SEEK_END)
except OSError as exc:
if exc.errno != errno.ESPIPE: raise
|
msg273223 - (view) |
Author: Shane Hathaway (hathawsh) |
Date: 2016-08-20 16:35 |
Thanks for the analysis. I have already started a pull request to fix this in Supervisor, but I also thought this change to Python might be gratuitous and accidental. It seems like open('/dev/stdout', 'a') ought to work the same as Python 2. If not, the Python documentation should warn people that using 'a' with character devices and FIFOs will cause an OSError.
|
msg273225 - (view) |
Author: R. David Murray (r.david.murray) *  |
Date: 2016-08-20 16:57 |
And the fact that python deviates from posix in this regard seems like a bug to me, since the posix behavior is, as noted, very useful. Can't we handle ESPIPE like the C library does, instead of forcing users to learn that arcane incantation?
|
msg273262 - (view) |
Author: Martin Panter (martin.panter) *  |
Date: 2016-08-21 03:13 |
Handling ESPIPE for append mode seems reasonable to me, even as a bug fix for existing versions.
But there is a similar problem with "r+" and "w+" modes and unseekable files (unless buffering=0). See Issue 20074. So we can’t say in general that Python 3 faithfully implements all aspects of Python 2’s / C’s / Posix’s file modes.
|
msg273274 - (view) |
Author: Antti Haapala (ztane) * |
Date: 2016-08-21 07:23 |
Yeah, it definitely is a bug in CPython. open(mode='a') should always append to the end of the given file.
If you're writing an append-only text log to some file-like object, that's the mode you use, not some version/platform/filesystem specific voodoo to find out what's the least incorrect way to work around Python implementation deficiencies.
|
msg315005 - (view) |
Author: Skip Montanaro (skip.montanaro) *  |
Date: 2018-04-05 19:40 |
I was bitten by this porting a system from Python 2.7 to 3.6. "/dev/stderr" is a very nice default for logfiles. Users will frequently override the default, so you really want to open the logfile in append mode. Having to jump through hoops to avoid blasting a user's logfile is kinda dumb, and as others have pointed out, error-prone.
os.open works just fine with O_WRONLY|O_APPEND as the flags. /dev/null can be opened in append mode, but not /dev/stderr or /dev/stdout.
|
msg353343 - (view) |
Author: Benjamin Peterson (benjamin.peterson) *  |
Date: 2019-09-27 03:41 |
If we were starting from zero, I would suggest omitting the lseek() after open. Sure .tell() might report 0 on newly opened files, but avoiding unnecessary io operations is nicer.
|
msg356503 - (view) |
Author: Benjamin Peterson (benjamin.peterson) *  |
Date: 2019-11-12 22:51 |
New changeset 74fa9f723f700a342e582b5ad4b51a2c4801cd1c by Benjamin Peterson in branch 'master':
closes bpo-27805: Ignore ESPIPE in initializing seek of append-mode files. (GH-17112)
https://github.com/python/cpython/commit/74fa9f723f700a342e582b5ad4b51a2c4801cd1c
|
msg356506 - (view) |
Author: Benjamin Peterson (benjamin.peterson) *  |
Date: 2019-11-12 23:34 |
New changeset b8b3e4377ec38c7d64570afabbd0923a51f0666c by Benjamin Peterson in branch '3.7':
[3.7] closes bpo-27805: Ignore ESPIPE in initializing seek of append-mode files. (GH-17137)
https://github.com/python/cpython/commit/b8b3e4377ec38c7d64570afabbd0923a51f0666c
|
msg356508 - (view) |
Author: Benjamin Peterson (benjamin.peterson) *  |
Date: 2019-11-12 23:54 |
New changeset 9788f97bf69230ec6b38a483c90e88828eba9a1b by Benjamin Peterson in branch '3.8':
[3.8] closes bpo-27805: Ignore ESPIPE in initializing seek of append-mode files. (GH-17136)
https://github.com/python/cpython/commit/9788f97bf69230ec6b38a483c90e88828eba9a1b
|
|
Date |
User |
Action |
Args |
2022-04-11 14:58:35 | admin | set | github: 71992 |
2019-11-12 23:54:27 | benjamin.peterson | set | messages:
+ msg356508 |
2019-11-12 23:34:50 | benjamin.peterson | set | messages:
+ msg356506 |
2019-11-12 22:57:39 | benjamin.peterson | set | pull_requests:
+ pull_request16647 |
2019-11-12 22:56:17 | benjamin.peterson | set | pull_requests:
+ pull_request16646 |
2019-11-12 22:51:45 | benjamin.peterson | set | status: open -> closed resolution: fixed messages:
+ msg356503
stage: patch review -> resolved |
2019-11-12 05:45:38 | benjamin.peterson | set | pull_requests:
+ pull_request16619 |
2019-09-27 03:41:57 | benjamin.peterson | set | nosy:
+ benjamin.peterson messages:
+ msg353343
|
2019-05-11 21:51:52 | ZackerySpytz | set | nosy:
+ ZackerySpytz
versions:
+ Python 3.7, Python 3.8, - Python 3.5, Python 3.6 |
2019-05-11 21:50:42 | ZackerySpytz | set | keywords:
+ patch stage: test needed -> patch review pull_requests:
+ pull_request13166 |
2018-07-21 12:56:54 | javabrett | set | nosy:
+ javabrett
|
2018-04-05 19:40:33 | skip.montanaro | set | nosy:
+ skip.montanaro messages:
+ msg315005
|
2016-08-26 22:30:05 | martin.panter | set | title: os.open('/dev/stdout', 'a') raises OSError with errno=ESPIPE -> io.open('/dev/stdout', 'a') raises OSError with errno=ESPIPE |
2016-08-26 18:03:12 | terry.reedy | set | title: In Python 3, open('/dev/stdout', 'a') raises OSError with errno=ESPIPE -> os.open('/dev/stdout', 'a') raises OSError with errno=ESPIPE type: behavior stage: test needed |
2016-08-21 07:23:29 | ztane | set | messages:
+ msg273274 |
2016-08-21 03:13:01 | martin.panter | set | messages:
+ msg273262 |
2016-08-20 16:57:54 | r.david.murray | set | versions:
+ Python 3.6 |
2016-08-20 16:57:45 | r.david.murray | set | nosy:
+ r.david.murray messages:
+ msg273225
|
2016-08-20 16:35:13 | hathawsh | set | messages:
+ msg273223 |
2016-08-20 12:36:06 | vstinner | set | messages:
+ msg273207 |
2016-08-20 11:51:49 | ztane | set | nosy:
+ ztane messages:
+ msg273204
|
2016-08-20 00:32:24 | vstinner | set | messages:
+ msg273172 |
2016-08-20 00:31:52 | vstinner | set | messages:
+ msg273171 |
2016-08-20 00:15:46 | martin.panter | set | nosy:
+ vstinner, martin.panter
messages:
+ msg273169 versions:
+ Python 2.7 |
2016-08-19 20:11:12 | hathawsh | create | |