With Python 2, the following call worked:

open('/dev/stdout', 'a')

Users of Supervisor have been depending on that for a long time.

With Python 3.5, this is what happens:

>>> open('/dev/stdout', 'a')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 29] Illegal seek

Presumably, this happens because Python 3 opens the file in 'w' mode and seeks to the end, while Python 2 passed the 'a' flag directly to the underlying C library; the underlying library is apparently smart enough to treat 'a' and 'w' identically when opening character device files and FIFOs.

It's a nasty little surprise for those upgrading from Python 2.

The origin of this seems to be r68835 and bpo-5008. Victor mentioned imitating the Gnu C library. Maybe there is a better way that also supports non-seekable files better, perhaps handle ESPIPE without failing.

This also affects Python 2, if you consider io.open() or io.FileIO directly.

Syscalls made by open("/dev/stdout", "a") in Python 2:

open("/dev/stdout", O_WRONLY|O_CREAT|O_APPEND, 0666) = 3
lseek(3, 0, SEEK_END) = -1 ESPIPE (Illegal seek)
fstat(3, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0

I used this comand:

  $ strace -o trace python2 -c 'import os; os.uname();os.uname();os.uname(); f=open("/dev/stdout", "a"); os.uname(); f.close()'

It looks like the C library simply ignores ESPIPE on lseek(fd, 0, SEEK_END) when opening a file in append mode:
https://sourceware.org/git/?p=glibc.git;a=blob;f=libio/fileops.c;h=13157354272ff9ab1832d4a619a81f05898fcd69;hb=HEAD#l242

  if ((read_write & _IO_IS_APPENDING) && (read_write & _IO_NO_READS))
    if (_IO_SEEKOFF (fp, (_IO_off64_t)0, _IO_seek_end, _IOS_INPUT|_IOS_OUTPUT)
	== _IO_pos_BAD && errno != ESPIPE)
      {
	close_not_cancel (fdesc);
	return NULL;
      }

from _IO_file_open() file operation.

Users of Supervisor have been depending on that for a long time.

Hum, the workaround is very simple no? Just open /dev/stdout with mode "w", no?

Presumably the case was that a *named* log file is opened with 'a' mode, and one could pass '/dev/stdout' just like any other name of a file, and it did work, but not in Python 3.5.

Antti Haapala added the comment:

Presumably the case was that a *named* log file is opened with 'a' mode, and one could pass '/dev/stdout' just like any other name of a file, and it did work, but not in Python 3.5.

Oh ok, in this case, you need something smarter like:

mode = 'a' if not stat.S_ISCHR(os.stat(filename).st_mode) else 'w'
fp = open(filename, mode)

or something like (emulate append mode, but catch ESPIPE):

fp = open(filename, 'w')
try:
  fp.seek(0, os.SEEK_END)
except OSError as exc:
  if exc.errno != errno.ESPIPE: raise

Thanks for the analysis. I have already started a pull request to fix this in Supervisor, but I also thought this change to Python might be gratuitous and accidental. It seems like open('/dev/stdout', 'a') ought to work the same as Python 2. If not, the Python documentation should warn people that using 'a' with character devices and FIFOs will cause an OSError.

And the fact that python deviates from posix in this regard seems like a bug to me, since the posix behavior is, as noted, very useful. Can't we handle ESPIPE like the C library does, instead of forcing users to learn that arcane incantation?

Handling ESPIPE for append mode seems reasonable to me, even as a bug fix for existing versions.

But there is a similar problem with "r+" and "w+" modes and unseekable files (unless buffering=0). See bpo-20074. So we can’t say in general that Python 3 faithfully implements all aspects of Python 2’s / C’s / Posix’s file modes.

Yeah, it definitely is a bug in CPython. open(mode='a') should always append to the end of the given file.

If you're writing an append-only text log to some file-like object, that's the mode you use, not some version/platform/filesystem specific voodoo to find out what's the least incorrect way to work around Python implementation deficiencies.

I was bitten by this porting a system from Python 2.7 to 3.6. "/dev/stderr" is a very nice default for logfiles. Users will frequently override the default, so you really want to open the logfile in append mode. Having to jump through hoops to avoid blasting a user's logfile is kinda dumb, and as others have pointed out, error-prone.

os.open works just fine with O_WRONLY|O_APPEND as the flags. /dev/null can be opened in append mode, but not /dev/stderr or /dev/stdout.

If we were starting from zero, I would suggest omitting the lseek() after open. Sure .tell() might report 0 on newly opened files, but avoiding unnecessary io operations is nicer.

New changeset 74fa9f7 by Benjamin Peterson in branch 'master':
closes bpo-27805: Ignore ESPIPE in initializing seek of append-mode files. (GH-17112)
74fa9f7

New changeset b8b3e43 by Benjamin Peterson in branch '3.7':
[3.7] closes bpo-27805: Ignore ESPIPE in initializing seek of append-mode files. (GH-17137)
b8b3e43

New changeset 9788f97 by Benjamin Peterson in branch '3.8':
[3.8] closes bpo-27805: Ignore ESPIPE in initializing seek of append-mode files. (GH-17136)
9788f97