classification
Title: Crash in ZipFile.close() when writing zip file to /dev/null
Type: behavior Stage:
Components: IO, Library (Lib) Versions:
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: erik.bray, serhiy.storchaka, xtreak
Priority: normal Keywords:

Created on 2018-10-05 10:06 by erik.bray, last changed 2018-10-05 13:12 by erik.bray.

Messages (7)
msg327118 - (view) Author: Erik Bray (erik.bray) * (Python triager) Date: 2018-10-05 10:06
Not that there is any great reason to write a zip file to /dev/null, but I had some code that happened to do so which worked on Python 2.7, but at some point this broke:

Python 3.8.0a0 (heads/master:fc7d1b3, Oct  5 2018, 09:49:57)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import zipfile
>>> f = zipfile.ZipFile('/dev/null', 'w')
>>> f.writestr('foo.txt', 'testtesttesttesttest')
>>> f.close()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/embray/src/python/cpython/Lib/zipfile.py", line 1813, in close
    self._write_end_record()
  File "/home/embray/src/python/cpython/Lib/zipfile.py", line 1914, in _write_end_record
    endrec = struct.pack(structEndArchive, stringEndArchive,
struct.error: argument out of range
msg327124 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2018-10-05 11:23
Is this specific to Linux? I can reproduce this on master branch on Ubuntu but there is no error on Mac OS with the master branch.
msg327128 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-10-05 11:51
This may be a Linux bug. The /dev/null file is seekable, but seek() doesn't work correctly for it. It is especially confusing for buffered files. seek() always returns 0 and reset the file position.

>>> f = open('/dev/null', 'wb')
>>> f.seekable()
True
>>> f.tell()
0
>>> f.write(b'abcdefgh')
8
>>> f.tell()
8
>>> f.seek(8)
0
>>> f.tell()
0

In contrary, files like /dev/stdout are not seekable, and writing a ZIP file to them works properly.
msg327130 - (view) Author: Erik Bray (erik.bray) * (Python triager) Date: 2018-10-05 12:10
The regression was introduced by issue26039.

It does seem to be Linux-specific with seek/tell on /dev/null.  For example, I cannot reproduce the issue on Cygwin.
msg327131 - (view) Author: Erik Bray (erik.bray) * (Python triager) Date: 2018-10-05 12:13
On Cygwin the same tests give

>>> f = open('/dev/null', 'wb')
>>> f.seekable()
True
>>> f.write(b'abcdefgh')
8
>>> f.tell()
8
>>> f.seek(8)
8
>>> f.tell()
8


I would also try macOS if I could.  But yes, I wonder if it's a Linux bug.
msg327132 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2018-10-05 12:21
Thanks Erik for the details. On Mac with master also it works like Cygwin.

./python.exe
Python 3.8.0a0 (heads/master:6f85b826b5, Oct  4 2018, 22:44:36)
[Clang 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> f = open('/dev/null', 'wb')
>>> f.seekable()
True
>>> f.write(b'abcdefgh')
8
>>> f.tell()
8
>>> f.seek(8)
8
>>> f.tell()
8
msg327135 - (view) Author: Erik Bray (erik.bray) * (Python triager) Date: 2018-10-05 13:12
For the sake of completeness, same deal in pure C:

$ cat devnul.c
#include <stdio.h>


int main(void) {
    FILE *f = fopen("/dev/null", "w");
    printf("tell() = %ld\n", ftell(f));
    printf("write(\"abcdefgh\") = %zu\n", fwrite("abcdefgh", 1, 8, f));
    printf("tell() = %ld\n", ftell(f));
    printf("seek(8) = %d\n", fseek(f, 8, 0));
    printf("tell() = %ld\n", ftell(f));
    return 0;
}

$ gcc devnulc -o devnul
$ ./devnull
tell() = 0
write("abcdefgh") = 8
tell() = 8
seek(8) = 0
tell() = 0
History
Date User Action Args
2018-10-05 13:12:57erik.braysetmessages: + msg327135
2018-10-05 12:21:16xtreaksetmessages: + msg327132
2018-10-05 12:13:31erik.braysetmessages: + msg327131
2018-10-05 12:10:34erik.braysetmessages: + msg327130
2018-10-05 11:51:38serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg327128

components: + IO
type: crash -> behavior
2018-10-05 11:23:37xtreaksetnosy: + xtreak
messages: + msg327124
2018-10-05 10:06:44erik.braycreate