New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
gzip module does the wrong thing with an os.fdopen()'ed fileobj #57990
Comments
gzip.GzipFile accepts a fileobj parameter with an open file object. Unfortunately gzip requires a filename be embedded in the gzip file and the gzip module code uses fileobj.name to get that. This results in the fake "<fdopen>" name from posixmodule.c being embedded in the output gzipped file when using Python 2.x. This causes problems when ungzipping these files with gzip -d or ungzip implementations that always rely on the embedded filename when writing their output file rather than stripping a suffix from the input filename as they cannot open a file called "<fdopen>" or if they do, each successive ungzip overwrites the previous... On Python 3.x the problem is different, the gzip module fails entirely when given an os.fdopen()'ed file object: $ ./python gzip_fdopen_prob.py
out_file <_io.BufferedWriter name='FOO.gz'>
out_fd 3
fd_out_file <_io.BufferedWriter name=3>
fd_out_file.name 3
Traceback (most recent call last):
File "gzip_fdopen_prob.py", line 13, in <module>
gz_out_file = gzip.GzipFile(fileobj=fd_out_file)
File "/home/gps/oss/cpython/default/Lib/gzip.py", line 184, in __init__
self._write_gzip_header()
File "/home/gps/oss/cpython/default/Lib/gzip.py", line 221, in _write_gzip_header
fname = os.path.basename(self.name)
File "/home/gps/oss/cpython/default/Lib/posixpath.py", line 132, in basename
i = p.rfind(sep) + 1
AttributeError: 'int' object has no attribute 'rfind' (code attached) The os.fdopen()'ed file object is kindly using the integer file descriptor as its .name attribute. That might or might not be an issue, but regardless of that:
Fixing the gzip module to catch errors and use an empty b'' filename for the gzip code in the above error is easy. What should be done about the .name attribute on fake file objects? I don't think it should exist at all. (another quick test shows that gzip in python 3.x can't output to a BytesIO fileobj at all, it thinks it is readonly) |
For 3.x, I think that ignoring non-string names is a reasonable fix. The docs As for the case of 2.7's bogus "<fdopen>" name attribute, I'm not sure what the One alternative would be for GzipFile to specifically check whether a file
Are you sure about this? I can't reproduce the problem. Running this script: import gzip, io
b = io.BytesIO()
with gzip.GzipFile(fileobj=b, mode="w") as g:
g.write(b"asdf ghjk")
print(b.getvalue())
b.seek(0)
with gzip.GzipFile(fileobj=b, mode="r") as g:
print(g.read()) I get the following output:
|
Attached is a fix for 3.x. |
thanks that looks good. As far as fixing this for 2.7 goes, i don't like the _sound_ of it because it is gross... But i'm actually okay with having special case code in the gzip module that rejects '<fdopen>' as an actual filename and uses '' instead in that case. It is VERY unlikely that anyone ever intentionally wants to use that as a filename. Anything more than that (changing the actual '<fdopen>' string for example) seems too invasive and might break someone's doctests and does genuinely make it more difficult to see what a fdopened file object is from its repr. |
New changeset 7d405058e458 by Nadeem Vawda in branch '3.2': New changeset fe36edf3a341 by Nadeem Vawda in branch 'default': |
I agree - it sounds ugly, but pragmatically it seems like the best option. |
Looks like you've got commit privs (yay) so i'm assigning this to you to take care of that way for 2.7 as well. I'd add a comment to the fdopen C code where the "<fdopen>" constant lives as well as to the gzip.py module around the special case for this mentioning that they should be kept in sync. (not that either is _ever_ likely to be changed in 2.7) |
New changeset a08e9e84f33f by Nadeem Vawda in branch '2.7': |
Done. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: