Issue10436
Created on 2010-11-16 18:17 by David.Nesting, last changed 2011-07-22 19:45 by terry.reedy.
| Messages (3) | |||
|---|---|---|---|
| msg121308 - (view) | Author: David Nesting (David.Nesting) | Date: 2010-11-16 18:17 | |
When opening a tarfile with mode "r|" (streaming mode), extractfile("filename") and extractfile(mytarfile.getmembers()[0]) raise "tarfile.StreamError: seeking backwards is not allowed". extractfile(mytarfile.next()) succeeds. A more complete test case:
"""
import tarfile
import StringIO
# Create a simple tar file in memory. This could easily be a real tar file
# though.
data = StringIO.StringIO()
tf = tarfile.open(fileobj=data, mode="w")
tarinfo = tarfile.TarInfo(name="testfile")
filedata = StringIO.StringIO("test data")
tarinfo.size = len(filedata.getvalue())
tf.addfile(tarinfo, fileobj=filedata)
tf.close()
data.seek(0)
# Open as an uncompressed stream
tf = tarfile.open(fileobj=data, mode="r|")
#f = tf.extractfile("testfile")
#print "%s: %s" % (f.name, f.read())
#
#Traceback (most recent call last):
# File "./bug.py", line 19, in <module>
# print "%s: %s" % (f.name, f.read())
# File "/usr/lib/python2.7/tarfile.py", line 815, in read
# buf += self.fileobj.read()
# File "/usr/lib/python2.7/tarfile.py", line 735, in read
# return self.readnormal(size)
# File "/usr/lib/python2.7/tarfile.py", line 742, in readnormal
# self.fileobj.seek(self.offset + self.position)
# File "/usr/lib/python2.7/tarfile.py", line 554, in seek
# raise StreamError("seeking backwards is not allowed")
#tarfile.StreamError: seeking backwards is not allowed
#for member in tf.getmembers():
# f = tf.extractfile(member)
# print "%s: %s" % (f.name, f.read())
#
# Same traceback
while True:
member = tf.next()
if member is None:
break
f = tf.extractfile(member)
print "%s: %s" % (f.name, f.read())
# This works.
"""
It appears that extractfile("filename") invokes getmember("filename"), which invokes getmembers(). getmembers() scans the entire file before returning results, and by doing so, it's read past and discarded the actual file data, which makes it impossible for us to actually extract it.
If this is accurate, this seems tricky to completely fix. You could make getmembers() a generator that doesn't read too far ahead so that the file's contents are still available if someone wants to retrieve them for each file yielded. getmember("filename") could just scan forward through the file until it hits a match, but you'd still lose the ability to do a getmember("filename") on a file that we skipped over.
If nothing else, document that extractfile("filename"), getmember() and getmembers() won't work reliably in streaming mode, and possibly raise an exception whenever someone tries just to make behavior consistent.
|
|||
| msg121339 - (view) | Author: Lars Gustäbel (lars.gustaebel) * ![]() |
Date: 2010-11-17 09:32 | |
This behaviour is intentional. A tar archive does not contain a central directory structure, it is just a chain of files. As a side-effect it is possible to have multiple files with the same name in one archive, e.g. when append mode was used. That's why the archive must be scanned from the beginning to the end as soon as you reference an archive member by its name. The best way to deal with this issue in my opinion is to improve the documentation for the stream interface. |
|||
| msg121362 - (view) | Author: David Nesting (David.Nesting) | Date: 2010-11-17 16:03 | |
Thanks, Lars. And this does make complete sense to me in retrospect. Better documentation here would help a lot. I'm happy to take a stab at this. Short of labeling methods as "safe for streaming" versus "unsafe for streaming", it occurs to me that it would be a lot cleaner if TarFile were actually broken up into two classes: one streaming-safe, and the other layering random access convenience methods on top of that. For compatibility's sake the open method should probably still return an instance of the composite class, but at least it keeps these logically separate internally and makes it easier to document. |
|||
| History | |||
|---|---|---|---|
| Date | User | Action | Args |
| 2011-07-22 19:45:42 | terry.reedy | set | versions: + Python 3.3, - Python 3.1 |
| 2010-11-19 14:18:17 | eric.araujo | set | nosy:
+ docs@python stage: needs patch components: + Documentation, - Library (Lib) versions: - Python 2.6, Python 3.3 |
| 2010-11-17 16:03:15 | David.Nesting | set | messages: + msg121362 |
| 2010-11-17 09:32:29 | lars.gustaebel | set | priority: normal -> low nosy: + lars.gustaebel versions: + Python 3.1, Python 3.2, Python 3.3 messages: + msg121339 assignee: lars.gustaebel |
| 2010-11-16 18:17:43 | David.Nesting | create | |
