Currently, the tarfile module only supports adding files if their size is known prior to adding.  However, I think it'd be helpful to be able to store large dynamically generated streams straight into the (uncompressed) .tar file without being able to precalculate the final size and without having to use a temporary file.

I'm not really sure how the API should look like (i.e. whether it should be a new method or extension of addfile()) but the mechanism would be rather simple -- write TarInfo with size of 0, write data until end of stream, write padding appropriately to written data, seek back and update TarInfo.

Of course, the use of this API would have to be restricted to cases when underlying file supports seeking back and random writes, i.e. not stream, not compressed.

Adding this API would require to provide a way to set file status like mode, uid, gid, mtime, type, linkname, uname and gname.

Adding a new argument to gettarinfo looks weird to me, adding a new method may be better. I will try to propose a working implementation.

> Adding this API would require to provide a way to set file status like mode, uid, gid, mtime, type, linkname, uname and gname.

That's why I mentioned addfile() -- it takes TarInfo object for that purpose.  I suppose the new function should have the same parameters, except it would set size in the TarInfo instance instead of getting it.

Yes, but in the same way that there is the add method to conveniently build the TarInfo object for the user, shouldn't we provide a new convenience method to TarFile to support this (in addition to modifying TarInfo)?

Hi @mgorny, the changeset in PR 10714 should  do what you are looking for.

Thanks a lot! I've left a few comments based on eyeball review. I'm going to test it later today.

The changes at <https://github.com/python/cpython/pull/10714/files/11ca0f0> have various other behaviour changes which are not discussed here. They seem to be there just so that you can use the TCP socket from “urlopen” with “gettarinfo”. But “gettarinfo” is supposed to be for named filesystem objects that could be members of a tar file. See Issue 22208 discussing how to create TarInfo objects without using the OS filesystem.

I came across this thread while working on the PR, creating tarinfo as Lars Gustäbel suggests does not work since you still need to get the size before reading.

Do you think the API should be different?

If something like your “addbuffer” method existed, then you won’t need to get the size first, right? We don’t need the changes in “gettarinfo” for “addbuffer” to be useful.

BTW have you considered returning a file writer rather than accepting a file reader? Similar to ZipFile.open(..., mode='w'). It would be a bit more complicated to implement, but also more flexible for the user:

# File downloaded with “urlopen”, also possible with TarFile.addbuffer API:
with tf.get_file_writer(download_tarinfo) as writer:
    shutil.copyfileobj(urlopen_response, writer)

# SVG file generated on the fly, encoded with UTF-8 and Gzip compressed; not possible with “addbuffer”:
writer = tf.get_file_writer(svgz_tarinfo)
gzip_writer = gzip.GzipFile(fileobj=writer, mode='w')
with io.TextIOWrapper(gzip_writer, 'utf-8') as text_writer:
    svg = xml.sax.saxutils.XMLGenerator(text_writer, 'UTF-8')
    svg.startDocument()
    ...

Thanks, this looks interesting.

How will the file writer know the whole file has been read? The override of the Tar header is done on `close`?

Are `download_tarinfo` and `svgz_tarinfo` built by hand if we don't make changes in `gettarinfo`?

Yeah, the TarFile class would fix up the header when the user calls “close”. I think this is how it was done for ZipFile (implemented in Issue 26039).

Yes currently you would have to build the tarinfo object by hand. I think a helper function would be nice, but the “TarFile.gettarinfo” method is not a good place for that. In fact, documenting the default settings for the TarInfo constructor may be enough. Then you might get away with the following if you just want a regular file with no specific attributes:

download_tarinfo = TarInfo('index.html')
svgz_tarinfo = TarInfo('image.svg.gz')