classification
Title: tarfile requires an actual file on disc; a file-like object is insufficient
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.2
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: lars.gustaebel Nosy List: eric.araujo, lars.gustaebel, strombrg
Priority: normal Keywords: patch

Created on 2010-11-08 23:42 by strombrg, last changed 2012-03-06 10:23 by lars.gustaebel. This issue is now closed.

Files
File name Uploaded Description Edit
tarfile.diff strombrg, 2010-11-08 23:42 Patch to enable passing a stat result object to gettarinfo
Messages (2)
msg120822 - (view) Author: Dan Stromberg (strombrg) Date: 2010-11-08 23:42
The tarfile module's gettarinfo callable insists on stat'ing the file in question, preventing one from dynamically generating file content by passing a file-like object for addfile's fileobj argument.

I believe the attached patch fixes this issue.  I generated the patch against 2.7 and tested it with 2.7, but it applies cleanly against 3.1 and "feels innocuous".  I've also included my test code at the bottom of this comment.

Why would you want to do this?  Imagine you've stored a file in three smaller files (perhaps to save the pieces on small external media, or as part of a deduplication system), with the content divided up into thirds.  To subsequently put this file as a whole into a tar archive, it'd be nice if you could just create a file-like object to emit the catenation, rather than having to create a temporary file holding that catenation.

It's occurred to me that this should be done in a more object oriented style, but that feels a bit inconsistent given that fstat is in the os module, and not provided as an attribute of a file(-like) object.  Comments?

Here's the test code:

#!/usr/local/cpython-2.7/bin/python

import os
import sys
import copy
import array
import stat_tarfile

def my_stat(filename):
        class mutable_stat:
                pass
        readonly_statobj = os.lstat(filename)
        mutable_statobj = mutable_stat()
        for attribute in dir(readonly_statobj):
                if not attribute.startswith('_'):
                        value = getattr(readonly_statobj, attribute)
                        setattr(mutable_statobj, attribute, value)
        return mutable_statobj

class generate_file_content:
        def __init__(self, number):
                self._multiplier = 100
                self._multipleno = 0
                self._number = str(number)
                self._buffer = ''

        def read(self, length):
                while self._multipleno < self._multiplier and len(self._buffer) < length:
                        self._buffer += self._number
                        self._multipleno += 1
                if self._buffer == '':
                        return ''
                else:
                        result = self._buffer[:length]
                        self._buffer = self._buffer[length:]
                        return result

def main():
        with stat_tarfile.open(fileobj = sys.stdout, mode = "w|") as tar:
                for number in xrange(100):
                        #string = str(number) * 100
                        fileobj = generate_file_content(number)
                        statobj = my_stat('/etc/passwd')
                        statobj.st_size = len(str(number)) * 100
                        filename = 'file-%d.txt' % number
                        tarinfo = tar.gettarinfo(filename, statobj = statobj)
                        tarinfo.uid = 1000
                        tarinfo.gid = 1000
                        tarinfo.uname = "dstromberg"
                        tarinfo.gname = "dstromberg"
                        tar.addfile(tarinfo, fileobj)

main()
msg120858 - (view) Author: Lars Gustäbel (lars.gustaebel) * (Python committer) Date: 2010-11-09 10:42
Hm, why don't you just do this:

with stat_tarfile.open(fileobj = sys.stdout, mode = "w|") as tar:
    for number in xrange(100):
        fileobj = generate_file_content(number)
        tarinfo = tar.gettarinfo(fileobj=open("/etc/passwd")) 
        tarinfo.name = 'file-%d.txt' % number
        tarinfo.size = len(str(number)) * 100
        tarinfo.uid = 1000
        tarinfo.gid = 1000
        tarinfo.uname = "dstromberg"
        tarinfo.gname = "dstromberg"
        tar.addfile(tarinfo, fileobj)

Wouldn't that work, too? Or am I missing something?
History
Date User Action Args
2012-03-06 10:23:52lars.gustaebelsetstatus: open -> closed
resolution: not a bug
stage: resolved
2010-11-21 03:30:47eric.araujosetnosy: + eric.araujo
2010-11-09 10:42:40lars.gustaebelsetassignee: lars.gustaebel

messages: + msg120858
nosy: + lars.gustaebel
2010-11-09 00:56:03r.david.murraysettype: enhancement
versions: + Python 3.2, - Python 3.1, Python 2.7
2010-11-08 23:42:04strombrgcreate