It would be most helpful if a method could be included in the TarFile
class of the tarfile module and the ZipFile class of the zipfile module
that would remove a particular file (either given by a name or a
TarInfo/ZipInfo object) from the archive.

Usage to remove a single file from an archive would be as follows:

import zipfile
zipFileObject = zipfile.ZipFile(archiveName,'a')
zipFileObject.remove(fileToRemove)
zipFileObject.close()

Such a method should probably only apply to archives that are in append
mode as write mode would erase the original archive and read mode should
render the archive immutable.

One possible extra to be included is to allow a list of file names or
ZipInfo/TarInfo objects to be passed into the remove method, such that
all items in the list would be removed from the archive.

+1

Slight change to:

"Such a method should probably only apply to archives that are in append
mode as write mode would erase the original archive and read mode should
render the archive immutable."

The method should probably still apply to an archive in write mode.  It
is conceivable that one may need to delete a file from the archive after
it has been written but before the archive object has been closed.

-1. I don't think this can be implemented in a reasonable way, assuming
that you want the file to become smaller as a consequence of removal.

-1, although I can only speak for tarfile. Removing members from a tar
archive sounds obvious and easy but it is *not*. A file in an archive is
stored as a header block (that contains the metadata) followed by a
number of data blocks (that contain the file's data). New files are
simply appended to the archive file. There is no central table of
contents whatsoever. To make things worse, a compressed archive is
compressed in one go from the beginning right up to the end, it is not
possible to access a member in the middle of an archive without having
to decompress all data before it.
Deleting files from an uncompressed archive is rather straightforward
implementation-wise but IO intensive and risky. In contrast, there is no
other way to delete files from a *compressed* tarfile than to make a
copy of it omitting the unwanted files.

In light of Lars's comment on the matter, perhaps this functionality
could be added to zip files only.  Surely it can be done, considering
that numerous utilities and even Windows Explorer provide such
functionality.  I must confess that I am unfamiliar with the inner
workings of file archives and compression, but seeing as it is
implemented in a number of places already, it seems logical that it
could be implemented in ZipFile as well.  I'll spend some time the next
few days educating myself about zip files and how this might be
accomplished.

> In light of Lars's comment on the matter, perhaps this functionality
> could be added to zip files only.  Surely it can be done, considering
> that numerous utilities and even Windows Explorer provide such
> functionality.

Are you sure they are not creating a new file in order to delete
content? I recall that early zip tools (e.g. pkzip) had a mode
where they would merely delete the entry from the directory, but
leave the actual data in the file. Would you consider that a correct
implementation? If so, *that* can be done, for zipfiles, AFAIU.

> I'll spend some time the next
> few days educating myself about zip files and how this might be
> accomplished.

Please do - you'll find that deletion from zipfiles comes in a can
full of worms.

I done it In a very *violent* way.
Is it ok for you thought?
if so, would anybody please fix it into the lib?

> I done it In a very *violent* way.
> Is it ok for you thought?

In the form in which you have done it, it is clearly
unacceptable for inclusion in the library: we don't
want to add two modules "delete" and "classtools".

In addition, notice that code is for tarfile, whereas
the OP was asking for a similar feature for zipfile.

> if so, would anybody please fix it into the lib?

This is not how this works. If you want us to take
action, please submit a complete and correct patch.

I have attempted to implement a ZipFile.remove function.  It seems to work fine.  I have submitted a patch.

The method of implementation is: find the file's index in the file list, then sum the lengths of the file entries before it to find its location in the archive.  Then simply read in all the bytes after it, write them out at that location, and truncate the file x bytes shorter, where x is the length of the record.  This works because the directory listing is created when the file is closed, so there's no harm in truncating.

I've also made it truncate the zip file after reading in the existing files upon creation, because the directory information is not used after this point.

This could use some testing on large files.

This is my first patch, so let me know if I've done anything wrong.

My patch had some bugs, I'll need to do some more testing.  Sorry about that.

What's the status with this patch? If nobody's looking at it I can try to see if it works and write the test and documentation for it.

Please feel free to test, revise, and write. Though 'removed', the file is still accessible via the history list. (Click 'zipfile_remove.patch' and then 'download'.)

I fixed the bugs I found, added tests and documentation. What do you guys think?

Fixed the bugs Martin pointed out and added the relevant tests. Sadly I had to move some stuff around, but I think the changes are all for the better. I wasn't sure about the right convention for the 2 constants I added btw.

Martin did a review of the newer patch; maybe you didn’t get the mail (there’s a Rietveld bug when a user name without email is given to the Cc field).

I'm not sure I understand how http://bugs.python.org/review/6818/show works. I've looked all over and only found remarks for "zipfile.remove.patch" and not for "zipfile.remove.2.patch" which addressed all the aforementioned issues.

Also, I don't understand how to add myself to the CC of this issue's review page.

Yuval, can you please submit a contributor agreement? See

http://www.python.org/psf/contrib/

As for adding yourself to the CC list: notice the string "ubershmekel" appearing in the "CC" field of http://bugs.python.org/review/6818/show. It means that you are already on the CC list.

Yuval has submitted a CLA. I'm moving the proposal to 3.4 as 3.3 is in feature freeze mode.

Ping. Has this been postponed?

I agree with Martin and Lars, this issue is not so easy at looks at first glance.

For ZIP files we should distinct two different operations.

1. Remove the entry from the central directory (and may be mark local file header as invalid if it is possible). This is easy, fast and safe, but it doesn't change the size of ZIP file.

2. Physical remove the content of the file from ZIP file. This is so easy as remove a line from the text file. In worst case it has linear complexity from the size of ZIP file.

2a. The safer way is to create temporary file in the same directory, copy the content of original ZIP file excluding deleted file, and then replace original ZIP file by modified copy. Be aware about file and parent directory permissions, owners, and disk space.

2b. The faster but less safe way is to "shift" the content of the ZIP file after deleted file by reading it and writing back in the same ZIP file at different position. This way is not safe because when something bad happen at writing, we can lost all data. And of course there are crafty ZIP files in which the order of files doesn't match the order in central directory or even files data overlap.

For performance may be we should implement (2) not as a method to remove single file, but as a method which takes the filter function and then left in the ZIP file only files for which it returns true.

Or may be implement (1) and then add a method which cleans up the ZIP archive be removing all files removed from the central directory. We should discuss alternatives.

And as for concrete patch, zipfile.remove.2.patch can read the content of all ZIP file in the memory. This is not appropriate, because ZIP file can be very large.

I'd be interested in taking up the zip portion at Pycon 2015 this year. I recently had need to delete file(s) from a zipfile.

To do it today with the existing API requires you to unpack the zip and repack it. The unpack is slow and you need enough free disk space for the uncompressed files.

My strategy is essentially exactly what msg229893 2a said: copy binary blobs to a tempfile, then overwrite the original when complete. I would use a name filter function to decide what to delete and optional parameter for the temp file (falling back to tempfile.tempfile if None). IIRC, this is the same strategy used in the dotNet zip library.

I think it would be better if new method would copy filtered content to other ZIP file. Not alway you want to modifying the original file.

The question is what to do with the data before the start of the ZIP file or between files in the ZIP file, or if files in the ZIP file are overlapped. Here is a sample of such file.

Hi,

I've recently been working on a Python module for the Adobe universal container format (UCF) which extends the zip specification - as part of this I wanted to be able to remove and rename files in an archive.

I discovered this issue when writing the module so realised there wasn't currently a solution - so I went down the rabbit hole.

I've attached a patch which supports the removal and renaming of files in a zip archive. You can also look at this python module in a git-repo which is a the same code but separated out into a class that extends ZipFile: https://github.com/gambl/zipextended.

The patch provides 4 main new "public" functions for the zipfile library:

- remove(self, zinfo_or_arcname):
- rename(self, zinfo_or_arcname, filename):
- commit(self):
- clone(self, file, filenames_or_infolist=None, ignore_hidden_files=False)

The patch is in part modelled on the rubyzip solution. Remove and rename will initially only update the ZipFile's infolist. Changes are then persisted via a commit function which can be called manually - or will be called automatically upon close. Commit will then clone the zipfile with the necessary changes to a temporary file and replace the original file when that operation has completed successfully.

An alternative to remove files without modifying the original is via the clone method directly. This is in the spirit of Serhiy's suggestion of filtering the content and not modifying the original. You can pass a list of filenames or fileinfos of the files to be included in the clone.
So that clone can be performed without decompressing and then recompressing the files in the archive I have added two functions write_compressed and read_compressed.

I have also attempted to address Serhiy's concern with respect to the tricky.zip - "hidden files" in between members of the archive. The clone method will by default retain any hidden files and maintain the same relative order in the archive. You can also elect to ignore the hidden files, and clone with just the files listed in the central directory.

I did have to modify the tricky.zip attached to this issue manually as the CRC of file two (with file three embedded) was incorrect - and would therefore fail testzip(). I'm not actually sure how one would create such an archive - but I think that it's valid according to the zip spec. I've actually included the modified version in the patch for a few of the tests.

I appreciate that this is a large-ish patch and may take some time to review - but as suggested in the comments - this wasn't as straight forward as is seems!

Look forward to your comments. 

The signatures of the main functions are described below:

remove(self, zinfo_or_arcname):

    Remove a member from the archive.

    Args:
      zinfo_or_arcname (ZipInfo, str) ZipInfo object or filename of the
        member.

    Raises:
      RuntimeError: If attempting to modify an Zip archive that is closed.
---

rename(self, zinfo_or_arcname, filename):

    Rename a member in the archive.

    Args:
      zinfo_or_arcname (ZipInfo, str): ZipInfo object or filename of the
        member.
      filename (str): the new name for the member.

    Raises:
      RuntimeError: If attempting to modify an Zip archive that is closed.


clone(self, file, filenames_or_infolist=None, ignore_hidden_files=False):

    Clone the a zip file using the given file (filename or filepointer).

    Args:
      file (File, str): file-like object or filename of file to write the
        new zip file to.
      filenames_or_infolist (list(str), list(ZipInfo), optional): list of
        members from this zip file to include in the new zip file.
      ignore_hidden_files (boolean): flag to indicate wether hidden files
        (data inbetween managed memebers of the archive) should be included.

    Returns:
        A new ZipFile object of the cloned zipfile open in append mode.

        If copying hidden files then clone will attempt to maintain the
        relative order between the files and members in the archive

commit(self):
     Commit any inline modifications (removal and rename) to the zip archive.

     This makes use of a temporary file to create a new zip archive with the
     required modifications and then replaces the original.

     This therefore requires write access to either the directory where the
     original zipfile lives, or to python's default tempfile location.

All of you who have or might submit patches -- Victorlee, Troy, Mathew, or anyone else, please sign a PSF contributor agreement.  We should not even look at a patch from you before you do.

Info: https://www.python.org/psf/contrib/
Form: https://www.python.org/psf/contrib/contrib-form/

Receipt is official when a * appears after your name.  This usually takes about a week.

Thanks Terry - apologies - I meant to sign before I submitted the patch. I have signed the CLA now.

Hi all,

Apologies again, I've had to pull the patch just temporarily whilst I check with my employer (The University of Manchester) that everything is OK with me contributing this. 

Everything should be OK - but I just wanted to do things correctly. 

I'll re-submit the patch for review when I get the OK.

- Matthew

The zipfile way to delete or rename would be to just change the index. It really doesn't want to be re-written as it is designed to span disks. Many old versions of files can be scattered within the zip. In addition self-extracting zip files will have executable code in the front of the zip. To make a zip smaller, it's desirable to have a filter function. There were points brought up about zip with overlapping data blocks (and you could envision saving space by having many identical files share the same compressed block... but this is not legal zip. A file info header block must precede EACH individual file data block. This is brought home by there being only a length field in the info header.

I can add some more tests to bring up the coverage, but wanted to get reviewer opinion on the direction of this before doing more work.

Around PyCon it might take a little longer than a week. IIRC Ewa does the paper work. She is also on the organizing committee of PyCon.

Maybe it takes a little longer than a week. I have a final signed agreement from Ewa (https://secure.echosign.com/public/viewAgreement?aid=X88L4EVP5IXC289&eid=X88M6DGQ93J5K38&)

signed on
04/17/2014 6:48 PM
Wow, exactly one year ago!

has this been merged?

I wasn't aware of this issue and opened another one which I just closed as a duplicate -
https://bugs.python.org/issue40175

I also submitted a PR for the other one -
https://github.com/python/cpython/pull/19336

The implementation does involve shifting the bytes of the files and updating the offset of each file in the central directory.
I do handle the case in which the central directory order and the order of the actual files in the archive isn't the same simply by sorting the central directory in memory before execution.

The only issue I see here is the fact that if something happens during the process, the archive might get corrupted - but IMO it's worth it.
Many tools provide that functionality (Windows, Total Commander) and as far as I know they do it in place.
In the worst case, it's possible to implement it by creating a temp archive like Serhiy suggested - not too difficult to implement either since it simply requires duplicating the file and operating on it.

Closed my initial PR and opened this one instead -
https://github.com/python/cpython/pull/19336

Sorry, typo, here's the new PR -
https://github.com/python/cpython/pull/19358