classification
Title: make zipfile.PyZipFile more usable
Type: enhancement Stage: patch review
Components: Library (Lib) Versions: Python 3.4
process
Status: closed Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Christian.Tismer, georg.brandl, python-dev, r.david.murray, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2013-10-16 21:43 by Christian.Tismer, last changed 2014-03-06 15:09 by python-dev. This issue is now closed.

Files
File name Uploaded Description Edit
zipfile.diff Christian.Tismer, 2013-10-16 21:43 small addition to zipfile that makes it useful for the python library
make_libzip.py Christian.Tismer, 2013-10-16 21:59
Messages (17)
msg200088 - (view) Author: Christian Tismer (Christian.Tismer) * (Python committer) Date: 2013-10-16 21:43
zipfile.PyZipFile needs a filter function.

Reason:
When creating an archive of the python lib, we don't want the tests.
Especially the test file "badsyntax_future3.py" does not compile.

Use case:

With this little addition, it becomes very easy to create a zip file
of the whole python library. See the attached use case.
msg200089 - (view) Author: Christian Tismer (Christian.Tismer) * (Python committer) Date: 2013-10-16 21:59
Here is my use case as an example.
With this patch above, I can easily create a .zip file of the standard lib.

This was no longer possible at all, after revision 17558,
from 2001-04-18:

"""This is a test"""
from __future__ import nested_scopes
from __future__ import rested_snopes

def f(x):
    def g(y):
        return x + y
    return g

print f(2)(4)
msg200112 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2013-10-17 06:21
Looks fine but -- as a new feature -- is 3.4 only, and needs docs and tests.
msg200269 - (view) Author: Christian Tismer (Christian.Tismer) * (Python committer) Date: 2013-10-18 15:04
Hi Georg,
So do you think it is ok this way?
I was not sure if extending the function with an optional
arg is ok, or if a method to configure PyZipFile would be better.
At the moment I just needed the simple functionality.
Should it maybe get a regex like compileall.py ?

And a general question:
What is the right constraint for a filter function?
As I wrote it, returning nothing would simply be treated as "False".
Maybe it is better to enforce a return value which explicitly forbids
to be just None, which often just means "I forgot the return value" ?

About feature or fix: Well, I need this for python2.7, because without
it, the whole purpose of PyZipFile is pretty questionable. I might argue
it a fix, because it crashes on the standard library ;-)

Anyway, tell me and I'l add test, docs and put it into dev.

cheers - chris
msg200275 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-10-18 15:33
I don't think this is needed. You can walk a tree and call writepy() for files and directories which you want.
msg200279 - (view) Author: Christian Tismer (Christian.Tismer) * (Python committer) Date: 2013-10-18 15:56
@serhiy.storchaka

> I don't think this is needed. You can walk a tree and call writepy()
> for files and directories which you want.

What exactly do mean by "this" and "needed"?
I cannot see the connection of my initial post and your reply.

Running PyZipFile on a package dir of the standard lib _does_ traverse
the tree, and there is no way to stop it from doing that.
That was the whole point of the issue.
Please correct me if I'm missing something.
msg200282 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-10-18 16:34
"this" is a filter function. "Not needed" means that you can got what you want without adding a filter function to zipfile.PyZipFile. Just don't call writepy() on directories which contains files which shouldn't be zipped.
msg200290 - (view) Author: Christian Tismer (Christian.Tismer) * (Python committer) Date: 2013-10-18 17:24
Ah, I understand:

The case that does not compile comes from the toplevel "test" folder,
which I could have excluded explicitly.

But it is not a complete solution:
If I want to add every package from the standard lib, then I necessarily
encounter enclosed test folders, for instance:

    Lib/unittest

contains

    Lib/unittest/test

Your hint to just not call writepy() on directories which contain files
which shouldn't be zipped means that I cannot use writepy at all,
because many library packages contain tests.
But that is the only purpose of class PyZipFile, therefore the patch.
msg200684 - (view) Author: Roundup Robot (python-dev) Date: 2013-10-21 01:59
New changeset 34fb83421119 by Christian Tismer in branch 'default':
add a filterfunc to zip file.PyZipFile.writepy, issue 19274
http://hg.python.org/cpython/rev/34fb83421119
msg200701 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2013-10-21 06:26
Hi Chris, your commit is a bit hard to review due to all the unrelated spacing changes.  I assume this is done automatically by your editor?  It's probably best to switch off that feature for CPython development :)
msg200702 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2013-10-21 06:38
While reviewing: is it intended that the filter is only called for directories and not for individual files?
msg200703 - (view) Author: Roundup Robot (python-dev) Date: 2013-10-21 06:42
New changeset 2d39b3555951 by Georg Brandl in branch 'default':
#19274: use captured_stdout() in the test suite; add NEWS entry.
http://hg.python.org/cpython/rev/2d39b3555951
msg200750 - (view) Author: Christian Tismer (Christian.Tismer) * (Python committer) Date: 2013-10-21 11:39
ah, I just see that.
The problem was that the checkin drove me nuts. It forced me to
run reindent to normalize the code. I did that with my WindIde
editor, but this did not help.
The point was: Actually an end-of-line was missing at the end of
the files.
Sorry, I did not see that at all, because the indentation changes
are at the end. I usually avoid this strictly. It was just the check-in
rejection...
msg200856 - (view) Author: Christian Tismer (Christian.Tismer) * (Python committer) Date: 2013-10-22 00:28
@georg:
> While reviewing: is it intended that the filter is only called for directories and not for individual files?

Not really. I will add this, later. Just wanted to see if this makes
sense and it's worth the effort to extend it.
msg200859 - (view) Author: Christian Tismer (Christian.Tismer) * (Python committer) Date: 2013-10-22 02:13
added that with tests.
msg212762 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-03-05 15:01
For future reference, the update Christian refers to in the previous message is 4f1121ae1cb5.
msg212810 - (view) Author: Roundup Robot (python-dev) Date: 2014-03-06 15:09
New changeset 064ee489982e by R David Murray in branch 'default':
whatsnew: improve PyZipFile filterfuc entry, and its docs (#19274).
http://hg.python.org/cpython/rev/064ee489982e
History
Date User Action Args
2014-03-06 15:09:34python-devsetmessages: + msg212810
2014-03-05 15:01:56r.david.murraysetnosy: + r.david.murray
messages: + msg212762
2013-10-22 02:13:11Christian.Tismersetmessages: + msg200859
2013-10-22 00:28:31Christian.Tismersetmessages: + msg200856
2013-10-21 11:39:20Christian.Tismersetmessages: + msg200750
2013-10-21 06:42:19python-devsetmessages: + msg200703
2013-10-21 06:38:51georg.brandlsetmessages: + msg200702
2013-10-21 06:26:15georg.brandlsetmessages: + msg200701
2013-10-21 02:01:06Christian.Tismersetstatus: open -> closed
2013-10-21 01:59:55python-devsetnosy: + python-dev
messages: + msg200684
2013-10-18 17:24:55Christian.Tismersetmessages: + msg200290
2013-10-18 16:34:19serhiy.storchakasetmessages: + msg200282
2013-10-18 15:56:02Christian.Tismersetmessages: + msg200279
2013-10-18 15:33:43serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg200275
2013-10-18 15:04:57Christian.Tismersetmessages: + msg200269
2013-10-17 06:21:21georg.brandlsetversions: + Python 3.4, - Python 2.7
nosy: + georg.brandl

messages: + msg200112

stage: patch review
2013-10-16 21:59:41Christian.Tismersetfiles: + make_libzip.py

messages: + msg200089
2013-10-16 21:43:52Christian.Tismercreate