classification
Title: PEP 441 - Improving Python Zip Application Support
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: steve.dower Nosy List: Jim.Jewett, berker.peksag, brett.cannon, dholth, ethan.furman, paul.moore, python-dev, serhiy.storchaka, steve.dower
Priority: normal Keywords: patch

Created on 2015-02-20 12:52 by paul.moore, last changed 2015-03-13 15:06 by paul.moore. This issue is now closed.

Files
File name Uploaded Description Edit
pep441.patch paul.moore, 2015-02-20 12:52 review
pep441.patch paul.moore, 2015-02-20 17:01 review
pep-441.patch paul.moore, 2015-02-26 17:00 review
pep-441.v4.patch paul.moore, 2015-02-27 11:22 review
pep-441.v5.patch paul.moore, 2015-03-08 16:59 review
pep-441.v6.patch paul.moore, 2015-03-10 10:27 review
Messages (42)
msg236276 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2015-02-20 12:52
This is the patch for PEP 441 (Zip Application Support).

Steve, could you check the installer changes, please? I haven't managed to get a setup where I can test the installer, and I'm not aware of any WiX coding tools, so I just edited the XML files by hand with grep and vim, and checked that Tools/buildmsi.bat doesn't error in anything I wrote... (There's a CRC check error in TCL, but that was there before I made changes)

  light.exe : error LGHT0216: An unexpected Win32 exception with error code 0x17 occurred while accessing file 'C:\Work\Projects\cpython\externals\tcltk64\lib\tcl8.6\tzdata\America\Detroit': Data error (cyclic redundancy check) [C:\Work\Projects\cpython\Tools\msi\tcltk\tcltk.wixproj]

Also, I don't *think* I need to do anything for the new files in Doc\Lib, Lib and Lib\test to get picked up, but I'm not sure how to check that.
msg236285 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-02-20 14:12
The installer changes look fine, though there shouldn't be PATH related strings in exe/, so there's probably a separate issue there. Wix doesn't really have any editors that aren't just XML editors.

We probably want to update the ContentType value from text/plain for all the shortcuts, though I'm not sure what to (it affects the implicit associations and extra commands on the files, inc. Open With).

The easiest way to make sure the files are in the installer is to install it :) But if they're just .py files under Lib/ then they'll be there.
msg236286 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-02-20 14:12
I haven't looked closely at the rest, btw. Will try and find time today or tomorrow.
msg236291 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2015-02-20 15:22
Thanks for checking, Steve. I don't get an installer because of the checksum error quoted, although I did get the component msi files.

As far as content type is concerned, I wasn't sure what effect it had so I just copied what was there. I guess application/zip might be better for .pyz files, based on what you said?

I'm not sure what you meant by the comment about exe/. All I did was mention .pyz as well as .py - although I don't see where the PathDescription string us used so I may be missing something important!
msg236301 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-02-20 16:23
Either "application/zip" or "application/x-zip-compressed", I'm not sure exactly what the difference is, but the default .zip association has the latter.

The CRC error may be your machine or it may be because we're currently running on unstable WiX releases (hey, everything else is unstable right now :) ). If you can edit and re-save that file, I'd expect it to work, otherwise the 32-bit installer should be okay.

The exe/ comment is more a note for myself, sorry. Those strings are unused and should be removed (I'll do that now - got a few things to fix).
msg236308 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2015-02-20 17:01
Thanks. Updated patch with the new mime-type.

Looks like there's disk errors on that file with the CRC check. Lovely :-)
msg236455 - (view) Author: Daniel Holth (dholth) Date: 2015-02-23 18:16
It looks like your patch does not register PYZW in Windows
msg236457 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-02-23 18:19
It registers the association, but doesn't add it to PATHEXT.
msg236458 - (view) Author: Daniel Holth (dholth) Date: 2015-02-23 18:25
I don't see the string PYZW anywhere in the patch, I only see PYZ,
perhaps I don't know enough about this bug tracker or registering file
extensions?

On Mon, Feb 23, 2015, at 01:19 PM, Steve Dower wrote:
> 
> Steve Dower added the comment:
> 
> It registers the association, but doesn't add it to PATHEXT.
> 
> ----------
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue23491>
> _______________________________________
msg236459 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-02-23 18:28
It's this line:

    <Extension Id="$(var.ArchiveFileExtension)w" ContentType="application/x-zip-compressed">

$(var.ArchiveFileExtension) expands to "pyz" or "pxz" depending on the build options, and the literal w makes it pyzw or pxzw.
msg236531 - (view) Author: Daniel Holth (dholth) Date: 2015-02-24 19:26
Spelling

raise PackError("Cannot spacify entry point if the source has __main__.py")
msg236534 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2015-02-24 19:34
Thanks, I'll fix for the next iteration of the patch.
msg236669 - (view) Author: Daniel Holth (dholth) Date: 2015-02-26 13:56
Create and open executable file respecting the Unix user's umask:

os.fdopen(os.open(filename, os.O_CREAT|os.O_RDWR), "rw")

On Tue, Feb 24, 2015, at 02:34 PM, Paul Moore wrote:
> 
> Paul Moore added the comment:
> 
> Thanks, I'll fix for the next iteration of the patch.
> 
> ----------
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue23491>
> _______________________________________
msg236670 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2015-02-26 14:18
I don't follow (and I don't really want to do things this low level without a compelling reason...)
msg236672 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2015-02-26 14:41
Following on from that, the code to make an archive executable is currently

os.chmod(new_archive, os.stat(new_archive).st_mode | stat.S_IEXEC)

Should I use "... | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH"? If so, do I need to protect that with an "if not Windows" test? (I've tested the existing code and it does nothing on Windows, so I omitted the test at the moment). Is there any *other* way I should be making a file executable on Unix?

(Side note: Maybe there should be an os.make_executable(pathname) or similar that does the right thing in a cross-platform way?)
msg236673 - (view) Author: Jim Jewett (Jim.Jewett) Date: 2015-02-26 15:18
I think that instead of application/zip, you want application/zipped-python (or whatever the precise term was).  This was one reason to register the MIME type.  That said, application/zip is probably not harmful; the worst it should do it hand the archive to a zip program.

What is pxzw for, other than amusing mispronunciations?
msg236675 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-02-26 15:29
The .px* extensions are for testing builds without messing up your actual file associations. I ported them forward from the old installer, but most people will never see them.
msg236676 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2015-02-26 15:30
If I understand Steve's comments, the mime type is used by Windows, so application/zip lets Windows know that this filetype is fundamentally a zip file (and so it'll offer to open it with your zip program if you right click, stuff like that).

pxzw - the "w" on the end marks it as a GUI script, to be run by pythonw.exe. The x is a "test mode" thing for development builds of Python. You wouldn't see it in a production build (the production filetypes are pyz and pyzw).
msg236678 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-02-26 15:37
There's also a PerceivedType property which we could set to make sure that archivers correctly light up. I think that'll get you the built-in Extract command. Most tools appear on every file and will try and read it to see what commands make sense, which will work fine for a zip file.
msg236680 - (view) Author: Daniel Holth (dholth) Date: 2015-02-26 15:57
On Thu, Feb 26, 2015, at 09:41 AM, Paul Moore wrote:
> 
> Paul Moore added the comment:
> 
> Following on from that, the code to make an archive executable is
> currently
> 
> os.chmod(new_archive, os.stat(new_archive).st_mode | stat.S_IEXEC)
> 
> Should I use "... | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH"? If so,
> do I need to protect that with an "if not Windows" test? (I've tested the
> existing code and it does nothing on Windows, so I omitted the test at
> the moment). Is there any *other* way I should be making a file
> executable on Unix?
> 
> (Side note: Maybe there should be an os.make_executable(pathname) or
> similar that does the right thing in a cross-platform way?)

The chmod + umask analog that will work not just on a newly created file
is

umask = os.umask(0) # must change umask to get umask

os.umask(umask) # restore previous umask

os.chmod(new_archive, os.stat(new_archive).st_mode |
((stat.stat.S_IXUSR|stat.S_IXGRP|stat.S_IXOTH) & ~umask))

If I understand the man page correctly, "chmod +x filename" does exactly
the above. Depending on the umask the command may or may not create a
world / group / user executable file.
msg236682 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2015-02-26 16:19
OK, thanks. I don't propose to go there with the initial implementation. If it's a problem in practice, someone can raise a bug and we'll fix it then. (I've never seen actual Python code in the wild that does all of that...)
msg236688 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2015-02-26 16:57
OK, here is an updated patch based on the python-dev discussions
msg236689 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2015-02-26 17:00
Sorry - typo. Try again...
msg236694 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-02-26 17:55
Did you noticed my comments to previous patch Paul?
msg236698 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2015-02-26 18:57
Serhiy - I just got a notification from the review tool - I'd not used it before and didn't know to check it. But I've read them now and corrected a number of places based on your comments and added a few replies to the review. Thanks for the review.
msg236700 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-02-26 19:55
Emails from Rietveld often can be found in the spam folder.
msg236704 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-02-26 20:40
Paul, don't click on Reply below the comment on the main page, it never works. Instead click the link above the comment, then click on the comment to unfold it, click on the Reply below the comment, enter your reply, press Send, and after replying all comments click "Publish+Mail Comments" and then press "Pubblish All My Drafts".

Your replies by email are not archived and are received only by these who already is added comments on Rietveld or added to the nosy list on Roundup.
msg236705 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-02-26 20:53
As for zipimport, it doesn't support namespace packages when no 'directory' entry exists (issue14905). The zipfile module CLI now adds entries for directories (issue22219).
msg236707 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2015-02-26 21:32
Serihy, thanks for the explanation. I was aware that my replies weren't getting archived, good to know how to avoid that in future.

I've worked out what you mean over the directory entries now. It was easy enough to fix - I just removed the is_file check before writing entries to the zipfile. Test added so it's checked in future as well.
msg236736 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2015-02-27 11:22
Updated patch including (I hope!) all review comments so far.
msg236740 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-02-27 12:36
There is not well known old way how to make executable Python files on Windows. Add line "@python -x %0 %*" at the start of your Python script and rename the script to *.bat. The "-x" option makes Python to skip first line that is not Python, and @ forbids echoing executed command in bat-file.

Could zipapp support this feature?
msg236745 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2015-02-27 13:33
I'm -1 on this. The whole point of having a .pyz extension is so that you don't need to use an extension that's for files containing text to hold binary data.

If you want to do this, use zipapp and specify no interpreter line, then add the @python line yourself. Or just use a zip utility. Or build your own command line app. With the ability to supply an open file to create_archive, it's a 3-liner:

with open(dest_filename, 'wb') as f:
    f.write(b'@python -x %0 %*\n')
    zipapp.create_archive(source_dir, f)
msg237304 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2015-03-05 23:05
There have been no further comments for a while now. Could this be committed by someone?
msg237539 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2015-03-08 15:18
I left a couple of minor comments on Rietveld.
msg237544 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2015-03-08 16:59
New patch incorporating Berker's comments
msg237740 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2015-03-10 10:07
There's a problem with the latest patch, it's lost the MSI updates somehow. I'll add them back in this evening, please don't commit before a v6 patch is available,
msg237741 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2015-03-10 10:27
Steve Dower - could you please review the updated patch here. The WiX changes specifically are intended to register the .pyz extension with the launcher, but I can see no code that does this for a per-user install. I've copied the same code for the .py extension so I don't think I've missed anything, but I can't see how this would work in practice.
msg237764 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-03-10 13:06
The whole installer is run with a different global flag depending on the type of install. So it's automatic, though sometimes you can't do both per-user and all-user changes at the same time (not that you should anyway).
msg237766 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2015-03-10 13:28
OK, so if the installer changes look good to you, then this PR is once again ready to go. Sorry to anyone watching for the glitch.
msg238028 - (view) Author: Roundup Robot (python-dev) Date: 2015-03-13 14:42
New changeset d1e9f337fea1 by Brett Cannon in branch 'default':
Issue #23491: Implement PEP 441: Improving Python Zip Application Support
https://hg.python.org/cpython/rev/d1e9f337fea1
msg238029 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2015-03-13 14:45
Thanks for the patch, Paul! I touched up the formatting to be more PEP 8 compliant and fixed the tests as they initially failed for me (TESTFN didn't exist prior to you trying to make a directory, so it just failed; switched to tempfile.TemporaryDirectory instead).
msg238032 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2015-03-13 15:06
Thanks Brett! Apologies about the TESTFN issue - I changed it (from TemporaryDirectory, in fact) based on one of the review comments - it looks like I didn't test the change properly. My mistake, thanks for catching it.
History
Date User Action Args
2015-03-13 15:06:38paul.mooresetmessages: + msg238032
2015-03-13 14:45:26brett.cannonsetstatus: open -> closed

nosy: + brett.cannon
messages: + msg238029

resolution: fixed
stage: patch review -> resolved
2015-03-13 14:42:18python-devsetnosy: + python-dev
messages: + msg238028
2015-03-10 13:28:02paul.mooresetmessages: + msg237766
2015-03-10 13:06:38steve.dowersetmessages: + msg237764
2015-03-10 10:27:16paul.mooresetfiles: + pep-441.v6.patch

messages: + msg237741
2015-03-10 10:07:42paul.mooresetmessages: + msg237740
2015-03-08 16:59:35paul.mooresetfiles: + pep-441.v5.patch

messages: + msg237544
2015-03-08 15:18:44berker.peksagsetnosy: + berker.peksag

messages: + msg237539
stage: patch review
2015-03-05 23:05:54paul.mooresetmessages: + msg237304
2015-02-27 13:33:23paul.mooresetmessages: + msg236745
2015-02-27 12:36:44serhiy.storchakasetmessages: + msg236740
2015-02-27 11:22:56paul.mooresetfiles: + pep-441.v4.patch

messages: + msg236736
2015-02-26 21:32:07paul.mooresetmessages: + msg236707
2015-02-26 20:53:51serhiy.storchakasetmessages: + msg236705
2015-02-26 20:40:59serhiy.storchakasetmessages: + msg236704
2015-02-26 19:55:02serhiy.storchakasetmessages: + msg236700
2015-02-26 18:57:20paul.mooresetmessages: + msg236698
2015-02-26 17:55:13serhiy.storchakasetmessages: + msg236694
2015-02-26 17:41:19ethan.furmansetnosy: + ethan.furman
2015-02-26 17:00:28paul.mooresetfiles: + pep-441.patch

messages: + msg236689
2015-02-26 16:59:58paul.mooresetfiles: - pep-441.patch
2015-02-26 16:57:18paul.mooresetfiles: + pep-441.patch

messages: + msg236688
2015-02-26 16:19:18paul.mooresetmessages: + msg236682
2015-02-26 15:57:08dholthsetmessages: + msg236680
2015-02-26 15:37:44steve.dowersetmessages: + msg236678
2015-02-26 15:30:58paul.mooresetmessages: + msg236676
2015-02-26 15:29:10steve.dowersetmessages: + msg236675
2015-02-26 15:18:11Jim.Jewettsetnosy: + Jim.Jewett
messages: + msg236673
2015-02-26 14:41:15paul.mooresetmessages: + msg236672
2015-02-26 14:18:30paul.mooresetmessages: + msg236670
2015-02-26 13:56:18dholthsetmessages: + msg236669
2015-02-24 19:34:40paul.mooresetmessages: + msg236534
2015-02-24 19:26:17dholthsetmessages: + msg236531
2015-02-23 18:28:16steve.dowersetmessages: + msg236459
2015-02-23 18:25:52dholthsetmessages: + msg236458
2015-02-23 18:19:32steve.dowersetmessages: + msg236457
2015-02-23 18:16:07dholthsetnosy: + dholth
messages: + msg236455
2015-02-20 17:01:36paul.mooresetfiles: + pep441.patch

messages: + msg236308
2015-02-20 16:23:29steve.dowersetmessages: + msg236301
2015-02-20 15:22:54paul.mooresetmessages: + msg236291
2015-02-20 14:12:52steve.dowersetmessages: + msg236286
2015-02-20 14:12:08steve.dowersetmessages: + msg236285
2015-02-20 13:55:53serhiy.storchakasetnosy: + serhiy.storchaka
2015-02-20 12:52:51paul.moorecreate