Issue33

If you're reporting an issue for setuptools 0.7 or higher, please use BitBucket

Title zipped eggs cause various problems -- perhaps set default setting to unzipped?
Priority wish Status resolved
Superseder Nosy List exarkun, ghazel, pje, zooko
Assigned To Keywords

Created on 2008-08-15.15:41:15 by zooko, last changed 2009-10-10.21:15:28 by pje.

Messages
msg373 (view) Author: pje Date: 2009-10-10.21:15:28
Per msg168, the "nest" tool will resolve this.  Closing since participants'
questions are answered and the issue itself is a won't-fix.
msg338 (view) Author: pje Date: 2009-07-28.17:15:54
At 03:05 PM 7/28/2009 +0000, Jean-Paul Calderone wrote:
>Two DLLs are included in the
>egg, and they can't be loaded from a zip file.  Possibly this could 
>be addressed
>by some setuptools option I'm not aware of, to get these DLLs extracted to a
>temporary directory at runtime where they can be loaded, but I 
>haven't been able
>to find such an option.

The option is eager_resources, passed to setup().  See:

http://peak.telecommunity.com/DevCenter/setuptools#new-and-changed-setup-keywords

(Also, if you have setuptools build the DLLs, it will handle this for you.)
msg337 (view) Author: exarkun Date: 2009-07-28.15:05:41
For pyOpenSSL, zipped eggs also cause a problem.  Two DLLs are included in the
egg, and they can't be loaded from a zip file.  Possibly this could be addressed
by some setuptools option I'm not aware of, to get these DLLs extracted to a
temporary directory at runtime where they can be loaded, but I haven't been able
to find such an option.  Even if such an option exists, making unzipped eggs the
default still makes sense to me, since the *default* behavior should be to work.

For now, I'm passing zip_safe=False in pyOpenSSL's setup.py, which is annoying,
since it produces a warning when setuptools isn't available.
msg336 (view) Author: zooko Date: 2009-07-28.14:25:34
P.S.  Oh, I see that this issue with py2exe was the same problem I encountered 15 
months ago -- msg85 .
msg335 (view) Author: zooko Date: 2009-07-28.14:24:28
On this thread on distutils-sig, I suggest that the Distribute fork might make 
unzipped be the default:

http://mail.python.org/pipermail/distutils-sig/2009-July/012816.html
msg334 (view) Author: zooko Date: 2009-07-28.14:20:53
So this just happened again -- it turns out that py2exe won't include dependencies 
packages into the package it is building if they are present as zipped eggs, but 
it will if they are present as unzipped eggs.  The workaround from msg223 fixed 
the problem -- remove the dependencies, reinstall them after setting the distutil 
config file to always-unzip, and then py2exe picks up the dependencies correctly.
msg333 (view) Author: zooko Date: 2009-07-28.14:04:14
By the way, since nobody can remember where the Python distutils config files go 
on Windows:

http://www.python.org/doc/2.5.4/inst/config-syntax.html
msg224 (view) Author: zooko Date: 2008-12-23.14:19:43
Here's another person who had a problem with {{{co_filename}}} not working in a 
zipped egg:

http://mail.python.org/pipermail/distutils-sig/2008-December/010649.html
msg223 (view) Author: zooko Date: 2008-12-23.14:15:02
Suggested workaround for people finding this ticket from a search engine query 
such as "distutils zip problem":

1.  Put the following in your distutils config file:

{{{
[easy_install]
zip_ok=False
}}}

Then all packages that you easy_install will be unzipped.

2.  When creating a package for others, pass {{{zip_ok=False}}} to 
{{{setup()}}}.  This is a bit nasty, because that flag is really supposed to 
indicate whether there is something in the package that can't work from zip, and 
if you set that flag then this abrogrates the user's preference when installing 
-- there is no way for them to tell easy_install to install the package zipped 
in spite of the {{{zip_ok=False}}} flag.  However, I have seen so many reports 
of problems with zip files, and zero reports of problems with unzipped files, 
and zero requests for zipped files, so I'm personally comfortable setting 
{{{zip_ok=False}}} just to help the user avoid the problems.
msg177 (view) Author: ghazel Date: 2008-09-18.20:39:10
I had problems with always-unzip that lead me to this same conclusion. As a 
developer I would much rather be able to read source files than save a few 
bytes. This is not something that improved Python or 3rd party tools can really 
support (my editor and shell are not likely to add support for opening files 
from zips transparently).

I'm not sure why the zip file would be any easier, since the install command 
is "easy_install foo" either way. I have distributed py2exe'd apps, and they 
zip themselves when built. The zip tradeoff is worth it in this case - users 
don't need to read tracebacks or open source files (they are pycs anyways) 
since they probably don't even have python (or easy_install) outside of my app. 
If they do have python, I would like for them to be able to use easy_install to 
get my source distribution in a full and unzipped form.

It sounds like "nest" is going to work this way, though. I look forward to that!
msg168 (view) Author: pje Date: 2008-09-11.16:01:31
Pronouncement: this will not be changed in any version of easy_install.  In
"nest" (its successor) however, installation will follow the distutils pattern
(i.e. no .egg directory) and use a file list for uninstall.

Rationale: Skip's issue isn't going to be resolved by avoiding zipfiles in the
easy_install case.  Really, none of the other issues you've mentioned in this
ticket will be either.  Zipfiles are a useful installation target for
drop-and-go plugin installation, and they're useful for py2exe, py2app, etc.  A
library that has problems being deployed in a zipfile will have problems in
other places besides easy_install.  The fix isn't to stop using zipfiles, it's
to fix the places in Python (and 3rd-party tools) where zipfile support isn't
finished.  easy_install simply forces the errors to show up earlier... and for
the community as a whole, that's a *good* thing.
msg161 (view) Author: zooko Date: 2008-09-10.17:24:14
Here's the latest person -- Skip Montanaro -- who was inconvenienced by the
default-zip policy:

http://mail.python.org/pipermail/distutils-sig/2008-September/009970.html

By the way, someone else -- I think it might have been Steve Holden ?? --
mentioned that he tested it and that unzipped eggs load faster than zipped eggs.
msg85 (view) Author: zooko Date: 2008-08-15.15:41:14
per http://mail.python.org/pipermail/python-dev/2008-April/078505.html

installing an egg (with easy_install or ./setup.py install) in zipped form
results in stack traces without source code lines attached. I hereby propose
that zip_safe=False should be the default, unless the package producer and/or
the package consumer specifically indicates that they prefer to zip the egg.

I might be persuaded to change my mind about this if I saw some performance
measurements which showed that zip_safe=False had a high cost. I'm skeptical
that it has a high cost, since I've used it extensively for months (habitually
specifying zip_safe=False in my setup.py files) and I haven't noticed any
performance problem. 

PJE wrote, on the python-dev thread:

"Are you using Python 2.5?  As of 2.5, the linecache module should correctly
read the source line from the present location of the source file the module was
loaded from, regardless of the file name specified in the traceback."

Yes, this was with Python 2.5.1. Note that the module was loaded from a .pyc
file inside a zipped .egg (in the failing case), or from a .pyc file in an
unzipped .egg (in the succeeding case). In the succeeding case, the name of the
.py file, which is encoded inside the .pyc file, had been rewritten during the
install process. 

 Changed 4 months ago by zooko  ¶

Here is another case that I encountered yesterday in which eggs being zipped
caused a problem: we're using py2exe to build a Windows executable. (Also we are
using py2app to build a Mac app.)

py2exe can't find Python modules inside zipped eggs, although it can of course
find them in unzipped eggs if the egg is added to the PYTHONPATH. At one point
my programming partner Rob Kinninmont was using a script named ["hatch-eggs.py"
http://allmydata.org/trac/tahoe/browser/misc/hatch-eggs.py?rev=1879]. That
script no longer appears to be needed in our build system, but at this moment I
can't figure out how we give py2exe access to the modules from the eggs that we
installed... It is possible that we've customized the build machine by setting a
distutils config file to say "always_unzip=True". I hope not, because that would
mean our build will fail in mysterious ways when run on a different machine. 

 Changed 4 months ago by zooko  ¶

Oh, there it is: our [setup.cfg
http://allmydata.org/trac/tahoe/browser/setup.cfg?rev=1938]. This means our
Windows build will work on other machines, unless they have one of our
dependencies already installed and that dependency is installed in zipped form.
In that case, we'll get some strange error quite late in the process -- possibly
something like the module not being found at run-time.

So, this is another example of how unzipped eggs Just Work in more contexts
(py2exe) than zipped ones do. 

 Changed 4 months ago by nbecker  ¶

zipped eggs also breaks pydoc:

http://permalink.gmane.org/gmane.comp.python.devel/93788

Also, not very convenient if you need to view the source or debug. 



Changed 1 month ago by zooko ¶

Here's another one:

http://mail.python.org/pipermail/distutils-sig/2008-July/009694.html
Changed 1 month ago by zooko ¶

Here's another person who had trouble which he wouldn't have had if eggs were
unzipped by default:

http://mail.python.org/pipermail/distutils-sig/2007-December/008554.html 


Okay, here's the latest entry: my co-worker Greg Hazel, not knowing about the
setup.cfg hack that I described earlier in this ticket, installed one of Tahoe's
dependencies onto a build machine, using the default behavior of easy_install. 
Therefore, later there was a mysterious run-time error in the resulting package
of Tahoe.  Unfortunately I didn't catch this and explain it to Greg right away,
so instead he spent some of his time figuring it out himself.  After
working-around the problem somehow, he reported to his manager: "fixed. you can
say thank you to python eggs for screwing up py2exe's automatic dependency
detection".

(This ticket was imported from my interim setuptools bug tracker:
http://allmydata.org/trac/setuptools/ticket/4
)
History
Date User Action Args
2009-10-10 21:15:28pjesetstatus: chatting -> resolved
messages: + msg373
2009-07-28 17:15:54pjesetmessages: + msg338
2009-07-28 15:05:41exarkunsetnosy: + exarkun
messages: + msg337
2009-07-28 14:25:34zookosetmessages: + msg336
2009-07-28 14:24:28zookosetmessages: + msg335
2009-07-28 14:20:54zookosetmessages: + msg334
2009-07-28 14:04:15zookosetmessages: + msg333
2008-12-23 14:19:43zookosetmessages: + msg224
2008-12-23 14:15:02zookosetstatus: resolved -> chatting
messages: + msg223
2008-09-22 16:48:15pjesetstatus: chatting -> resolved
2008-09-18 20:39:11ghazelsetstatus: resolved -> chatting
nosy: + ghazel
messages: + msg177
2008-09-11 16:01:31pjesetstatus: chatting -> resolved
nosy: + pje
messages: + msg168
2008-09-10 17:24:14zookosetstatus: unread -> chatting
messages: + msg161
2008-08-16 02:55:51pjesetpriority: bug -> wish
2008-08-15 15:41:15zookocreate