Hi Tarek,


moving a package into place right at the end is the best thing to do I
think.

It solves a couple common problems:
    - broken packages after an install is stopped half way for one of
many common reasons.
    - old files left around will not be there, since they would be
moved.  numpy 1.4 breaks for example if installing over an old version
first and not removing the old version.  So did pygame in the past...
and likely other packages.
    - race condition whilst installing (not so common, but happens).

I'm not sure I did say something like 'this race condition can happen
even if you copy the file in some temp dir, then move them to python.'.
 But now I see a problem... if the existing directory has files in it,
you can not move another directory over the top of it atomically.

I don't think there is a way to solve this completely, without changing
the packaging system quite dramatically.  However we can make it much
more likely we will win the race!!!  First trying to install into a temp
directory, then if on the same file system, moving the old directory to
a backup one, then moving the directory into place, finally removing the
old directory.

This would reduce the time of the race condition to a fraction of a
second(0.01 seconds on my box) compared to many seconds on average, and
longer for big packages with big files.

Packages not installing correctly is a big problem for C/C++ packages
more so than simply python packages.

Fixing this will mean distutils becomes more robust, and also will stop
a security issue.


I tried a method similar to this pseudo code as a work around:
python setup.py install --home=/tmp/apackagetmp
move temp package into place...

Here is a script I started on... for pygame installs on ubuntu karmic koala.
http://rene.f0o.com/~rene/stuff/safer_install.py

I thought I'd try it out as a separate thing first to try and get it
right before trying a patch with distutils.  There's still a bunch of
issues with it, and it needs tests written.
- when installing to a root place the permissions are not correct.  They
are the ones of the user it is ran as.  ie, 'chmod -R root:staff dest'
would probably fix it.
- it would be good to extract the paths for various things from
distutils... rather than guessing.  For example the dist-packages
directory, where the include files are etc etc.
- I'm going to put in a test step.  So it installs to the temp directory
first, then tries to run the package tests before installing.  The idea
being that if all the tests past *then* install.  This should make it
more robust still.  It will probably try as a separate process with
"python -m apackage.tests.__main__" or something like that.
- it always copies in all of the files.  Which is good and bad.  It
means that there won't be old files, but it also means that it can't
avoid work... like say only one file changes.
- probably many other issues...


cheers,

ps. happy new year!