pysetup3 install should not only record files that it installs, but also any directories it creates. pysetup3 remove should then delete those directories (if empty after removing their installed contents).

I see that there's code to do this in install.py, but for some reason it seems not to be working in Windows. Investigating.

More info on the problem (not Windows specific): if the installation creates a file several levels down

a/b/c/file

then during removal, only a/b/c is regarded as a candidate for removal; a/b and a are not considered as deletion candidates, and so never deleted, even if empty (apart from subdirectories).

IMO in this situation, a and b were created by the installation and so should be removed, unless they contain files not added during installation.

> IMO in this situation, a and b were created by the installation and so
> should be removed, unless they contain files not added during installation.

I share that opinion.  So, the first step would be to record created directories.  We could extend PEP 376 and write entries without hash nor length record for directories in RECORD, or use another file in the dist-info dir.

> I share that opinion.  So, the first step would be to record created 

> directories.  We could extend PEP 376 and write entries without hash nor length 
> record for directories in RECORD, or use another file in the dist-info dir.

IMO using RECORD would be preferable to another file, otherwise RECORD is not a complete record :-(

If one project creates a/b/thing and another project uses a/b/otherthing, then the directories would be recorded in the first project’s RECORD, but they should be removed only when both projects are removed.

> If one project creates a/b/thing and another project uses
> a/b/otherthing, then the directories would be recorded in the first
> project’s RECORD, but they should be removed only when both projects
> are removed.

I'm not sure what you mean by "using". AFAIK, each distribution's files (recorded in RECORD) would be unique to that distribution (else distros like Debian will have problems, since files are owned by one package and one package only).

I think all that is needed is as follows:

1. Record any directories that are created in RECORD, ideally bottom-up.

2. On uninstallation, remove all .pyc/.pyo files and __pycache__ directories, and remove all directories named in RECORD, unless they are non-empty (once .pyc/.pyo and __pycache__ are removed).

If you are talking about dependencies between dependencies, I still don't see what your point is. IIUC, if distA depends on distB, distA's RECORD will contain all files which were installed as part of the installation of distA, and likewise for distB.

s/dependencies between dependencies/dependencies between distributions/

> I'm not sure what you mean by "using". AFAIK, each distribution's files (recorded in RECORD)
> would be unique to that distribution (else distros like Debian will have problems, since
> files are owned by one package and one package only).
Files need to belong to one distribution only, but directories can be shared by more than one.  In my example, I meant that ProjectX can create the directories a and a/b and the file a/b/thing, then ProjectZ can create the file a/b/otherthing, and if we remove ProjectX and then ProjectZ, a and a/b won’t be in ProjectZ’s RECORD, because they were in ProjectX’s.

Carl: Can you tell us how pip removes directories?

> 1. Record any directories that are created in RECORD, ideally bottom-up.
To solve the problem I mentioned above (hopefully making sense this time :), we’d need to record all directories that would have been created.  It may be hard or even unfeasible to sort “directory that existed before Python was installed”, e.g. /usr/lib/python2.7/site-packages, and “directory created by one project”, e.g. /usr/lib/python2.7/site-packages/somelib.

s/directory that existed before Python was installed/directory that existed before any distribution was installed/

>s/directory that existed before Python was installed/directory that existed before any distribution was installed/

IMO there is no need to remember any directory which isn't actually created by pysetup3. Deleting a __pycache__ in a directory created by pysetup3 would be implicit, and not need to be recorded.

> IMO there is no need to remember any directory which isn't actually
> created by pysetup3

I did not propose such a thing.

> I did not propose such a thing.

Sorry, I misunderstood your reference to /usr/lib/python2.7/site-packages.

Let me rephrase my example with real paths.

Python creates /usr/lib/python3.3/site-packages.  (I’ll call this $stdlib.)

pysetup3.3 install paste.script creates $stdlib/paste/script/ and files therein.  The paste and paste/script directories are recorded.

pysetup3.3 install PasteUtil creates $stdlib/paste/util/ and files therein.  Only the paste/script directory is recorded.

pysetup3.3 remove PasteScript deletes the $stdlib/paste/script dir, but not its parent, as it’s not empty after removing all files and dirs in PasteScript’s RECORD.

pysetup3.3 remove PasteUtil deletes the $stdlib/paste/util dir.

The $stdlib/paste directory is created by either distribution when it is installed, but if it’s recorded only by one of the distributions, then it can’t be removed when the other distribution is removed last.  This is the problem I’m seeing.

> Let me rephrase my example with real paths.

Okay, now I see what you're getting at.

> Python creates /usr/lib/python3.3/site-packages.  (I’ll call this $stdlib.)
> 
> pysetup3.3 install paste.script creates $stdlib/paste/script/ and files 
> therein.  The paste and paste/script directories are recorded.
> 
> pysetup3.3 install PasteUtil creates $stdlib/paste/util/ and files therein.  
> Only the paste/script directory is recorded.

I think you mean "Only the paste/util directory is recorded."

> The $stdlib/paste directory is created by either distribution when it is 
> installed, but if it’s recorded only by one of the distributions, then it can’t 
> be removed when the other distribution is removed last.  This is the problem I’m 
> seeing.

Yes, it's a valid point. How about if you record all directories that you would create if they didn't exist, as well as those actually created? That way, you would record the paste directory under both distributions, but would only delete it when deleting the last distribution which uses it (paste/util in your example), because you're checking for emptiness (not including bytecode files and directories).

> Carl: Can you tell us how pip removes directories?

In short - pip would _love_ to have directories recorded as well as files, exactly as Vinay has proposed. We don't have that info (even the distutils --record option currently doesn't record directories, thus installed-files.txt doesn't contain directories), so we are reduced to some nasty things like referring to top-level.txt in order to avoid lots of empty directories hanging about, which in itself was the subject of recent controversy re Twisted's custom namespace packages implementation.

Please, let's have directories recorded in RECORD, and yes, if a directory would have been created but already existed, it should also be recorded (so that shared directories are in the RECORD file for both/all of the sharing distributions).

[Vinay]
> I think you mean "Only the paste/util directory is recorded."
Obviously :)

> How about if you record all directories that you would create if they didn't exist, as well
> as those actually created? That way, you would record the paste directory under both
> distributions, but would only delete it when deleting the last distribution which uses it
This is what I proposed earlier: we’d need to record all directories that would have been created, but I’m not sure if it will be possible.  For example, if one uses --prefix /tmp/usr and pysetup install creates /tmp/usr, /tmp/usr/lib, /tmp/usr/lib/python2.7, /tmp/usr/lib/python2.7/site-packages, /tmp/usr/lib/python2.7/site-packages/spam and /tmp/usr/lib/python2.7/site-packages/Spam-0.1.dist-info, then we pysetup should Spam, should packaging remove only the package and dist-info directories or also the site-packages, python2.7, lib and usr directories?

[Carl]
> In short - pip would _love_ to have directories recorded as well as files, exactly as Vinay
> has proposed. We don't have that info (even the distutils --record option currently doesn't
> record directories, thus installed-files.txt doesn't contain directories), so we are reduced
> to some nasty things like referring to top-level.txt in order to avoid lots of empty
> directories hanging about,
Oh.  Ouch.

> which in itself was the subject of recent controversy re Twisted's custom namespace
> packages implementation.
Ah, I understand that better now.  Maybe now would be a good time to start an alpha pip-on-distutils2 branch, to see if the API we offer is good enough and to work out possible bugs in new features offered to pip by d2 (removing directories, installing setup.cfg-based projects, supporting bdists).

> Please, let's have directories recorded in RECORD,
Okay, so I will champion a patch to PEP 376.

> This is what I proposed earlier: we’d need to record all directories that would have been created, but I’m not sure if it will be possible.  For example, if one uses --prefix /tmp/usr and pysetup install creates /tmp/usr, /tmp/usr/lib, /tmp/usr/lib/python2.7, /tmp/usr/lib/python2.7/site-packages, /tmp/usr/lib/python2.7/site-packages/spam and /tmp/usr/lib/python2.7/site-packages/Spam-0.1.dist-info, then we pysetup should Spam, should packaging remove only the package and dist-info directories or also the site-packages, python2.7, lib and usr directories?

I think it would make sense to draw a distinction between "creating the prefix directories (including site-packages)" and "creating the distribution-specific directories within the prefix directories." And only record the latter in RECORD for the given installed distribution.

If I use --prefix and install some things, and then uninstall them, I would not consider it a bug to find the empty site-packages directory still remaining under that prefix. (In fact, I'd be surprised if it were removed).

> Okay, so I will champion a patch to PEP 376.

Thank you!

Obsolete for distutils2, but something to keep in mind for RECORD 2.0 or its successor.