classification
Title: "make distclean" deletes files under .hg directory
Type: behavior Stage: resolved
Components: Build Versions: Python 3.3, Python 3.4, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: eric.smith Nosy List: barry, eric.smith, lemburg, python-dev, r.david.murray, vstinner
Priority: normal Keywords: patch

Created on 2013-06-27 01:27 by eric.smith, last changed 2013-07-02 16:36 by r.david.murray. This issue is now closed.

Files
File name Uploaded Description Edit
make_distclean.patch vstinner, 2013-06-27 07:36 review
Messages (10)
msg191928 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2013-06-27 01:27
See:
http://mail.python.org/pipermail/python-dev/2013-June/127068.html

The find command:
	find $(srcdir) '(' -name '*.fdc' -o -name '*~' \
			   -o -name '[@,#]*' -o -name '*.old' \
			   -o -name '*.orig' -o -name '*.rej' \
			   -o -name '*.bak' ')' \
			   -exec rm -f {} ';'

will walk into .hg (and other dot directories) and delete files there. The particular problem noted in the email was a filename beginning with '@'.

Some suggestions are to change it to:
  find \( -type d -name .hg -prune \) -o ...
or
  find $(srcdir)/[a-zA-Z]* ...

Other suggestions are included in the thread starting at the above link. These include deleting the find command, or using "hg purge".

I think that we have enough history of using and suggesting "make distclean" that we should just fix up the find command to do what it does now, but not recurse into any dot directories (.hg, .svn, .git, etc.).
msg191933 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-06-27 07:36
Extract of "GNU standards":

# make distclean
#      Delete all files from the current directory that are created by
#      configuring or building the program.  If you have unpacked the
#      source and built the program without creating any other files,
#      `make distclean' should leave only the files that were in the
#      distribution.

Python should only remove files generated by the build process: Makefile, pyconfig.h, *.o, etc. But not files generated by Mercurial (.orig, .rej), backup files (.old, .bak), nor any other files which is not generated by the build. These files may be important for the developer (Mercurial keeps modified files after "hg revert --all" for example).

If someone wants an hardcore cleaner, the command can be kept, but please, under a different name.

I use "make distclean" when I change compiler options (rerun Makefile), when a new C file is added (calling "make" is usually not enough, and I prefer to restart from a "clean" source tree), or more generally when Python doesn't compile for an unknown reason. For example, "make distclean" was need when the new _stat (Modules/_stat.c) module was added.

R. David Murray wrote on python-dev: "We also sometimes ask someone reporting an issue to do a make distclean and recompile (...)".

Here is a patch simply removing the "find -exec rm {};" command from "make distclean". Remaining commands:

--------------
# Make things extra clean, before making a distribution:
# remove all generated files, even Makefile[.pre]
# Keep configure and Python-ast.[ch], it's possible they can't be generated
distclean: clobber
        for file in Lib/test/data/* ; do \
            if test "$$file" != "Lib/test/data/README"; then rm "$$file"; fi; \
        done
        -rm -f core Makefile Makefile.pre config.status \
                Modules/Setup Modules/Setup.local Modules/Setup.config \
                Modules/ld_so_aix Modules/python.exp Misc/python.pc
        -rm -f python*-gdb.py
--------------
msg191934 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2013-06-27 08:45
On 27.06.2013 09:36, STINNER Victor wrote:
> 
> STINNER Victor added the comment:
> 
> Extract of "GNU standards":
> 
> # make distclean
> #      Delete all files from the current directory that are created by
> #      configuring or building the program.  If you have unpacked the
> #      source and built the program without creating any other files,
> #      `make distclean' should leave only the files that were in the
> #      distribution.
> 
> Python should only remove files generated by the build process: Makefile, pyconfig.h, *.o, etc. But not files generated by Mercurial (.orig, .rej), backup files (.old, .bak), nor any other files which is not generated by the build. These files may be important for the developer (Mercurial keeps modified files after "hg revert --all" for example).
> 
> If someone wants an hardcore cleaner, the command can be kept, but please, under a different name.

Hmm, but distclean is exactly the kind of cleaner it's meant to be:
you use the command to prepare for a source code distribution and
you don't want any backup files or failed patch traces in your
source code distribution.

Perhaps we should introduce a softer version that leaves the
files you mention in untouched, so it can be used by developers,
e.g. make devclean.
msg191943 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2013-06-27 11:41
My plan is to just fix this issue, right now, by changing the find command to be:
   find $(srcdir)/[a-zA-Z]* ...

That fixes this bug and keeps the current functionality.

If someone wants to open another issue about changing what distclean does, I think that's okay. I suggest discussing it on python-dev first.
msg192125 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-07-01 12:20
Is there at least one developer against my change make_distclean.patch? If not, I suggest to apply this patch.
msg192126 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-07-01 12:25
Yes, I am against it.  See python-dev for reason (it would no longer be a 'distclean').
msg192194 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-07-02 13:08
New changeset 5896f887a93a by Eric V. Smith in branch '2.7':
Closes #18312: 'make distclean' no longer deletes files in dot-directories.
http://hg.python.org/cpython/rev/5896f887a93a

New changeset 910ec3471d55 by Eric V. Smith in branch '3.3':
Closes #18312: 'make distclean' no longer deletes files in dot-directories.
http://hg.python.org/cpython/rev/910ec3471d55

New changeset 8e838598eed1 by Eric V. Smith in branch 'default':
#18312: merge from 3.3.
http://hg.python.org/cpython/rev/8e838598eed1
msg192195 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-07-02 13:42
Unfortunately make distclean is now failing on the buildbots.

Which reminds us that there is another place that that target, including the 'find' at issue, is used.

Presumably there is an issue with tests not cleaning up after themselves that ought to be fixed, but for now we just need to re-fix make distclean.

http://buildbot.python.org/all/builders/x86%20Tiger%202.7/builds/2030/steps/clean/logs/stdio
msg192196 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-07-02 14:08
Hmm.  I may be pointing the finger at the wrong thing here...looking at more logs I'm now not sure what is up with the buildbots.
msg192210 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-07-02 16:36
OK, this looks like a false alarm.  I still don't understand those @ failures or the input error on the makefile, but there doesn't seem to be a problem with this patch.
History
Date User Action Args
2013-07-02 16:36:13r.david.murraysetstatus: open -> closed

messages: + msg192210
stage: needs patch -> resolved
2013-07-02 14:08:31r.david.murraysetmessages: + msg192196
2013-07-02 13:42:39r.david.murraysetstatus: closed -> open

messages: + msg192195
stage: resolved -> needs patch
2013-07-02 13:08:49python-devsetstatus: open -> closed

nosy: + python-dev
messages: + msg192194

resolution: fixed
stage: resolved
2013-07-01 12:25:50r.david.murraysetnosy: + r.david.murray
messages: + msg192126
2013-07-01 12:20:16vstinnersetmessages: + msg192125
2013-06-27 11:41:39eric.smithsetassignee: eric.smith
messages: + msg191943
2013-06-27 08:45:03lemburgsetnosy: + lemburg
messages: + msg191934
2013-06-27 07:36:23vstinnersetversions: + Python 3.3, - Python 3.5
2013-06-27 07:36:17vstinnersetfiles: + make_distclean.patch

nosy: + vstinner
messages: + msg191933

keywords: + patch
2013-06-27 02:04:51barrysetnosy: + barry
2013-06-27 01:27:44eric.smithcreate