classification
Title: 'make touch' does not work with git clones of the source repository
Type: enhancement Stage: needs patch
Components: Build Versions: Python 3.7, Python 3.6, Python 3.5, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Chi Hsuan Yen, berker.peksag, brett.cannon, ezio.melotti, haypo, josh.r, koobs, martin.panter, ncoghlan, ned.deily, python-dev, vlee, willingc, zach.ware
Priority: critical Keywords: buildbot, patch

Created on 2015-02-06 22:46 by vlee, last changed 2017-05-05 10:29 by ncoghlan.

Files
File name Uploaded Description Edit
0001-Issue-23404-Do-not-use-Python-2.7-constructs-in-Pars.patch vlee, 2015-02-07 06:17 Do not use Python 2.7 constructs in Parser/asdl.py.
iss23404devguide.patch willingc, 2015-07-28 09:17
boot-flag.patch martin.panter, 2016-09-13 11:29 review
boot-flag.patch Chi Hsuan Yen, 2017-02-10 09:25 Based on Martin's version, with a minor fix review
boot-flag.py3.5.patch martin.panter, 2017-04-03 13:06 Needs _freeze_importlib line merged review
boot-flag.py2.patch martin.panter, 2017-04-03 13:06 review
Pull Requests
URL Status Linked Edit
PR 1405 merged haypo, 2017-05-02 23:37
PR 1461 merged haypo, 2017-05-04 17:12
PR 1464 merged haypo, 2017-05-04 21:09
PR 1465 merged haypo, 2017-05-04 23:56
PR 1466 merged haypo, 2017-05-05 00:51
Messages (40)
msg235505 - (view) Author: Vinson Lee (vlee) * Date: 2015-02-06 22:46
Is there a minimum Python requirement to build Python?

Python 3.5 does not build with Python 2.6. Python 3.4, Python 3.3, and Python 2.7 build with Python 2.6 so this is recent change in build requirements.

For example, this build failure occurs on CentOS 6.

python ./Parser/asdl_c.py -h Include ./Parser/Python.asdl
Traceback (most recent call last):
  File "./Parser/asdl_c.py", line 6, in <module>
    import asdl
  File "Parser/asdl.py", line 36
    builtin_types = {'identifier', 'string', 'bytes', 'int', 'object', 'singleton'}
                                 ^
SyntaxError: invalid syntax
msg235508 - (view) Author: Josh Rosenberg (josh.r) * Date: 2015-02-07 02:44
Looks like this revision, part of #22823, changed it: https://hg.python.org/cpython/rev/4480506137ed
msg235509 - (view) Author: Josh Rosenberg (josh.r) * Date: 2015-02-07 02:51
To answer your question: Since asdl uses set literals (introduced in 2.7 and 3.0), that appears to put an effective minimum version requirement of 2.7 to build 3.5. Whether that was the intent is unclear (they were changing a lot of places from set([a, b, c]) to {a, b, c}; causing problems building Python was probably not the intent, but 2.6 is well out of support at this point).
msg235510 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2015-02-07 03:21
Try using "make touch" before "make".  Because hg does not preserve precise time stamps when creating working directories, some build steps are run unnecessarily after an initial checkout.  'make touch' updates the file time stamps so that these steps are skipped and a "bootstrap" Python is not needed.
msg235511 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2015-02-07 03:23
https://docs.python.org/devguide/setup.html#avoiding-re-creating-auto-generated-files
msg235513 - (view) Author: Vinson Lee (vlee) * Date: 2015-02-07 06:17
I am building from a git copy of the source repository. "make touch" before "make" does not work for me.

$ make touch
cd .; \
	hg --config extensions.touch=Tools/hg/hgtouch.py touch -v
abort: There is no Mercurial repository here (.hg not found)!

I've attached a patch that addresses the Python 3.5 build issue for me. The patch contains two syntax construct changes to Parser/asdl.py. Please consider applying this patch upstream.
msg235514 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2015-02-07 06:34
Unfortunately, those syntax differences aren't the only problems you could run into so making these syntax changes isn't really a general solution, IMO. 'make touch' is the documented and supported way to ensure that the unnecessary build steps are not run.  But, as you discovered, 'make touch' only works with an hg copy of the repo; it was not designed for git clones.  To fully support git clones, there would need to be a git equivalent of the hg scripts called by 'make touch'.  This would be a worthwhile project for someone interested in git support to work on and would be considered for inclusion in the repo.
msg235515 - (view) Author: Vinson Lee (vlee) * Date: 2015-02-07 06:52
With the supplied patch on CentOS 6 , "make" passes and "make test" passes with the exception of test_readline that is Issue19884.

372 tests OK.
1 test failed:
    test_readline
7 tests altered the execution environment:
    test_calendar test_distutils test_float test_locale test_strptime
    test_types test_warnings
11 tests skipped:
    test_devpoll test_gdb test_kqueue test_msilib test_ossaudiodev
    test_startfile test_tk test_ttk_guionly test_winreg test_winsound
    test_zipfile64
Re-running failed tests in verbose mode
Re-running test 'test_readline' in verbose mode
testHistoryUpdates (test.test_readline.TestHistoryManipulation) ... ok
test_write_read_append (test.test_readline.TestHistoryManipulation) ... ok
test_init (test.test_readline.TestReadline) ... FAIL
msg235516 - (view) Author: Vinson Lee (vlee) * Date: 2015-02-07 07:08
I tried a hg copy of the repository. "make touch" worked for me.

./configure
make touch
make
msg247510 - (view) Author: Carol Willing (willingc) * Date: 2015-07-28 09:17
Using Ned Deily's feedback, I have added a devguide patch to clarify that 'make touch' only works with a Mercurial clone and was not designed for git clones. This patch, iss23404devguide.patch, resolves the devguide portion of this issue unless anyone has further wording changes.
msg247574 - (view) Author: Roundup Robot (python-dev) Date: 2015-07-29 13:58
New changeset ffbb19026599 by Berker Peksag in branch 'default':
Issue #23404: Add a note to clarify that "make touch" only works with a hg clone.
https://hg.python.org/devguide/rev/ffbb19026599
msg247575 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2015-07-29 14:07
Patch LGTM. Thanks Carol.

I will left this issue open in case someone wants to work on a git equivalent of "make touch" as described by Ned in msg235514.
msg275285 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-09-09 08:09
Perhaps you can use “make -t <file>” instead of “make touch”. If you know the filename you want to avoid regenerating, that would not need any extra tools, no Mercurial or Git or any other version control. E.g.:

make -t Include/Python-ast.h
msg276252 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-09-13 11:29
Okay so my “make -t” trick has various flaws. You still have to know the filenames to specify, it requires Makefile to be generated by configure in the source tree, and it creates empty files if you use it in a separate build directory.

Another idea: instead of the “make touch” recipe, we add a simple shell script. Call it say “touch-bootstrap.sh”. Like what “make touch” already does in Python 2, but without embedding it as a makefile rule.

Or add a flag variable so you can do a build without running any of the regeneration rules or worrying about timestamps: make BOOT="#".
msg287493 - (view) Author: Chi Hsuan Yen (Chi Hsuan Yen) * Date: 2017-02-10 09:25
Just tried boot-flag.patch with https://github.com/python/cpython/commit/16e07881bd3867d471dd0a25c5117672b65c90ee. Doesn't work out-of-box:

#./Programs/_freeze_importlib \
    ./Lib/importlib/_bootstrap.py Python/importlib.h
/bin/sh: line 1: ./Lib/importlib/_bootstrap.py: Permission denied
make: *** [Makefile:749: Python/importlib.h] Error 126

My fix is: merge the two lines into a single one. See my attached patch.

By the way, the BOOT hack is quite similar to GENERATED_COMMENT in introduced in issue27641. Maybe they can merge?

My environment:

* Arch Linux x86_64
* /bin/sh symlinks to bash, which is 4.4.12
* GNU Make 4.2.1
msg287580 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2017-02-11 00:38
I've disabled the Touch step on the buildbots until this is fixed.
msg291069 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2017-04-03 13:06
I had mainly been using my boot-flag.patch with BSD Make (bmake) rather than Gnu Make. It seems I was relying on a bug in BSD Make that merges escaped newlines in command lines, despite Posix and Gnu Make. Anyway, Chi’s fix seems appropriate.

I am posting the patches I used for 3.5 and 2.7, in case others are interested. The 3.5 one would also need Chi’s fix merging the “_freeze_importlib” line.

It may be possible to have the configure script adjust the default for the BOOT flag rather than setting GENERATED_COMMENT. But the side effect would be disabling regeneration of additional things (e.g. that aren’t specifically broken with cross compilation.

Anyway my preference was to stop automatically regenerating files, and to keep the build system as simple as practical without hacks like the BOOT="#" flag.
msg292067 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-04-21 16:57
FYI 3 FreeBSD buildbots are currently broken because of this issue.
msg292068 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-04-21 17:00
> Anyway my preference was to stop automatically regenerating files, and to keep the build system as simple as practical without hacks like the BOOT="#" flag.

In the past, this issue bite me so many times on Solaris, FreeBSD, etc. One because there was not "python" command, then because "python" was an old python 2.6 and so hg touch didn't work, etc.

It's "nice" to try to keep track of build dependencies, but in pratice it is super painful. So I'm also in favor of removing these build dependencies based on file modification file, and maybe document how to force a rebuild (do you really need to do that? developers hacking special files know how to rebuild generated files, no?).
msg292092 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2017-04-22 00:17
Last time I proposed removing the automatic rebuilding of checked-in generated files, it seemed getting a consensus would not be trivial. Nick seemed strongly against changing the status quo: <https://mail-archive.com/search?l=mid&q=CADiSq7d2NV0hbcQB1bYvc9M2qUYZiYC7RsG=fnw7tdWia2YmQg@mail.gmail.com>. He did say it has been too hard to remember how to rebuild generated files in the past.

Documenting the quirks of the build system would be good even if we keep the automatic regeneration.
msg292126 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2017-04-22 16:50
Would it make sense to have a `make rebuild` or something which explicitly ignores prebuilt files? That could then also serve as non-obvious documentation on how to build those files in the first place.
msg292143 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2017-04-23 01:02
Do you mean a separate makefile rule to rebuild the generated files, rather than rebuilding them in a normal “make all” build? I would support this; this is what I meant with my “make regenerate” proposal in response to Nick linked above.
msg292170 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2017-04-23 16:06
Yep, that's exactly what I was suggesting, Martin (and sorry if I missed you saying that earlier; on vacation so trying to keep email/open source time to minimum :) .
msg292194 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-04-24 06:01
As Martin notes, my concern is that I can go for years without having to touch the low level files (since syntax changes are rare), so it's important to me to make it easy to remember and/or relearn how to regenerate them.

However, suitable make commands (e.g. a "make regen" target, with "make regen-X" subtargets) would qualify as "easy to remember", and would also be amenable to automated testing in combination with a `make clobber-gen` command.

(Tangent on target naming: I think 'rebuild' would be confusing, while 'regenerate' is a bit long, especially if there are more specific subtargets. 'regen' is just a shortened version of the latter)
msg292207 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2017-04-24 08:43
A while ago I wrote a patch targetting Issue 22359 that may be a starting point for “make regen”: <https://bugs.python.org/file42169/separate-regen.patch>. It pulled out three recipes into separate “phony” targets: “make graminit importlib importlib_external”.
msg292822 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-05-02 23:44
I rewrote Python (UNIX/BSD) build system to not rebuild generated files based on file modification time: the action is now explicit. This change should not only fix buildbots, but also ease cross-compilation, and more generally make Python build less painful! No more funny error when the system only provides Python 2.6 and so "make touch" doesn't work, whereas "make touch" was the obvious workaround to avoid trying to regenerated files.

The change:
https://github.com/python/cpython/pull/1405

My change adds a global "rebuild-all", but this command is made of multiple subcommands:
    
      - rebuild-ast: Include/Python-ast.h and Python/Python-ast.c
      - rebuild-grammar: Include/graminit.h and Python/graminit.c
      - rebuild-importlib: Python/importlib_external.h and Python/importlib.h
      - rebuild-opcode: Include/opcode.h
      - rebuild-opcode-targets: Python/opcode_targets.h
      - rebuild-typeslots: Objects/typeslots.inc

rebuild-all is not needed if you only want to update AST for example: just run "make rebuild-ast && make".

--

As you may expect, the change is quite big for a very sensitive part of Python: the build system. Do we want to backport such major change?

Another solution for stable branches to to remove the rebuild of generated files from configure and Makefile... Just hope that we will not need it anymore? Not sure if it's a good idea. I expect fixes in importlib, so "rebuild-importlib" will be needed at least.

Even if I don't trust my own work(!), I would suggest to review carefully my change, test it on all buildbots, play with it, and then backport it to 2.7, 3.5 and 3.6.

Note: Windows is not affected by this bug, right?
msg292823 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-05-02 23:45
The issue #23404 has been marked as a duplicate of this bug: "AMD64 FreeBSD 9.x 3.x" tries to rebuild Include/opcode.h, timestamp issue.
msg292829 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-05-03 00:56
https://github.com/python/cpython/pull/1405#pullrequestreview-35923931

Zachary Ware: "LGTM, but I haven't fully parsed how the old mess worked. We should also have CI (either Travis or buildbot) confirming that make rebuild-all doesn't leave any checked-in files modified."

Yeah, such CI would be useful.

--

If we go in this direction, maybe we can go further and also run autoconf && autoheader?

While fixing bpo-30232, I had to do a second commit (9ed34a89532763cf89f5e11fffb91ef7dee29fed) because my first one (5facdbb29169c2799c42f887cef4cd9d087b0167) only modified configure.ac. Well, it might be a bot on pull requests, but an extra check wouldn't hurt anyway ;-)

A problem with autoconf is that the result depends on the exact autoconf version :-/ For example, autoconf 2.69 of my Fedora 25 wants to remove runstatedir from configure. I don't know what is runstatedir, so I had to revert these autoconf changes on configure before pushing...

haypo@selma$ touch configure.ac && autoconf && git diff

diff --git a/configure b/configure
index a20cf97..a6b13e8 100755
--- a/configure
+++ b/configure
@@ -783,7 +783,6 @@ infodir
 docdir
 oldincludedir
 includedir
-runstatedir
 localstatedir
 sharedstatedir
 sysconfdir
(...)

A solution would be to use a fixed autoconf version and ensure that configure is always generated with the same autoconf version. Maybe Gentoo is a good OS for such CI such Gentoo provides multiple versions of autotools, whereas other Linux distro usually only provide one version of all autotools tools.
msg292855 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-05-03 07:42
Could we use "regen-*" as the pattern for the file regeneration targets?

I'd expect "make rebuild-all" to be akin to "make clean && make", since in the context of a C project, "build" maps to "generate the output binaries" for me.
msg292858 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-05-03 08:01
Nick Coghlan: "Could we use "regen-*" as the pattern for the file regeneration targets?"

Sure, done.

I checked the CPython documentation, build scripts, etc. I confirm that "rebuild" usually means "recompile Python from scratch", it's even an explicit action in the Windows build system.
msg292860 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-05-03 08:01
> If we go in this direction, maybe we can go further and also run autoconf && autoheader?

Oh, I just noticed that there is already a "make autoconf" target, I didn't know.
msg292913 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-05-03 16:21
New changeset a5c62a8e9f0de6c4133825a5710984a3cd5e102b by Victor Stinner in branch 'master':
bpo-23404: make touch becomes make regen-all (#1405)
https://github.com/python/cpython/commit/a5c62a8e9f0de6c4133825a5710984a3cd5e102b
msg292916 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-05-03 16:27
I just created the issue #30259: Test somehow that generated files are up to date: run make regen-all.
msg293002 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-05-04 17:14
I started a thread on python-dev to ask if it's ok to backport the change to 2.7, 3.5 and 3.6.

I started to backport the change to 3.6:
https://github.com/python/cpython/pull/1461
msg293018 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-05-04 21:29
New changeset b109a1d3360fc4bb87b9887264e3634632d392ca by Victor Stinner in branch 'master':
bpo-30273: Update sysconfig (#1464)
https://github.com/python/cpython/commit/b109a1d3360fc4bb87b9887264e3634632d392ca
msg293026 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-05-04 22:46
New changeset 9d02f562961efd12d3c8317a10916db7f77330cc by Victor Stinner in branch '3.6':
[3.6] bpo-23404: make touch becomes make regen-all (#1405) (#1461)
https://github.com/python/cpython/commit/9d02f562961efd12d3c8317a10916db7f77330cc
msg293030 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-05-05 00:20
New changeset ab6b962ef241be97536573d7490ce1cfc74fde18 by Victor Stinner in branch '3.5':
bpo-23404: make touch becomes make regen-all (#1405) (#1461) (#1465)
https://github.com/python/cpython/commit/ab6b962ef241be97536573d7490ce1cfc74fde18
msg293033 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-05-05 01:14
New changeset 8a19eb24c97ef43e9fc7d45af180334ac8093545 by Victor Stinner in branch '2.7':
bpo-23404: make touch becomes make regen-all (#1466)
https://github.com/python/cpython/commit/8a19eb24c97ef43e9fc7d45af180334ac8093545
msg293034 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-05-05 01:21
I modified Python 2.7, 3.5, 3.6 and master (3.7). I prefer to keep the issue open at least one week in a regression is found, like issue #30273 (this one should already be fixed).

Maybe the devguide should be updated too?
msg293094 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-05-05 10:29
As noted on python-dev, it would be helpful if this change was mentioned in the 2.7, 3.5, and 3.6 What's New documents under a "Notable changes in maintenance releases" section.

For 2.7, that would be a retitling of the existing section "New features added to Python 2.7 maintenance releases": https://docs.python.org/dev/whatsnew/2.7.html#new-features-added-to-python-2-7-maintenance-releases

For 3.5 and 3.6, this would be a new entry at the end of the document, with "Change to handling of pre-generated files" as the sole subsection.

(As another example of an existing section along these lines, see https://docs.python.org/dev/whatsnew/3.4.html#changed-in-3-4-3)
History
Date User Action Args
2017-05-05 10:29:41ncoghlansetmessages: + msg293094
2017-05-05 01:21:50hayposetmessages: + msg293034
2017-05-05 01:14:25hayposetmessages: + msg293033
2017-05-05 00:51:17hayposetpull_requests: + pull_request1565
2017-05-05 00:20:02hayposetmessages: + msg293030
2017-05-04 23:56:57hayposetpull_requests: + pull_request1563
2017-05-04 22:46:58hayposetmessages: + msg293026
2017-05-04 21:29:11hayposetmessages: + msg293018
2017-05-04 21:09:00hayposetpull_requests: + pull_request1562
2017-05-04 17:14:24hayposetmessages: + msg293002
2017-05-04 17:12:56hayposetpull_requests: + pull_request1557
2017-05-03 16:27:19hayposetmessages: + msg292916
2017-05-03 16:21:50hayposetmessages: + msg292913
2017-05-03 08:01:48hayposetmessages: + msg292860
2017-05-03 08:01:22hayposetmessages: + msg292858
2017-05-03 07:42:14ncoghlansetmessages: + msg292855
2017-05-03 00:57:00hayposetmessages: + msg292829
2017-05-02 23:45:21hayposetmessages: + msg292823
2017-05-02 23:44:57haypolinkissue29967 superseder
2017-05-02 23:44:14hayposetmessages: + msg292822
2017-05-02 23:37:37hayposetpull_requests: + pull_request1513
2017-04-24 08:43:17martin.pantersetmessages: + msg292207
2017-04-24 06:01:43ncoghlansetmessages: + msg292194
2017-04-23 16:06:43brett.cannonsetmessages: + msg292170
2017-04-23 01:02:22martin.pantersetmessages: + msg292143
2017-04-22 16:50:19brett.cannonsetnosy: + ncoghlan
messages: + msg292126
2017-04-22 02:03:29koobssetnosy: + koobs
2017-04-22 00:17:36martin.pantersetmessages: + msg292092
2017-04-21 17:00:53hayposetmessages: + msg292068
2017-04-21 16:57:30hayposetnosy: + haypo
messages: + msg292067
2017-04-03 13:06:55martin.pantersetfiles: + boot-flag.py2.patch
2017-04-03 13:06:45martin.pantersetfiles: + boot-flag.py3.5.patch

messages: + msg291069
2017-02-11 00:38:13zach.waresetpriority: normal -> critical
versions: + Python 2.7, Python 3.7
nosy: + zach.ware

messages: + msg287580

keywords: + buildbot
2017-02-10 09:25:37Chi Hsuan Yensetfiles: + boot-flag.patch
nosy: + Chi Hsuan Yen
messages: + msg287493

2016-09-13 11:29:24martin.pantersetfiles: + boot-flag.patch

messages: + msg276252
2016-09-09 08:09:10martin.pantersetnosy: + martin.panter
messages: + msg275285
2016-01-02 00:17:29ezio.melottisetnosy: + brett.cannon
2015-07-29 14:07:11berker.peksagsettype: enhancement
components: - Devguide
versions: + Python 3.6
nosy: + berker.peksag

messages: + msg247575
stage: patch review -> needs patch
2015-07-29 13:58:19python-devsetnosy: + python-dev
messages: + msg247574
2015-07-28 09:17:26willingcsetfiles: + iss23404devguide.patch

nosy: + willingc
messages: + msg247510

stage: needs patch -> patch review
2015-02-07 07:08:18vleesetmessages: + msg235516
2015-02-07 06:52:52vleesetmessages: + msg235515
2015-02-07 06:34:43ned.deilysettype: compile error -> (no value)
components: + Devguide
title: Python 3.5 does not build with Python 2.6. -> 'make touch' does not work with git clones of the source repository
nosy: + ezio.melotti

messages: + msg235514
resolution: not a bug ->
stage: resolved -> needs patch
2015-02-07 06:17:20vleesetstatus: closed -> open
files: + 0001-Issue-23404-Do-not-use-Python-2.7-constructs-in-Pars.patch
messages: + msg235513

keywords: + patch
2015-02-07 03:25:15ned.deilysetstatus: open -> closed
resolution: not a bug
stage: resolved
2015-02-07 03:23:14ned.deilysetmessages: + msg235511
2015-02-07 03:21:27ned.deilysetnosy: + ned.deily
messages: + msg235510
2015-02-07 02:51:31josh.rsetmessages: + msg235509
2015-02-07 02:44:56josh.rsetnosy: + josh.r
messages: + msg235508
2015-02-06 22:46:05vleecreate