classification
Title: Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots
Type: behavior Stage: resolved
Components: Build Versions: Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: eli.bendersky Nosy List: benjamin.peterson, eli.bendersky, ezio.melotti, gvanrossum, loewis, ncoghlan, pitrou, python-dev
Priority: normal Keywords: patch

Created on 2013-09-27 15:25 by eli.bendersky, last changed 2013-10-08 13:07 by eli.bendersky. This issue is now closed.

Files
File name Uploaded Description Edit
touch.diff loewis, 2013-09-30 13:33 review
Messages (23)
msg198482 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-09-27 15:25
Background: we'd like to enable running 'make touch' on the bots before 'make -jN', to avoid problems with scripts that auto-generate code for the Python build (bootstrapping). pydev thread: https://mail.python.org/pipermail/python-dev/2013-September/128992.html

A couple of issues are prerequisites:

1. Fix the asdl dependencies in .hgtouch:

Include/ast.h: Parser/Python.asdl Parser/asdl.py Parser/asdl_c.py
Python/Python-ast.c: Include/ast.h

The file Include/ast.h is not, in fact, auto-generated. But Include/Python-ast.h *is*, and it does not appear in this file.

2. Make sure 'hg touch' itself runs on Python 2.4 because that's what some bots have (and other older systems like RHEL 5)
msg198485 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-09-27 15:52
.hgtouch fixed in ac19ff225280 (I specified the issue number incorrectly so this one wasn't notified).

Curiously, `make touch` seems to think there's still things to do even after the first round of "touching":

$ touch Parser/asdl_c.py 
$ make touch
hg --config extensions.touch=Tools/hg/hgtouch.py touch -v
Touching Include/Python-ast.h
Touching Python/Python-ast.c
$ make touch
hg --config extensions.touch=Tools/hg/hgtouch.py touch -v
Touching Python/Python-ast.c
msg198489 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-09-27 16:20
Ooh, I think that's because the "# try processing all rules in topological order" in do_touch doesn't actually topo-sort.
msg198490 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-09-27 17:00
The problem in this case is different, actually. It's the comparison:

        if o_time <= i_time:
            # generated file is older, touch
            need_touch = True

In check_rule. The script is pretty quick so when it touches both Python-ast.h and .c they get the same stat time exactly. In the next run, then, this comparison succeeds and .c is touched again.

I'm not sure what's the right way to go about this? Changing the comparison to < may theoretically miss cases in which the input was updated an epsilon after it auto-generated its output, and the change will go unnoticed. A different solution would be to introduce a micro-wait between each 'touch' to make sure that transitive dependencies don't need to be revisited in the future.
msg198684 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2013-09-30 08:34
I see two solutions, both involving to track the newest timestamp that needs to pass.
a) touch the file to have a time stamp in the future. make might complain, though.
b) sleep until a second has passed, then touch (actually, sleep of 100ms multiple times until the time stamp is new enough)

The key issue is that hg touch should use the same comparison as make. So if make thinks that something need to be done, hg touch should touch the files.
msg198695 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2013-09-30 13:33
Here is a patch that backdates outputs 1s after their youngest input (and sleeps if necessary to avoid producing files in the future).

With that, I get

lap-le:3k loewis$ touch Parser/asdl_c.py ;date;make touch;date
Mo 30 Sep 2013 15:29:21 CEST
hg --config extensions.touch=Tools/hg/hgtouch.py touch -v
Touching Include/Python-ast.h
Touching Python/Python-ast.c
Mo 30 Sep 2013 15:29:23 CEST
lap-le:3k loewis$ ls -lT Parser/asdl_c.py Include/Python-ast.h Python/Python-ast.c
-rw-r--r--  1 loewis  admin   19489 30 Sep 15:29:22 2013 Include/Python-ast.h
-rwxr-xr-x  1 loewis  admin   43732 30 Sep 15:29:21 2013 Parser/asdl_c.py
-rw-r--r--  1 loewis  admin  228785 30 Sep 15:29:23 2013 Python/Python-ast.c

BTW, I think the Python-ast.c issue could be solved by just having Python-ast.c depend on the same inputs as Python-ast.h; the C file isn't actually depending on the H file.
msg198696 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-09-30 13:45
LGTM
msg198697 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-09-30 14:10
New changeset 477246839224 by Martin v. Löwis in branch '3.3':
Issue #19106: Touch generated files to be 1s newer than their youngest source.
http://hg.python.org/cpython/rev/477246839224
msg198699 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-09-30 14:22
New changeset 86eff5c4e698 by Martin v. Löwis in branch '3.3':
Issue #19106: Add buildbottouch target.
http://hg.python.org/cpython/rev/86eff5c4e698
msg198703 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2013-09-30 14:49
I have now updated the master.cfg to make buildbottouch a separate build step.
msg198707 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-09-30 15:23
With your fix, `make touch` now behaves as expected. Also, I can see the step added to the bots (e.g. http://buildbot.python.org/all/builders/AMD64%20Ubuntu%20LTS%203.x/builds/2597/steps/shell/logs/stdio)

Thanks, I think the issue is resolved then. We should now be able to have modern Python code in scripts that generate code such as asdl[_c].py because bots and people will only old versions of Python don't ever need to run them.

Should we add a note about this to the dev guide?
msg198709 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2013-09-30 15:44
"make touch" (or "hg touch") certainly should be mentioned. Details of the buildbot configuration need not.

Publishing the buildbot configuration file might be useful, except that it also contains all the builder passwords, which should not be published. If that is done, it should be the "life" configuration file (rather than a static copy created now). It might be easiest to publish it through a symlink on buildbot.python.org.
msg198771 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-10-01 13:13
New changeset 56ed149e597a by Eli Bendersky in branch 'default':
Mention 'make touch' in the devguide.
http://hg.python.org/devguide/rev/56ed149e597a
msg198774 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-10-01 13:30
I've updated the devguide about `make touch`.

As for the buildbot configuration, I agree we shouldn't document a static snapshot and also there's the passwords problem.

I was talking to Antoine about this the other day, and maybe there's sense to create an "infrastructure" section in the devguide that will explain how to log into the buildbot server and look at the configuration. This should be enough for folks who really need to figure it out and is also safe w.r.t. getting out of date and exposing passwords.

Also, IMHO #15964 can be closed.
msg198775 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-10-01 13:38
> I was talking to Antoine about this the other day, and maybe there's
> sense to create an "infrastructure" section in the devguide that
> will explain how to log into the buildbot server and look at the
> configuration.

IMHO it would make more sense to create a separate "infrastructure
guide". There's no need to bloat the devguide with such information,
which could quickly become quite complicated.
msg198776 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-10-01 13:44
I'm not sure a separate document is right here because that's one more repository to have. The devguide already contains sections for somewhat more esoteric things like compiler internals guide, coverity scans. Besides, section 18 already has some material about the buildbots.

But I don't feel strongly enough about this to argue :)
msg198779 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-10-01 14:00
> I'm not sure a separate document is right here because that's one
> more repository to have. The devguide already contains sections for
> somewhat more esoteric things like compiler internals guide,
> coverity scans. Besides, section 18 already has some material about
> the buildbots.

Well, ideally the more esoteric things wouldn't really be there.
Perhaps other people have different opinions?
msg198782 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-10-01 14:09
On Tue, Oct 1, 2013 at 7:00 AM, Antoine Pitrou <report@bugs.python.org>wrote:

>
> Antoine Pitrou added the comment:
>
> > I'm not sure a separate document is right here because that's one
> > more repository to have. The devguide already contains sections for
> > somewhat more esoteric things like compiler internals guide,
> > coverity scans. Besides, section 18 already has some material about
> > the buildbots.
>
> Well, ideally the more esoteric things wouldn't really be there.
>

I still see the current situation better than the previous one, where this
was scattered around PEPs, Wikis and just random web pages. Having it all
in the same repository is not ideal from a taxonomical point of view,
perhaps, but as a single collection of "everything related to the
development *of* Python" it's not too bad since keeping it in a single
place makes it easier to maintain and review; mostly, there's a somewhat
smaller chance that things will be forgotten and rot.

To put it differently, maintaining the devguide is kind-of a chore for devs
as it is; splitting it into multiple topical repositories will just make
the burden higher so we'll end up not maintaining it at all :)

> Perhaps other people have different opinions?
>
msg198783 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-10-01 14:10
+nick,guido,benjamin: in case you're interested in the discussion that takes place in the most recent messages of this issue
msg198785 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2013-10-01 14:17
Documenting the pydotorg setup is a field of ongoing discussions. If you ask three people involved, you get three answers where this should be documented, with no chance for consensus.

From an infrastructure point of view, it is up to the infrastructure head (i.e. Noah Kantrowitz) to rule on where this should be documented. I can see the python-dev view, to, although the documentation might be as short as "ask Noah".
msg198786 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-10-01 14:24
Where to document the infrastructure setup (including the buildbots) is up to Noah/infrastructure-sig, as the devguide just covers CPython development, while the PSF infrastructure team covers a lot more than that.

A pointer from the devguide docs to the appropriate infrastructure docs would make sense, but it isn't the right home for the docs themselves.

The interim solution I use is "ask on python-dev" or "ask on python-committers" :)
msg198787 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-10-01 14:41
> I still see the current situation better than the previous one, where
> this
> was scattered around PEPs, Wikis and just random web pages. Having it
> all
> in the same repository is not ideal from a taxonomical point of view,
> perhaps, but as a single collection of "everything related to the
> development *of* Python" it's not too bad since keeping it in a
> single
> place makes it easier to maintain and review;

Well, precisely, it's putting everything in a single collection that
I find bad, because it could become dreadful for beginner contributors.

The devguide is primarily meant to ensure contributors don't get lost,
even though it can serve as a reminder for core developers too.
msg199200 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-10-08 13:07
I think this issue can be closed, since Martin's touch step runs on the bots successfully, and the ASDL dependencies in .hgtouch were fixed.
History
Date User Action Args
2013-10-08 13:07:43eli.benderskysetstatus: open -> closed
resolution: fixed
messages: + msg199200

stage: resolved
2013-10-01 14:42:00pitrousetmessages: + msg198787
title: Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots -> Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots
2013-10-01 14:25:00ncoghlansetmessages: + msg198786
2013-10-01 14:17:03loewissetmessages: + msg198785
2013-10-01 14:10:45eli.benderskysetnosy: + gvanrossum, ncoghlan, benjamin.peterson
messages: + msg198783
2013-10-01 14:09:30eli.benderskysetmessages: + msg198782
title: Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots -> Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots
2013-10-01 14:00:19pitrousetmessages: + msg198779
2013-10-01 13:44:52eli.benderskysetmessages: + msg198776
2013-10-01 13:38:28pitrousetmessages: + msg198775
title: Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots -> Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots
2013-10-01 13:30:04eli.benderskysetmessages: + msg198774
2013-10-01 13:13:55python-devsetmessages: + msg198771
2013-09-30 15:44:53loewissetmessages: + msg198709
2013-09-30 15:23:47eli.benderskysetmessages: + msg198707
2013-09-30 14:49:38loewissetmessages: + msg198703
2013-09-30 14:22:21python-devsetmessages: + msg198699
2013-09-30 14:10:21python-devsetnosy: + python-dev
messages: + msg198697
2013-09-30 13:45:19eli.benderskysetmessages: + msg198696
2013-09-30 13:33:37loewissetfiles: + touch.diff
keywords: + patch
messages: + msg198695
2013-09-30 08:34:56loewissetmessages: + msg198684
2013-09-27 17:00:26eli.benderskysetmessages: + msg198490
2013-09-27 16:20:33eli.benderskysetmessages: + msg198489
2013-09-27 15:52:16eli.benderskysetmessages: + msg198485
2013-09-27 15:25:57eli.benderskycreate