Background: we'd like to enable running 'make touch' on the bots before 'make -jN', to avoid problems with scripts that auto-generate code for the Python build (bootstrapping). pydev thread: https://mail.python.org/pipermail/python-dev/2013-September/128992.html

A couple of issues are prerequisites:

1. Fix the asdl dependencies in .hgtouch:

Include/ast.h: Parser/Python.asdl Parser/asdl.py Parser/asdl_c.py
Python/Python-ast.c: Include/ast.h

The file Include/ast.h is not, in fact, auto-generated. But Include/Python-ast.h *is*, and it does not appear in this file.

2. Make sure 'hg touch' itself runs on Python 2.4 because that's what some bots have (and other older systems like RHEL 5)

.hgtouch fixed in ac19ff225280 (I specified the issue number incorrectly so this one wasn't notified).

Curiously, make touch seems to think there's still things to do even after the first round of "touching":

$ touch Parser/asdl_c.py 
$ make touch
hg --config extensions.touch=Tools/hg/hgtouch.py touch -v
Touching Include/Python-ast.h
Touching Python/Python-ast.c
$ make touch
hg --config extensions.touch=Tools/hg/hgtouch.py touch -v
Touching Python/Python-ast.c

Ooh, I think that's because the "# try processing all rules in topological order" in do_touch doesn't actually topo-sort.

The problem in this case is different, actually. It's the comparison:

        if o_time <= i_time:
            # generated file is older, touch
            need_touch = True

In check_rule. The script is pretty quick so when it touches both Python-ast.h and .c they get the same stat time exactly. In the next run, then, this comparison succeeds and .c is touched again.

I'm not sure what's the right way to go about this? Changing the comparison to < may theoretically miss cases in which the input was updated an epsilon after it auto-generated its output, and the change will go unnoticed. A different solution would be to introduce a micro-wait between each 'touch' to make sure that transitive dependencies don't need to be revisited in the future.

I see two solutions, both involving to track the newest timestamp that needs to pass.
a) touch the file to have a time stamp in the future. make might complain, though.
b) sleep until a second has passed, then touch (actually, sleep of 100ms multiple times until the time stamp is new enough)

The key issue is that hg touch should use the same comparison as make. So if make thinks that something need to be done, hg touch should touch the files.

Here is a patch that backdates outputs 1s after their youngest input (and sleeps if necessary to avoid producing files in the future).

With that, I get

lap-le:3k loewis$ touch Parser/asdl_c.py ;date;make touch;date
Mo 30 Sep 2013 15:29:21 CEST
hg --config extensions.touch=Tools/hg/hgtouch.py touch -v
Touching Include/Python-ast.h
Touching Python/Python-ast.c
Mo 30 Sep 2013 15:29:23 CEST
lap-le:3k loewis$ ls -lT Parser/asdl_c.py Include/Python-ast.h Python/Python-ast.c
-rw-r--r-- 1 loewis admin 19489 30 Sep 15:29:22 2013 Include/Python-ast.h
-rwxr-xr-x 1 loewis admin 43732 30 Sep 15:29:21 2013 Parser/asdl_c.py
-rw-r--r-- 1 loewis admin 228785 30 Sep 15:29:23 2013 Python/Python-ast.c

BTW, I think the Python-ast.c issue could be solved by just having Python-ast.c depend on the same inputs as Python-ast.h; the C file isn't actually depending on the H file.

LGTM

New changeset 477246839224 by Martin v. Löwis in branch '3.3':
Issue bpo-19106: Touch generated files to be 1s newer than their youngest source.
http://hg.python.org/cpython/rev/477246839224

New changeset 86eff5c4e698 by Martin v. Löwis in branch '3.3':
Issue bpo-19106: Add buildbottouch target.
http://hg.python.org/cpython/rev/86eff5c4e698

I have now updated the master.cfg to make buildbottouch a separate build step.

With your fix, make touch now behaves as expected. Also, I can see the step added to the bots (e.g. http://buildbot.python.org/all/builders/AMD64%20Ubuntu%20LTS%203.x/builds/2597/steps/shell/logs/stdio)

Thanks, I think the issue is resolved then. We should now be able to have modern Python code in scripts that generate code such as asdl[_c].py because bots and people will only old versions of Python don't ever need to run them.

Should we add a note about this to the dev guide?

"make touch" (or "hg touch") certainly should be mentioned. Details of the buildbot configuration need not.

Publishing the buildbot configuration file might be useful, except that it also contains all the builder passwords, which should not be published. If that is done, it should be the "life" configuration file (rather than a static copy created now). It might be easiest to publish it through a symlink on buildbot.python.org.

New changeset 56ed149e597a by Eli Bendersky in branch 'default':
Mention 'make touch' in the devguide.
http://hg.python.org/devguide/rev/56ed149e597a

I've updated the devguide about make touch.

As for the buildbot configuration, I agree we shouldn't document a static snapshot and also there's the passwords problem.

I was talking to Antoine about this the other day, and maybe there's sense to create an "infrastructure" section in the devguide that will explain how to log into the buildbot server and look at the configuration. This should be enough for folks who really need to figure it out and is also safe w.r.t. getting out of date and exposing passwords.

Also, IMHO bpo-15964 can be closed.

I was talking to Antoine about this the other day, and maybe there's
sense to create an "infrastructure" section in the devguide that
will explain how to log into the buildbot server and look at the
configuration.

IMHO it would make more sense to create a separate "infrastructure
guide". There's no need to bloat the devguide with such information,
which could quickly become quite complicated.

I'm not sure a separate document is right here because that's one more repository to have. The devguide already contains sections for somewhat more esoteric things like compiler internals guide, coverity scans. Besides, section 18 already has some material about the buildbots.

But I don't feel strongly enough about this to argue :)

I'm not sure a separate document is right here because that's one
more repository to have. The devguide already contains sections for
somewhat more esoteric things like compiler internals guide,
coverity scans. Besides, section 18 already has some material about
the buildbots.

Well, ideally the more esoteric things wouldn't really be there.
Perhaps other people have different opinions?

On Tue, Oct 1, 2013 at 7:00 AM, Antoine Pitrou <report@bugs.python.org>wrote:

Antoine Pitrou added the comment:

> I'm not sure a separate document is right here because that's one
> more repository to have. The devguide already contains sections for
> somewhat more esoteric things like compiler internals guide,
> coverity scans. Besides, section 18 already has some material about
> the buildbots.

Well, ideally the more esoteric things wouldn't really be there.

I still see the current situation better than the previous one, where this
was scattered around PEPs, Wikis and just random web pages. Having it all
in the same repository is not ideal from a taxonomical point of view,
perhaps, but as a single collection of "everything related to the
development *of* Python" it's not too bad since keeping it in a single
place makes it easier to maintain and review; mostly, there's a somewhat
smaller chance that things will be forgotten and rot.

To put it differently, maintaining the devguide is kind-of a chore for devs
as it is; splitting it into multiple topical repositories will just make
the burden higher so we'll end up not maintaining it at all :)

Perhaps other people have different opinions?

+nick,guido,benjamin: in case you're interested in the discussion that takes place in the most recent messages of this issue

Documenting the pydotorg setup is a field of ongoing discussions. If you ask three people involved, you get three answers where this should be documented, with no chance for consensus.

From an infrastructure point of view, it is up to the infrastructure head (i.e. Noah Kantrowitz) to rule on where this should be documented. I can see the python-dev view, to, although the documentation might be as short as "ask Noah".

Where to document the infrastructure setup (including the buildbots) is up to Noah/infrastructure-sig, as the devguide just covers CPython development, while the PSF infrastructure team covers a lot more than that.

A pointer from the devguide docs to the appropriate infrastructure docs would make sense, but it isn't the right home for the docs themselves.

The interim solution I use is "ask on python-dev" or "ask on python-committers" :)

I still see the current situation better than the previous one, where
this
was scattered around PEPs, Wikis and just random web pages. Having it
all
in the same repository is not ideal from a taxonomical point of view,
perhaps, but as a single collection of "everything related to the
development *of* Python" it's not too bad since keeping it in a
single
place makes it easier to maintain and review;

Well, precisely, it's putting everything in a single collection that
I find bad, because it could become dreadful for beginner contributors.

The devguide is primarily meant to ensure contributors don't get lost,
even though it can serve as a reminder for core developers too.

I think this issue can be closed, since Martin's touch step runs on the bots successfully, and the ASDL dependencies in .hgtouch were fixed.