Here's a patch to add a page to the devguide on running a build slave.  I copied pretty liberally from the wiki (http://wiki.python.org/moin/BuildBot).  That page may or may not be up to date (I don't know), so the bulk of this new page may also be out of date...

The current devguide buildbots page is oriented toward the use of buildbots in the dev process.  In contrast, this new page is all about running one.  I figured it is yet another easy way that people can contribute to Python.

I didn't stick in the windows-specific buildbot info (http://wiki.python.org/moin/BuildbotOnWindows), but did link to the wiki page.

I did include the info, as far as I understood it, from a recent python-dev thread (http://mail.python.org/pipermail/python-dev/2011-October/113935.html).

I added a link to the devguide index page, but it may need to be shifted to the right spot (maybe right after "Running & Writing Tests").  I also renamed buildbots.rst to continuous_integration.rst (and updated reference accordingly).

I want to be sure attribution is appropriate, regarding the text pulled from the wiki page and from the python-dev thread.  Is linking to the thread sufficient?

This is a rough stab (especially with the inclusion of the still-ongoing python-dev thread); but hopefully this gets a ball rolling and doesn't take much work to polish.  :)

A few comments:
  * are you following some convention about the spelling of buildbot/Buildbot/BuildBot?
  * "+3. the standard development toolchain."
    Which is? I think a compiler is the only missing bit.
  * "(in which case you'll have to install Python and Twisted first)."
    I would say "you'll need" and not assume that they are not installed.
  * "a couple of third-party libraries (such as zlib and OpenSSL)."
    Actually these are not required, and it might even be good to have some buildbot without e.g. zlib.
  * "+   % su - buildbot"
    I think we use $ elsewhere in the doc.  You can also set the highlightlang to sh (iirc).
  * "/usr/local/bin", these should get some markup: either ``..`` or :file:`...` (is :file: ok for dirs too?).
  * "You'll need to get someone to create the slavename/slavepasswd on dinsdale.python.org before doing this."
    So maybe put it before?
  * "monitor the build results and solve any setup issues causing test failures"
    If running a buildbot implies some responsibilities, maybe they should be mentioned earlier.
  * "executing arbitrarily horrible and/or malicious code"
    This sounds bad and misleading.  At worst the test can use resources extensively and/or cause crashes, but that doesn't mean that the code is arbitrary, horrible, or malicious.
  * "for cutting of the builder"
    s/of/off/
  * "Therefore builders should be set up in both configurations"
    This might lead build masters to think they have to set up both.
  * "Some tests, as the start of `this thread "
    You already have a link to this at the end of the section
  * "+Running a build slave requires some network access."
    Isn't 9020 the only *required* one?  If the others are not available, test will (should) be skipped.

Great feedback!  Keep in mind that nearly all the content in my patch is pulled unedited from either the wiki page and the python-dev thread I mentioned.

I'll work on cleaning it up when I get a chance (probably after Wednesday's PyCon CFP closes :).  If anyone wants to do it before I get a chance, they are welcome to it.  Either way I think such a page will be helpful to have in the devguide.

I think the whole security paragraph should be deleted. The
discussion on python-dev has overstated the risks.

Anyone who is *that* security conscious and in effect does
not trust the Python committers should also audit the thousands
of lines of ./configure and any code that is executed by
`make install`, to name a few "security risks".


What I would like to see in the devguide is a cookbook shell
script with minimal comments that interested persons can
simply run.


The easiest way for me (I'm running two bots) was to install
Python-2.7 from source, then pip, then `pip install buildslave`.
(Yes, as root :). That automatically gets everything right and
you don't depend on stale packages from the distro.

Per Ezio's and Stefan's feedback I cleaned things up a little:

* consolidated on "Buildbot"
* clarified the meaning of the "standard development toolchain"
* % changed to $
* cleaned up the network ports section
* removed a bunch of outdated/unnecessary info

I did not do anything about syntax highlighting for lack of know-how.  I also did not add anything about the responsibilities of running a build slave, mostly because I have no idea what those are.  :)

Much of the section on installation needs double-checking as I am not sure how much of it is necessary and how much not.

This is about as far as I think I will be able to take this tracker issue.  Unfortunately I don't have a lot of time to get up to speed on how Buildbot works, and I know nearly nothing about it currently.

I'm hoping that finishing this won't be much work for someone with a working knowledge of our buildbots.  I've tried to get it close.  It's a gap in the devguide that I think would be worth filling.

Incidently, I pulled the section on security (which motivated me here in the first place) and stuck it into the wiki page.  If any of that is worth pulling back in, there it is.  Thanks!

Here's a terse shell script that IMO even moderately experienced admins
will prefer to the current version.

I'm not sure if the devguide is the right place for this, since
non-devs are very welcome to set up buildbots.

> Here's a terse shell script that IMO even moderately experienced admins
> will prefer to the current version.

I'm sure some admins will prefer using their system's packages (I think
buildbot is packaged for Debian/Ubuntu, I see it in Mageia's packages,
not sure about Fedora).

Anyway, the current instructions are on the wiki:
http://wiki.python.org/moin/BuildBot
You could add your script or link to it there.

Antoine Pitrou <report@bugs.python.org> wrote:
> I'm sure some admins will prefer using their system's packages (I think
> buildbot is packaged for Debian/Ubuntu, I see it in Mageia's packages,
> not sure about Fedora).

Yes, certainly. I had a bad experience using the packaged buildbot-slave,
but I don't remember which distro it was.

Generally speaking, I've had so much trouble with rarely used packages
on multiple distros that I *always* compile or install that kind of
software directly from upstream.

> Anyway, the current instructions are on the wiki:
> http://wiki.python.org/moin/BuildBot

That's probably the best place.

Eric: I'm afraid the excerpts from the mailing list discussion you put
there are a bit confusing and don't reflect any kind of majority opinion.

For example I'm running two bots and I don't even know what this means:

"Some tests, as the start of this thread [...] indicates, must have some
 special logic to make sure they do or do not run, or run differently, in
 privileged vs. unprivileged configurations, but generally speaking most
 things should work in both places."

That does not concern a buildbot owner, since he has no influence on that.

Also, you included that nasty quote "executing arbitrarily horrible and/or
malicious code". I re-read the thread, and that quote is a half-sarcastic
concession from Glyph (who actually had a much lighter view on the whole
situation initially!) to the concerns of the people who were almost accusing
buildbot owners of being clueless.

The other side of the discussion (Martin's view) is missing completely,
despite the fact that most core developers *probably* share that view.

I'm sure you meant well, but I think we should give buildbot admins some
credit. Experienced admins don't need any advice, inexperienced admins
would be served better by a cookbook approach.

The script I posted is a start: All buildbot software is in /home/buildbot
and runs under the password-less user buildbot. In combination with a VM,
this is enough for me. If anyone thinks this is not secure enough and wants
to add chroot, jails, whatever, the best way forward is to post an improved
version.

No problem, Stefan.  :)  The main motivation was to capture the discussion at the time so that something actually came of it.

Adding info to the devguide, essentially the catalog/how-to of core dev activities, was meant to simply provide another clear avenue for those that wanted to help with Python's core development.  If not a dedicated page, we should at least have a link in the "Contributing" section to the (single, authoritative) resource we have on setting-up/running a build slave, regardless of where that resource resides.

That said, I gladly defer to those that are more involved with the buildbots.  Incidentally, the second patch is a bit cleaner than the first.

Eric Snow <report@bugs.python.org> wrote:
> Incidentally, the second patch is a bit cleaner than the first.

Yes, indeed it is. :) To prevent a misunderstanding: In my last mail I was
talking about the "Build Slaves and Security" additions to:

http://wiki.python.org/moin/BuildBot

Did you want to commit this patch yourself, Eric, since you have commit privs now?

Yeah, at some point I will. :)  I've been pretty focused on implementing a C-OrderedDict, which I'm (hopefully) wrapping up pretty soon.  Once that happens I'm going to revisit a bunch of the open tickets I have.  If you want to push this sooner than that, feel free. :)

Looking this over, it seems like there were outstanding objections to adding this to the devguide and to the content.  I still think the devguide is the right place and that something like my second patch would be appropriate.

Before I take any time to update the patch, does anyone object to the location or intent of the changes?

Also, I'm fine with adapting/incorporating or linking to Stefan's script.  The whole point is to make it as easy as possible to spin up and run a build slave. :)

Eric Snow <report@bugs.python.org> wrote:
> Looking this over, it seems like there were outstanding objections to adding this to the devguide and to the content.

I have no issues with the content of the second patch. However, snakebite
has changed the situation a little: We don't need more buildbots, we need
experts in AIX and HP-UX.

For DragonFlyBSD, NetBSD and OpenBSD we'd ideally need someone who reports the
OS bugs (there are a couple) upstream so that the bots finally turn green and
become useful.

So on fact the first paragraph of the patch is now a little outdated.

Given the current situation I think it would be more useful to add

http://mail.python.org/pipermail/python-dev/2012-September/121651.html

to the devguide.

> Before I take any time to update the patch, does anyone object
> to the location or intent of the changes?

Adding a new page to the devguide seems OK to me.  It makes the devguide bigger, but it can easily be ignored by developers/contributors if they are not interested.

Closing this issue since a newer issue #24440 addresses devguide documentation of installing and running a buildbot. Thanks to R. David Murray for submitting the patch to #24440.

Hmm.  You'd think I'd remember to search the bug tracker before opening a new issue :(.

I'll review this and make sure I've incorporated anything relevant from here into my patch when I've updated it.

I think the security information is useful, but based on Stefan's feedback I'll fix the language to be informative rather than cautionary.  I'll be updating the patch in the linked issue in the near future.