diff -r 467285511629 buildslave.rst --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/buildslave.rst Fri Jun 12 12:24:36 2015 -0400 @@ -0,0 +1,200 @@ + +.. _buildslave: + +Running a buildslave +==================== + +Python's :ref:`buildbots` system was discussed earlier. We sometimes refer to +the collection of *build slaves* as our "buildbot fleet". The machines that +comprise the fleet are voluntarily contributed resources. Many are run by +individual volunteers out of their own pockets and time, while others are +supported by corporations. Even the corporate sponsored buildbots, however, +tend to exist because some individual championed them, made them a reality, and +is committed to maintaining them. + +Anyone can contribute a buildbot to the fleet. This chapter describes how +to go about setting up a buildslave, getting it added, and some hints about +buildbot maintenance. + +Anyone running a buildbot that is part of the fleet should subscribe to the +`python-buildbots `_ +mailing list. This mailing list is also the place to contact if you want to +contribute a buildbot but have questions. + + +Installing a buildslave +----------------------- + +You need a recent version of the `buildbot `_ software. +If your OS has a packaging system (for example, most linux distributions +do), it is probably available from the standard repositories. Otherwise, +make sure you have Python installed, and use ``pip install buildbot``. + +A buildslave must also have a `mercurial `_ +client installed. A windows buildslave also required a `subversion +`_ client, since the Windows build includes +some "vedored" versions of third party packages whose source is stored in a +legacy subversion repository. + +Since the goal is to build Python from source, the system will also need to +have the standard development tools for the platform installed: a compiler, a +linker, and the "development" headers for any of the optional modules (zlib, +OpenSSL, etc) supported by the platform. (This last does not apply to Windows, +since on that platform the supported optional modules are obtained from the svn +repository.) + +.. XXX: need a complete list of optional modules and headers. Pull from + dev setup guide. + +In order to set up the buildbot software, you will need to obtain an identifier +and password for your buildslave so it can join the fleet. Email +python-builsbots@python.org to discuss adding your buildslave and to obtain the +needed slavename and password. + +For Linux and the Mac: + +If your package manager has not already created a ``buildbot`` user as +part of the buildbot install, you should create one. Then:: + + su - buildbot + mkdir buildarea + buildslave create-slave buildarea buildbot.python.org:9020 slavename slavepasswd + +For Windows: + +TBD + +Once the initial slave setup is done, you should edit the files +``buildarea/info/admin`` and ``buildarea/info/host`` to provide your contact +info and information on the host configuration, respectively. This information +will be presented in the buildbot web pages that display information about the +builders running on your buildslave. + +You will also want to make sure that the buildslave is started when the +machine reboots. On most linux systems you should add something like +the following to ``/etc/crontab``:: + + @reboot buildslave restart /home/buildbot/buildslave + +Note that we use ``restart`` rather than ``start`` in case a crash has left a +``twistd.pid`` file behind. + +(Mac/Windows TBD) + +To start the buildslave running for your initial testing, you can do:: + + buildslave start ~/buildarea + +Then you can either wait for someone to make a commit, or you can pick a +builder from the `list of builders +`_ associated with your buildslave +and force a build. + +In any case you should monitor builds on your builders to make sure the tests +are passing and to resolve any platform issues that may be revealed by tests +that fail. Unfortunately we do not currently have a way to notify you only of +failures on your builders. + + +Builslave operation +------------------- + +Most of the time, running a buildslave is a "set and forget" operation, +depending on the level of involvement you want to have in resolving bugs +revealed by your builders. There are, however, times when it is helpful or +even necessary for you to get involved. As noted above, you should be +subscribed to ``python-buildbots@python.org`` so that you will be made +aware of any fleet-wide issues. + +Necessary tasks include, obviously, keeping the buildbot running. Currently +the system for notifying buildbot owners when their slaves go offline is not +working; this is something we hope to resolve. So currently it is helpful if +you periodically check the status of your buildslave. We will also contact you +via your contact address in ``buildarea/info/admin`` when we notice there is a +problem that has not been resolved for some period of time. + +We currently do not have a minimum version requirement for the buildslave +software. However, this is something we will probably establish as we tune the +fleet, so another task will be to occasionally upgrade the buildslave software. +Coordination for this will be done via ``python-buildbots@python.org``. + +The most interesting extra involvement is when your buildslave reveals a unique +or almost-unique problem: a test that is failing on your system but not on +other systems. In this case you should be prepared to offer debugging help to +the people working on the bug: running tests by hand on the buildslave machine +or, if possible, providing ssh access to a committer to run experiments to try +to resolve the issue. + + +Required Ports +-------------- + +The buildslave operates as a *client* to the *buildmaster*. This means that +all network connections are *outbound*. This is true also for the network +tests in the test suite. Most consumer firewalls will allow any outbound +traffic, so normally you do not need to worry about what ports the buildbot +uses. However, corporate firewalls are sometimes more restrictive, so here is +a table listing all of the outbound ports used by the buildbot and the python +test suite (this list may not be complete as new tests may have been added +since this table was last vetted): + +======= =================== ================================================ +Port Host Description +======= =================== ================================================ +20, 21 ftp.debian.org test_urllib2net +53 your DNS server test_socket, and others implicitly +80 python.org (several tests) + example.com +119 news.gmane.org test_nntplib +443 (various) test_ssl +465 smtp.gmail.com test_smtpnet +587 smtp.gmail.com test_smtpnet +9020 python.org connection to buildmaster +======= =================== ================================================ + +Many tests will also create local TCP sockets and connect to them, usually +using either ``localhost`` or ``127.0.0.1``. + + +Required Resources +------------------ + +Based on the last time we did a `survey +`_ on +buildbot requirements, the recommended resource allocations for a python +buildbot are at least: + + * 2 CPUs + * 512 MB RAM + * 30 GB free disk space + +The bigmem tests won't run in this configuration, since they require +substantially more memory, but these resources should be sufficient to ensure +that Python compiles correctly on the platform and can run the rest of the test +suite. + + +Security Considerations +----------------------- + +We only allow builds to be triggered against commits to the CPython repository, +or committer-initiated branches hosted on hg.python.org. This means that the +code your buildbot will run will have been vetted by a committer. However, +mistakes and bugs happen, as could a compromise, so keep this in mind when +siting your buildbot on your network and establishing the security around it. +The buildslave should be located in an environment where an out-of-control +slave, potentially executing arbitrarily horrible and/or malicious code, should +not damage anything. A virtual machine is a good default, but may not be +sufficient. Other tools for cutting the build slave off from the outside world +would be chroot jails, Solaris zones, etc. + +Code runs differently as privileged and unprivileged users. We would love to +have builders running as privileged accounts, but security considerations do +make that difficult, as access to root can provide access to surprising +resources (such as spoofed IP packets, changes in MAC addresses, etc). + +Note that the above is a summary of a `discussion +`_ on +python-dev about buildbot security that includes examples of the tests for +which privilege matters. There was no final consensus, but the information is +useful as a point of reference. diff -r 467285511629 index.rst --- a/index.rst Mon Jun 08 22:43:52 2015 -0500 +++ b/index.rst Fri Jun 12 12:24:36 2015 -0400 @@ -216,6 +216,7 @@ compiler coverity clang + buildslave gitdevs faq