diff --git a/buildslave.rst b/buildslave.rst new file mode 100644 --- /dev/null +++ b/buildslave.rst @@ -0,0 +1,141 @@ +.. _buildslave: + +Running a Build Slave +===================== + +The build slaves are an essential part of the Python development +process (see the :doc:`continuous_integration` page). Running a build +slave is a great way to help the Python community. It's a good way to +help even if you don't have much programming experience. + +We need to add more platforms (and compilers!) to the BuildBot farm. If +you have one of these machines you are willing to run BuildBot on, +please contact python-dev@python.org: + +* Windows (see below) +* AIX +* HP-UX +* FreeBSD, NetBSD, OpenBSD +* Solaris / OpenSolaris / OpenIndiana / Illumos + +It might also be beneficial to run on older versions of operating +systems or others not mentioned above. Feel free to contact us if you +would like to offer another type of system. + +Running a build slave in a virtual machine is a common practice and a +good idea. It allows you to run more than one build slave (for +different operating systems) on the same physical computer. However, +there is no requirement that you run build slaves in virtual machines. + +Installing a Build Salve (buildbot) +----------------------------------- + +Python core development uses `Buildbot `_ for +continuous integration. Installing a buildbot slave requires three +things: + +1. a recent version of Buildbot, +2. `Mercurial `_, +3. the standard development toolchain. + +If your OS has a packaging system (e.g. under Linux), Buildbot and +Mercurial are probably available from the standard repositories. +Otherwise, you'll have to build them from source (in which case you'll +have to install Python and Twisted first). + +A build slave needs to be able to build Python and some of its +extension modules. That means you should have the standard development +tools installed (compiler, linker), and the development headers for a +couple of third-party libraries (such as zlib and OpenSSL). + +Once all this is done, create a new user "buildbot" if it doesn't exist +(your package manager might have done it for you). Then:: + + % su - buildbot + % mkdir buildarea + % buildslave create-slave `pwd`/buildarea dinsdale.python.org:9020 slavename slavepasswd + +If you have an older version of buildbot installed, the command will be +"buildbot" instead of "buildslave". If this is the case, you should +upgrade to a newer version of buildbot. If you can't find either, you +should check to see that /usr/local/bin (the default installed location) +is on your path. + +You'll need to get someone to create the slavename/slavepasswd on +dinsdale.python.org before doing this. Ask on python-dev@python.org +once you're ready. + +Then edit buildarea/info/admin and buildarea/info/host to set them +appropriately. + +Finally, start buildbot (still under the 'buildbot' user) with: + + % buildslave start ~/buildarea + +Once the buildbot is running, don't forget to monitor the build results +and solve any setup issues causing test failures. + +Buildbot on Windows +------------------- + +See the page at http://wiki.python.org/moin/BuildbotOnWindows. + +Build Slaves and Security +------------------------- + +Here are some important points of which you should be aware when +running a build slave [1]_: + +1. Anyone setting up a buildslave should take care to invoke the build + in an environment where an out-of-control buildbot, potentially + executing arbitrarily horrible and/or malicious code, should not + damage anything. Builders should always be isolated from valuable + resources, although the specific mechanism of isolation may differ. + A virtual machine is a good default, but may not be sufficient; + other tools for cutting of the builder from the outside world would + be chroot jails, solaris zones, etc. +2. Code runs differently as privileged vs. unprivileged users. + Therefore builders should be set up in both configurations, running + the full test suite, to ensure that all code runs as expected in + both configurations. Some tests, as the start of `this thread + `_ + indicates, must have some special logic to make sure they do or do + not run, or run differently, in privileged vs. unprivileged + configurations, but generally speaking most things should work in + both places. +3. Access to root may provide access to slightly surprising resources, + even within a VM (such as the ability to send spoofed IP packets, + change the MAC address of even virtual ethernet cards, etc), and + administrators should be aware that this is the case when + configuring the host environment for a run-as-root builder. You + don't want to end up with a compromised test VM that can snoop on + your network. + +.. [1] See the related `python-dev thread `_. + +Required Network Ports +---------------------- + +Running a build slave requires some network access. Here are the +inbound network ports that need to be available on the host's public +interface. + +The description column mostly refers to the tests in the suite that +need that port. If any of those tests are failing, make sure the +corresponding port is open. + +========== ============================= =================================== +Port Host Description +========== ============================= =================================== +20, 21 ftp.kernel.org, ftp.mirror.nl test_urllib2net +53 your DNS server test_socket (and others implicitly) +80 python.org (several tests) +119 news.gmane.org test_nntplib +443 (various) test_ssl +465 smtp.gmail.com test_smtpnet +9020 python.org Buildbot connection +========== ============================= =================================== + +Many tests will also create local TCP sockets and connect to them +(usually through "localhost" or "127.0.0.1"), so make sure this +capability hasn't been disabled. diff --git a/buildbots.rst b/continuous_integration.rst rename from buildbots.rst rename to continuous_integration.rst --- a/buildbots.rst +++ b/continuous_integration.rst @@ -1,4 +1,4 @@ -.. _buildbots: +.. _continuous_integration: Continuous Integration ====================== @@ -23,6 +23,9 @@ important that you get acquainted with the way these results are presented, and how various kinds of failures can be explained and diagnosed. +.. note:: + See the :doc:`buildslave` page for more on running a build slave. + Checking results of automatic builds ------------------------------------ diff --git a/index.rst b/index.rst --- a/index.rst +++ b/index.rst @@ -66,17 +66,18 @@ * :doc:`coverage` * Advanced tasks for once you are comfortable * :doc:`silencewarnings` - * Fixing issues found by the :doc:`buildbots ` + * Fixing issues found by the :doc:`buildbots ` * :doc:`fixingissues` * :doc:`helptriage` * :doc:`devrole` * :doc:`triaging` * :doc:`languishing` * :doc:`communication` +* :doc:`buildslave` * :doc:`coredev` * :doc:`committing` * :doc:`devcycle` - * :doc:`buildbots` + * :doc:`continuous_integration` It is **recommended** that the above documents be read in the order listed. You can stop where you feel comfortable and begin contributing immediately