Our test suite currently contacts a range of hosts for different "online" tests, including python.org hosts.  This has led to sudden breakages in the past, e.g. when www.python.org changed URLs or switched to https by default.

It would probably a good idea to consolidate as much as possible into accessing a new VM called something like "testsuite.python.org", where the necessary services can be kept running regardless of the necessities of actual python.org services.

First step to do this is to compile a list of tests that could use the VM, and a list of services that need to be running.

There are two disabled tests in Lib/test/test_urllib2net.py:

* OtherNetworkTests.test_cnri:

    # XXX Following test depends on machine configurations that are internal
    # to CNRI.  Need to set up a public server with the right authentication
    # configuration for test purposes.

* AuthTests.test_basic_auth:

    ## Disabled at the moment since there is no page under python.org which
    ## could be used to HTTP authentication.

OK, a hopefully complete overview of the network-using tests and the services
they use, checked by tracing connect() calls during a test run:

* test_codecencodings_*, test_normalization, test_ucn: some Unicode-related data files from

  unicode.org:80
  icu-project.org:80
  people.freebsd.org:80

  Most of these files are not expected to change; they could be copied to our
  new host.  Some are dependent on the unidata version, new versions would have
  to be copied each time after updating the unicodedata database.

* test_httplib: checks HTTPS with

  svn.python.org:443

  This would require a basic HTTPS setup on the host.

* test_imaplib: checks IMAP protocol with

  cyrus.andrew.cmu.edu:143
  cyrus.andrew.cmu.edu:993

  Requires an IMAP server on the host that allows unencrypted and encrypted
  connections (both with STARTTLS and with IMAPS port) and speaks very basic
  IMAP (login then quit).

* test_nntplib: checks NNTP protocol with

  news.trigofacile.com:119
  nntp.aioe.org:563

  Requires an NNTP server on the host that allows unencrypted and encrypted
  connections (both with STARTTLS and on the NNTPS port) and has at least one
  group with some messages.

* test_smtpnet: checks SMTP protocol encryption with

  smtp.gmail.com:25
  smtp.gmail.com:465

  Requires an SMTP server that allows encrypted connections (both with STARTTLS
  and on the SMTPS port) and speaks very basic SMTP (just EHLO and then QUIT).

* test_ssl: checks encrypted connections to

  svn.python.org:443
  sha256.tbs-internet.com:443

  The svn.python.org is probably a basic HTTPS setup.  The second host is
  apparently used for especially checking SHA256.

* test_socket: makes some connection to

  example.com

  while resolving the IDNA name испытание.python.org (which is an alias).

* test_timeout: checks TCP timeout with

  blackhole.snakebite.net:56666
  whitehole.snakebite.net:56667
  www.python.org:80

  The first two are testing timeout with a connection that silently drops
  packets.  It is set up using some packet filtering rules for the two ports.

* test_urllibnet and test_urllib2net: various checks of HTTP, FTP protocols on

  example.com:80
  www.google.com:80
  bit.ly:80
  docs.python.org:443
  imdb.com:80
  ftp.debian.org:21
  ftp.mirror.nl:21
  gatekeeper.research.compaq.com:21

  Most of them are probably fine with a basic readonly FTP server that servers a
  few files and a basic HTTP server.  Some check special cases with broken
  servers (like imdb.com) that might be a little harder to reproduce.

  The two disabled tests mentioned by Berker need HTTP Basic Auth enabled
  on some URL.

* test_xmlrpc_net: checks XML-RPC protocol with

  buildbot.python.org:80

  With the current test, serving a static XML page at a certain URL is
  sufficient: it makes only one request.  Another possibility is setting up a
  small XML-RPC server on a dedicated port.

Note we now have pythontest.net set up.

Nice! How do you plan to organize its setup?

Why a separate domain rather than a subdomain?  I'm not objecting, but I am curious.

It is configured using salt, see https://github.com/python/psf-salt/blob/master/salt/pythontest/init.sls.

A separate domain just makes it easier to do whatever we need with it without needing to worry about getting confused between live sites and test sites.

Not to mention possible cookie tests, I suppose.
(I would personally have preferred something like "comfychair.net", but I guess pythontest.net is ok too :-))

I created a repository https://hg.python.org/pythontestdotnet, which is cloned on the server.

4985375db40f takes care of test_httplib.

New changeset 39536b377241 by Georg Brandl in branch '3.4':
#22650: test suite: load Unicode test data files from www.pythontest.net
https://hg.python.org/cpython/rev/39536b377241

New changeset 0af36ea1d010 by Georg Brandl in branch '2.7':
#22650: test suite: load Unicode test data files from www.pythontest.net
https://hg.python.org/cpython/rev/0af36ea1d010

Opened up https://github.com/python/cpython/pull/2755/ in order to address https://bugs.python.org/issue30883

This changes the ftp servers from debian.org to pythontest.net

FYI, sha256.tbs-internet.com is no longer used in tests.