Background: I came across this issue when trying to track down why 
test_asynchat would periodically wedge python processes on the Windows 
buildbots, to the point that they wouldn't even respond to SIGKILL (or 
ctrl-c on the console).

What I found after a bit of digging is that Windows doesn't raise 
EADDRINUSE socket.errors when you bind() two sockets to identical 
host/ports *IFF* SO_REUSEADDR has been set as a socket option.

Decided to brighten up my tube journey into work this morning by 
reading the Gospel's take on the situation.  As per the 'SO_REUSEADDR 
and SO_REUSEPORT Socket Options' section in chapter 7.5 of Stevens' 
UNIX Network Programming Volume 1 (2nd Ed):

"With TCP, we are never able to start multiple servers that bind
 the same IP address and same port: a completely duplicate binding.
 That is, we cannot start one server that binds 198.69.10.2 port 80
 and start another that also binds 198.69.10.2 port 80, even if we
 set the SO_REUSEADDR socket option for the second server."

So, it seems at least Windows isn't adhering to this, at least on XP 
and Server 2008 with 2.5-2.6.  I've patched test_socket.py to 
explicitly test for this situation -- as expected, it passes on Unix 
(tested on FreeBSD in particular), and fails on Windows.  I'd like to 
commit this to trunk to see if any of the buildbots for different 
platforms match the behaviour of Windows.

[Updating the issue with relevant mailing list conversation]
Interesting results!  I committed the patch to test_socket.py in 
r62152.  I was expecting all other platforms except for Windows to 
behave consistently (i.e. pass).  That is, given the following:

        import socket
        host = '127.0.0.1'
        sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        sock.bind((host, 0))
        port = sock.getsockname()[1]
        sock.close()
        del sock

        sock1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        sock1.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        sock1.bind((host, port))
        sock2 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        sock2.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        sock2.bind((host, port))
        ^^^^

....the second bind should fail with EADDRINUSE, at least according to 
the 'SO_REUSEADDR and SO_REUSEPORT Socket Options' section in chapter 
7.5 of Stevens' UNIX Network Programming Volume 1 (2nd Ed):

"With TCP, we are never able to start multiple servers that bind
 the same IP address and same port: a completely duplicate binding.
 That is, we cannot start one server that binds 198.69.10.2 port 80
 and start another that also binds 198.69.10.2 port 80, even if we
 set the SO_REUSEADDR socket option for the second server."

The results: both Windows *and* Linux fail the patched test; none of 
the buildbots for either platform encountered an EADDRINUSE 
socket.error after the second bind().  FreeBSD, OS X, Solaris and Tru64 
pass the test -- EADDRINUSE is raised on the second bind.  (Interesting 
that all the ones that passed have a BSD lineage.)

I've just reverted the test in r62156 as planned.  The real issue now 
is that there are tests that are calling test_support.bind_socket() 
with the assumption that the port returned by this method is 'unbound', 
when in fact, the current implementation can't guarantee this:

def bind_port(sock, host='', preferred_port=54321):
    for port in [preferred_port, 9907, 10243, 32999, 0]:
        try:
            sock.bind((host, port))
            if port == 0:
                port = sock.getsockname()[1]
            return port
        except socket.error, (err, msg):
            if err != errno.EADDRINUSE:
                raise
            print >>sys.__stderr__, \
                '  WARNING: failed to listen on port %d, trying 
another' % port

This logic is only correct for platforms other than Windows and Linux.  
I haven't looked into all the networking test cases that rely on 
bind_port(), but I would think an implementation such as this would be 
much more reliable than what we've got for returning an unused port:

def bind_port(sock, host='127.0.0.1', *args):
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.bind((host, 0))
    port = s.getsockname()[1]
    s.close()
    del s

    sock.bind((host, port))
    return port

Actually, FWIW, I just ran a full regrtest.py against trunk on Win32 
with this change in place and all the tests still pass.

Thoughts?

    Trent.

[Updating issue with mailing list discussion; Jean-Paul's reply]
On Fri, 4 Apr 2008 13:24:49 -0700, Trent Nelson <tnelson@onresolve.com> 
wrote:
>Interesting results!  I committed the patch to test_socket.py in 
r62152.  I was expecting all other platforms except for Windows to 
behave consistently (i.e. pass).  That is, given the following:
>
>        import socket
>        host = '127.0.0.1'
>        sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>        sock.bind((host, 0))
>        port = sock.getsockname()[1]
>        sock.close()
>        del sock
>
>        sock1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>        sock1.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
>        sock1.bind((host, port))
>        sock2 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>        sock2.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
>        sock2.bind((host, port))
>        ^^^^
>
>....the second bind should fail with EADDRINUSE, at least according to 
the 'SO_REUSEADDR and SO_REUSEPORT Socket Options' section in chapter 
7.5 of Stevens' UNIX Network Programming Volume 1 (2nd Ed):
>
>"With TCP, we are never able to start multiple servers that bind
> the same IP address and same port: a completely duplicate binding.
> That is, we cannot start one server that binds 198.69.10.2 port 80
> and start another that also binds 198.69.10.2 port 80, even if we
> set the SO_REUSEADDR socket option for the second server."
>
>The results: both Windows *and* Linux fail the patched test; none of 
the buildbots for either platform encountered an EADDRINUSE 
socket.error after the second bind().  FreeBSD, OS X, Solaris and Tru64 
pass the test -- EADDRINUSE is raised on the second bind.  (Interesting 
that all the ones that passed have a BSD lineage.)

Notice that the quoted text explains that you cannot start multiple 
servers
that etc.  Since you didn't call listen on either socket, it's arguable 
that
you didn't start any servers, so there should be no surprise regarding 
the
behavior.  Try adding listen calls at various places in the example and
you'll see something different happen.

FWIW, AIUI, SO_REUSEADDR behaves just as described in the above quote on
Linux/BSD/UNIX/etc.  On Windows, however, that option actually means
something quite different.  It means that the address should be stolen 
from
any process which happens to be using it at the moment.

There is another option, SO_EXCLUSIVEADDRUSE, only on Windows I think,
which, AIUI, makes it impossible for another process to steal the port
using SO_REUSEADDR.

Hope this helps,

Jean-Paul

[Updating issue with mailing list discussion; my reply to Jean-Paul]
> >"With TCP, we are never able to start multiple servers that bind
> > the same IP address and same port: a completely duplicate binding.
> > That is, we cannot start one server that binds 198.69.10.2 port 80
> > and start another that also binds 198.69.10.2 port 80, even if we
> > set the SO_REUSEADDR socket option for the second server."

> Notice that the quoted text explains that you cannot start multiple
> servers that etc.  Since you didn't call listen on either socket, it's
> arguable that you didn't start any servers, so there should be no
> surprise regarding the behavior.  Try adding listen calls at various
> places in the example and you'll see something different happen.

I agree in principle, Stevens says nothing about what happens if you 
*do* try and bind two sockets on two identical host/port addresses.  
Even so, test_support.bind_port() makes an assumption that bind() will 
raise EADDRINUSE if the port is not available, which, as has been 
demonstrated, won't be the case on Windows or Linux.

> FWIW, AIUI, SO_REUSEADDR behaves just as described in the above quote
> on Linux/BSD/UNIX/etc.  On Windows, however, that option actually 
means
> something quite different.  It means that the address should be stolen
> from any process which happens to be using it at the moment.

Probably explains why the python process wedges when this happens on 
Windows...

> There is another option, SO_EXCLUSIVEADDRUSE, only on Windows I think,
> which, AIUI, makes it impossible for another process to steal the port
> using SO_REUSEADDR.

Nod, if SO_EXCLUSIVEADDRUSE is used instead in the code I posted, 
Windows raises EADDRINUSE on the second bind().  I don't have access to 
any Linux boxes at the moment, so I can't test what sort of error is 
raised with the example I posted if listen() and accept() are called on 
the two sockets bound to identical addresses.  Can anyone else shed 
some light on this?  I'd be interested in knowing if the process wedges 
on Linux as badly as it does on Windows (to the point where it's not 
respecting ctrl-c or sigkill).


        Trent.

I've attached another patch that fixes test_support.bind_port() as well 
as a bunch of files that used that method.  The new implementation 
always uses an ephemeral port in order to elicit an unused port for 
subsequent binding.  Tested on Windows 32-bit & x64 and FreeBSD 6.2.  
Would like to apply sooner rather than later unless anyone has any 
objections as it'll fix my two Windows buildbots that are on the same 
machine from both hanging if they test asynchat at the same time (which 
happens more often than you'd think).

Trent, go ahead and try this out.  We should definitely be moving in
this direction.  So I'd rather fix the problem than keep suffering with
the current problems of not being able to run the test suite
concurrently.  I think bind_port might be documented, so you should
update the docs if so.  Also, please add a Misc/NEWS entry.

I don't like that the patch changes the API of a function in
test_support() (in particular changing the return type; adding optional
arguments is not a problem).  This could trip up 3rd party users of this
API.  I recommend creating a new API bind_host_and_port() (or whatever
you'd like to name it) and implement the original API in terms of the
new one.  (You can even add a warning if you think the original API is
always unsafe.)

To be honest, I wasn't really happy either with having to return HOST, 
it's somewhat redundant given that all these tests should be binding 
against localhost.  What about something like this for bind_port():

def bind_port(sock, host=''):
    """Bind the socket to a free port and return the port number.
    Relies on ephemeral ports in order to ensure we are using an
    unbound port.  This is important as many tests may be running
    simultaneously, especially in a buildbot environment."""

    # Use a temporary socket object to ensure we're not 
    # affected by any socket options that have already 
    # been set on the 'sock' object we're passed. 
    tempsock = socket.socket(sock.family, sock.type)
    tempsock.bind((host, 0))
    port = tempsock.getsockname()[1]
    tempsock.close()
    del tempsock

    sock.bind((host, port))
    return port

The tests would then look something like:

HOST = 'localhost'
PORT = None

class Foo(TestCase):
    def setUp(self):
        sock = socket.socket()
        global PORT
        PORT = test_support.bind_port(sock, HOST)

So, the return value is the port bound to, no change there, but we're 
abolishing preferred_port as an optional argument, which is important, 
IMO, as none of these tests should be stipulating which port they want 
to listen on.  That's actually the root of this entire problem.

Thanks, that's much better (though I'm not the authority on all details
of this patch).

Invested quite a few cycles on this issue last night.  The more time I 
spent on it, the more I became convinced that every single test working 
with sockets should be changed in one fell swoop in order to facilitate 
(virtually unlimited) parallel test execution without fear of port 
conflicts.

I've attached a second patch, trunk.2550-2.patch, which is my progress 
so far on doing just this.  The main changes can be expressed by the 
following two points:

a) do whatever it takes in network-oriented tests to ensure
   unique ports are obtained (relying on the bind_port() and
   find_unused_port() methods exposed by test_support)

b) never, ever, ever call SO_REUSEADDR on a socket from a test;
   because we're putting so much effort into obtaining a unique
   port, this should never be necessary -- in the rare cases that
   our attempts to obtain a unique port fail, then we absolutely
   should fail with EADDRINUSE, as the ability to obtain a unique
   port for the duration of a client/server test is an invariant
   that we *must* be able to depend upon.  If the invariant is
   broken, fail immediately (don't mask the problem with 
   SO_REUSEADDR).

With this patch applied, I can spawn a handful of Python processes and 
run the entire test suite (without -r, ensuring all tests are run in 
the same order, which should encourage port conflicts (if there were 
any)) without any errors*.  Doing that now is completely and utterly 
impossible.

[*] Well, almost without error.  All the I/O related tests that try and 
open @test fail.

I believe there's still outstanding work to do with this patch with 
regards to how the intracacies of SO_REUSEADDR and SO_EXCLUSIVEADDRUSE 
should be handled in the rest of the stdlib.  I'm still thinking about 
the best approach for this.  However, the patch as it currently stands 
is still quite substantial so I wanted to get it out sooner rather than 
later for review.

(I'll forward this to python-dev@ to try and encourage more eyes from 
people with far more network-fu than I.)

Committed updates to relevant network-oriented tests, as well as 
test_support changes discussed, in r62234.

This is now fixed, right? Personal experience as well as buildbot behaviour seems to show that parallel test execution (either through -j, or by running several test suites at the same time) works ok.