classification
Title: test_ftplib timeouts
Type: behavior Stage: resolved
Components: Tests Versions: Python 3.4, Python 3.3, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: giampaolo.rodola, neologix, pitrou, python-dev
Priority: normal Keywords: easy, patch

Created on 2013-08-20 21:01 by pitrou, last changed 2013-08-21 22:59 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
support_host.patch pitrou, 2013-08-21 08:58 review
Messages (12)
msg195717 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-08-20 21:01
Some ftplib tests sometimes time out. Seem they seem to try to connect to "localhost" (rather than "127.0.0.1"), I was wondering if this could be a DNS issue and if we should resolve "localhost" in advance like Charles-François did for some other tests (or simply hardcode "127.0.0.1", actually).

http://buildbot.python.org/all/builders/x86%20Windows7%203.x/builds/7060/steps/test/logs/stdio

(example:

======================================================================
ERROR: testTimeoutDefault (test.test_ftplib.TestTimeouts)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\test_ftplib.py", line 953, in testTimeoutDefault
    ftp = ftplib.FTP("localhost")
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\ftplib.py", line 115, in __init__
    self.connect(host)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\ftplib.py", line 150, in connect
    source_address=self.source_address)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\socket.py", line 454, in create_connection
    raise err
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\socket.py", line 445, in create_connection
    sock.connect(sa)
socket.timeout: timed out

)
msg195750 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-08-21 07:25
> Some ftplib tests sometimes time out. Seem they seem to try to connect to "localhost" (rather than "127.0.0.1"), I was wondering if this could be a DNS issue and if we should resolve "localhost" in advance like Charles-François did for some other tests (or simply hardcode "127.0.0.1", actually).

If it only happens on Windows >= 7 buildbots, then it's likely a
consequence of this:
http://serverfault.com/questions/4689/windows-7-localhost-name-resolution-is-handled-within-dns-itself-why

Apparently, since Windows 7, "localhost" doesn't appear in the hosts
file anymore: so, depending on your DNS resolver settings, this could
perfectly explain such timeouts (assuming it's not a firewall issue).

Hard-cording 127.0.0.1 for this test should be OK, because the server
setup explicitly binds an AF_INET socket. We'll probably have to check
the rest of the test suite for similar issues.
msg195752 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-08-21 08:04
> Hard-cording 127.0.0.1 for this test should be OK, because the server
> setup explicitly binds an AF_INET socket. We'll probably have to check
> the rest of the test suite for similar issues.

Ok, let's do it!
msg195754 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-08-21 08:58
Here is a patch for default.
msg195766 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-08-21 10:25
Changing support.HOST from 'localhost' to '127.0.0.1' means that on dual-stack
hosts (I don't think the test suite would run on IPv6-only hosts anyway), the
tests will now always use IPv4, whereas they are currently using either IPv6 or
IPv4 addresses depending on the host's name resolution settings.
For example, on a RHEL6 box:

>>> socket.getaddrinfo('localhost', 0)
[(10, 1, 6, '', ('::1', 0, 0, 0)), (10, 2, 17, '', ('::1', 0, 0, 0)),
(10, 3, 0, '', ('::1', 0, 0, 0)), (2, 1, 6, '', ('127.0.0.1', 0)), (2,
2, 17, '', ('127.0.0.1', 0)), (2, 3, 0, '', ('127.0.0.1', 0))]

So on this host, 'localhost' resolves to '::1'. I think it would be
better to have support.HOST set to socket.getaddrinfo('localhost', 0)[0][4][0]
(sic), so that the test will use the preferred protocol on the target host, and
also that IPv6 is also tested in tests which don't explicitely use
'::1'. But this
means that we should have an support.HOSTv4 for tests that explicitly rely on
IPv4, e.g.  test_ftplib.
msg195770 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-08-21 10:43
> Changing support.HOST from 'localhost' to '127.0.0.1' means that on
> dual-stack
> hosts (I don't think the test suite would run on IPv6-only hosts
> anyway), the
> tests will now always use IPv4, whereas they are currently using
> either IPv6 or
> IPv4 addresses depending on the host's name resolution settings.

That would only be true if the server we are contacting happens to
listen on both IPv6 and IPv4, right? I don't think we have such a
situation in the test suite. Clients can use create_connection()
for transparent support of IPv6 and IPv4, but servers have to choose
a family when creating their socket.
msg195779 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-08-21 12:03
> That would only be true if the server we are contacting happens to
> listen on both IPv6 and IPv4, right? I don't think we have such a
> situation in the test suite.

Ah, I thought some code was using getaddrinfo() to select an arbitrary
bind() address for localhost.

That's OK then.
msg195806 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-08-21 16:49
LGTM
msg195831 - (view) Author: Roundup Robot (python-dev) Date: 2013-08-21 22:39
New changeset 085ba7d85eb2 by Antoine Pitrou in branch 'default':
Issue #18792: Use "127.0.0.1" or "::1" instead of "localhost" as much as possible, since "localhost" goes through a DNS lookup under recent Windows versions.
http://hg.python.org/cpython/rev/085ba7d85eb2
msg195833 - (view) Author: Roundup Robot (python-dev) Date: 2013-08-21 22:49
New changeset 7728b073c77c by Antoine Pitrou in branch '3.3':
Issue #18792: Use "127.0.0.1" or "::1" instead of "localhost" as much as possible, since "localhost" goes through a DNS lookup under recent Windows versions.
http://hg.python.org/cpython/rev/7728b073c77c
msg195834 - (view) Author: Roundup Robot (python-dev) Date: 2013-08-21 22:58
New changeset 48de8df194d9 by Antoine Pitrou in branch '2.7':
Issue #18792: Use "127.0.0.1" or "::1" instead of "localhost" as much as possible, since "localhost" goes through a DNS lookup under recent Windows versions.
http://hg.python.org/cpython/rev/48de8df194d9
msg195835 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-08-21 22:59
Done, thanks :)
History
Date User Action Args
2013-08-21 22:59:07pitrousetstatus: open -> closed
resolution: fixed
messages: + msg195835

stage: needs patch -> resolved
2013-08-21 22:58:36python-devsetmessages: + msg195834
2013-08-21 22:49:00python-devsetmessages: + msg195833
2013-08-21 22:39:57python-devsetnosy: + python-dev
messages: + msg195831
2013-08-21 16:49:24neologixsetmessages: + msg195806
2013-08-21 12:03:15neologixsetmessages: + msg195779
2013-08-21 10:43:57pitrousetmessages: + msg195770
2013-08-21 10:25:14neologixsetmessages: + msg195766
2013-08-21 08:58:21pitrousetfiles: + support_host.patch
keywords: + patch
messages: + msg195754
2013-08-21 08:04:11pitrousetkeywords: + easy

messages: + msg195752
2013-08-21 07:25:08neologixsetmessages: + msg195750
2013-08-20 21:01:29pitroucreate