New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
buildbot: test_urllib2_localnet failures (Connection refused) on Tiger buildbot #52701
Comments
http://www.python.org/dev/buildbot/builders/x86 Tiger 3.x/builds/25/steps/test/logs/stdio test_urllib2_localnet ====================================================================== Traceback (most recent call last):
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/urllib/request.py", line 1072, in do_open
h.request(req.get_method(), req.selector, req.data, headers)
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/http/client.py", line 938, in request
self._send_request(method, url, body, headers)
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/http/client.py", line 976, in _send_request
self.endheaders(body)
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/http/client.py", line 934, in endheaders
self._send_output(message_body)
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/http/client.py", line 788, in _send_output
self.send(msg)
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/http/client.py", line 729, in send
self.connect()
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/http/client.py", line 711, in connect
self.timeout, self.source_address)
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/socket.py", line 321, in create_connection
raise error(msg)
socket.error: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/test_urllib2_localnet.py", line 278, in test_proxy_qop_auth_works
result = self.opener.open(self.URL)
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/urllib/request.py", line 349, in open
response = self._open(req, data)
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/urllib/request.py", line 367, in _open
'_open', req)
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/urllib/request.py", line 327, in _call_chain
result = func(*args)
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/urllib/request.py", line 1090, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/urllib/request.py", line 1075, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 61] Connection refused> ====================================================================== Traceback (most recent call last):
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/urllib/request.py", line 1072, in do_open
h.request(req.get_method(), req.selector, req.data, headers)
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/http/client.py", line 938, in request
self._send_request(method, url, body, headers)
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/http/client.py", line 976, in _send_request
self.endheaders(body)
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/http/client.py", line 934, in endheaders
self._send_output(message_body)
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/http/client.py", line 788, in _send_output
self.send(msg)
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/http/client.py", line 729, in send
self.connect()
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/http/client.py", line 711, in connect
self.timeout, self.source_address)
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/socket.py", line 321, in create_connection
raise error(msg)
socket.error: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/test_urllib2_localnet.py", line 266, in test_proxy_with_bad_password_raises_httperror
self.URL)
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/case.py", line 456, in assertRaises
callableObj(*args, **kwargs)
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/urllib/request.py", line 349, in open
response = self._open(req, data)
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/urllib/request.py", line 367, in _open
'_open', req)
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/urllib/request.py", line 327, in _call_chain
result = func(*args)
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/urllib/request.py", line 1090, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/urllib/request.py", line 1075, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 61] Connection refused> ====================================================================== Traceback (most recent call last):
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/urllib/request.py", line 1072, in do_open
h.request(req.get_method(), req.selector, req.data, headers)
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/http/client.py", line 938, in request
self._send_request(method, url, body, headers)
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/http/client.py", line 976, in _send_request
self.endheaders(body)
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/http/client.py", line 934, in endheaders
self._send_output(message_body)
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/http/client.py", line 788, in _send_output
self.send(msg)
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/http/client.py", line 729, in send
self.connect()
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/http/client.py", line 711, in connect
self.timeout, self.source_address)
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/socket.py", line 321, in create_connection
raise error(msg)
socket.error: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/test_urllib2_localnet.py", line 272, in test_proxy_with_no_password_raises_httperror
self.URL)
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/case.py", line 456, in assertRaises
callableObj(*args, **kwargs)
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/urllib/request.py", line 349, in open
response = self._open(req, data)
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/urllib/request.py", line 367, in _open
'_open', req)
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/urllib/request.py", line 327, in _call_chain
result = func(*args)
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/urllib/request.py", line 1090, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/urllib/request.py", line 1075, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 61] Connection refused> Ran 14 tests in 10.076s |
This appears to be a recently introduced failure; the same failure is seen on current trunk build on 10.5 and 10.6 as well. |
Maybe r80198 of bpo-7154? |
That wouldn't explain trunk failures. The _scproxy code has been in 26 and trunk for a long time. |
For what it's worth, I added a few debugging statements to a local trunk checkout, and it looks like the port is getting lost somewhere along the way to setup the proxy... So the refused error is accurate and because it's trying the default port 80. Note also that I think all the Proxy tests fail, even those saying they're ok, because the refused error shows up as a URLError which is what the test expects, but for other reasons. Oh, and small item I noted while doing the test - the urllib2_localnet module doesn't clean up properly during the tests - all the local servers are left listening on their ports until all the tests are done. Adding a call to self.httpd.server_close() at the end of run() in LoopbackHttpServerThread appears to fix that. Here's a sample of one test... test_proxy_qop_auth_int_works_or_throws_urlerror (test.test_urllib2_localnet.ProxyAuthTests) ... Init done |
I'm also seeing this on OS X 10.6. It seems to have started with r80243. |
To clarify, that last message was about trunk, where this test is failing for me since r80243. Adding 2.7 to the versions. |
Apparently, the failure on OSX is due to the fact that urllib.proxy_bypass('localhost') returns True. This makes the opener to bypass the ProxyHandler that is explicitly set up to use the correct server port: def setUp(self):
..
proxy_url = "http://127.0.0.1:%d" % self.server.port
handler = urllib2.ProxyHandler({"http" : proxy_url})
self.proxy_digest_handler = urllib2.ProxyDigestAuthHandler()
self.opener = urllib2.build_opener(handler, self.proxy_digest_handler) instead, the opener skips to the default HTTPHandler which attempts to connect on port 80 with sad results. Interestingly, >>> urllib.proxy_bypass('127.0.0.1')
False So the simplest fix is s/localhost/127.0.0.1/ as done in the attached patch (bpo-8455.diff). |
In fact, urllib.proxy_bypass() returns True for any simple host name (i.e. any string without '.' in it): >>> proxy_bypass('xyz')
True
>>> proxy_bypass('')
True
>>> proxy_bypass('whatever')
True I think the fix I am proposing makes sense regardless of platform because proxy_url is set up numerically: proxy_url = "http://127.0.0.1:%d" % self.server.port Maybe the above should be changed to proxy_url = "%s:%d" % (self.URL, self.server.port) once URL is changed to numerical form. |
Replacing 'http://localhost' with 'http://127.0.0.1' for this test is fine. But, urllib.proxy_bypass returning for True for any string, is not correct. In the code, I see that Proxy Settings from Mac OSX system configuration is obtained by calling _get_proxy_settings(). The is the check to to see if proxy_settings['exclude_simple'] is set. |
The mac-related code in urllib is correct, it uses the SystemConfiguration framework on MacOSX to get the proxy settings. That said, the code in _scproxy is not entirly correct, it makes the wrong assumption w.r.t. the value of exclude_simple when the corresponding value is not in the settings retrieved from the SystemConfiguration framework. The tests seem to make assumptions on how the proxy-detecting code functions, although I don't understand what's going on here (that is, I haven't waded through the test code and urllib to trace the code) The patch "issue-8455-scproxy.patch" ensures that the tests pass, unless the user changed their proxy settings and selected the checkbox "Exclude simple hostnames" (System Preferernces -> Network -> First network adaptor -> Advanced... -> Proxies) Please test if the patch solves the issue for you as well. NOTE: as I mention in the second paragraph the tests seem to make unwarrented assumptions about their environment. Further proof for this: |
I've applied my patch in r82150, which fixes the issue for me in the trunk. I'll merge the patch into the other branches if this does indeed fix the buildbot issue. |
Just to confirm that with this fix test_urllib2_localnet is now passing for me. Thanks! |
Applied to 2.6 (r82280), r82281 (3.2), r82282 (3.1). |
The problem still exists on OS X 10.3. See bpo-9568 for patches. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: