This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: tests for urllib2net are in bad shapes
Type: behavior Stage: resolved
Components: Tests Versions: Python 3.5
process
Status: closed Resolution: works for me
Dependencies: Superseder:
Assigned To: orsenthil Nosy List: karlcow, orsenthil
Priority: normal Keywords:

Created on 2014-09-24 07:49 by karlcow, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Messages (5)
msg227417 - (view) Author: karl (karlcow) * Date: 2014-09-24 07:49
→ ./python.exe -V
Python 3.4.2rc1+

→ hg tip
changeset:   92532:6dcc96fa3970
tag:         tip
parent:      92530:ad45c2707006
parent:      92531:8eb4eec8626c
user:        Benjamin Peterson <benjamin@python.org>
date:        Mon Sep 22 22:44:21 2014 -0400
summary:     merge 3.4 (#22459)


When working on issue #5550, I realized that some tests are currently failing.

Here the log of running:
→ ./python.exe -m unittest -v Lib/test/test_urllib2net.py 


test_close (Lib.test.test_urllib2net.CloseSocketTest) ... ok
test_custom_headers (Lib.test.test_urllib2net.OtherNetworkTests) ... FAIL
test_file (Lib.test.test_urllib2net.OtherNetworkTests) ... test_ftp (Lib.test.test_urllib2net.OtherNetworkTests) ... skipped "Resource 'ftp://gatekeeper.research.compaq.com/pub/DEC/SRC/research-reports/00README-Legal-Rules-Regs' is not available"
test_redirect_url_withfrag (Lib.test.test_urllib2net.OtherNetworkTests) ... ok
test_sites_no_connection_close (Lib.test.test_urllib2net.OtherNetworkTests) ... ok
test_urlwithfrag (Lib.test.test_urllib2net.OtherNetworkTests) ... ok
test_ftp_basic (Lib.test.test_urllib2net.TimeoutTest) ... ok
test_ftp_default_timeout (Lib.test.test_urllib2net.TimeoutTest) ... ok
test_ftp_no_timeout (Lib.test.test_urllib2net.TimeoutTest) ... ok
test_ftp_timeout (Lib.test.test_urllib2net.TimeoutTest) ... ok
test_http_basic (Lib.test.test_urllib2net.TimeoutTest) ... ok
test_http_default_timeout (Lib.test.test_urllib2net.TimeoutTest) ... ok
test_http_no_timeout (Lib.test.test_urllib2net.TimeoutTest) ... ok
test_http_timeout (Lib.test.test_urllib2net.TimeoutTest) ... ok

======================================================================
ERROR: test_file (Lib.test.test_urllib2net.OtherNetworkTests) (url='file:/Users/karl/code/cpython/%40test_61795_tmp')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/karl/code/cpython/Lib/test/test_urllib2net.py", line 243, in _test_urls
    f = urlopen(url, req, TIMEOUT)
  File "/Users/karl/code/cpython/Lib/test/test_urllib2net.py", line 33, in wrapped
    return _retry_thrice(func, exc, *args, **kwargs)
  File "/Users/karl/code/cpython/Lib/test/test_urllib2net.py", line 23, in _retry_thrice
    return func(*args, **kwargs)
  File "/Users/karl/code/cpython/Lib/urllib/request.py", line 447, in open
    req = Request(fullurl, data)
  File "/Users/karl/code/cpython/Lib/urllib/request.py", line 267, in __init__
    origin_req_host = request_host(self)
  File "/Users/karl/code/cpython/Lib/urllib/request.py", line 250, in request_host
    host = _cut_port_re.sub("", host, 1)
TypeError: expected string or buffer

======================================================================
ERROR: test_file (Lib.test.test_urllib2net.OtherNetworkTests) (url=('file:///nonsensename/etc/passwd', None, <class 'urllib.error.URLError'>))
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/karl/code/cpython/Lib/test/test_urllib2net.py", line 243, in _test_urls
    f = urlopen(url, req, TIMEOUT)
  File "/Users/karl/code/cpython/Lib/test/test_urllib2net.py", line 33, in wrapped
    return _retry_thrice(func, exc, *args, **kwargs)
  File "/Users/karl/code/cpython/Lib/test/test_urllib2net.py", line 23, in _retry_thrice
    return func(*args, **kwargs)
  File "/Users/karl/code/cpython/Lib/urllib/request.py", line 447, in open
    req = Request(fullurl, data)
  File "/Users/karl/code/cpython/Lib/urllib/request.py", line 267, in __init__
    origin_req_host = request_host(self)
  File "/Users/karl/code/cpython/Lib/urllib/request.py", line 250, in request_host
    host = _cut_port_re.sub("", host, 1)
TypeError: expected string or buffer

======================================================================
FAIL: test_custom_headers (Lib.test.test_urllib2net.OtherNetworkTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/karl/code/cpython/Lib/test/test_urllib2net.py", line 186, in test_custom_headers
    self.assertEqual(request.get_header('User-agent'), 'Test-Agent')
AssertionError: 'Python-urllib/3.4' != 'Test-Agent'
- Python-urllib/3.4
+ Test-Agent


----------------------------------------------------------------------
Ran 16 tests in 124.879s

FAILED (failures=1, errors=2, skipped=1)
msg227420 - (view) Author: karl (karlcow) * Date: 2014-09-24 08:22
ok let's see

→ ./python.exe -m unittest -v Lib.test.test_urllib2net.OtherNetworkTests.test_custom_headers
test_custom_headers (Lib.test.test_urllib2net.OtherNetworkTests) ... FAIL

======================================================================
FAIL: test_custom_headers (Lib.test.test_urllib2net.OtherNetworkTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/karl/code/cpython/Lib/test/test_urllib2net.py", line 186, in test_custom_headers
    self.assertEqual(request.get_header('User-agent'), 'Test-Agent')
AssertionError: 'Python-urllib/3.4' != 'Test-Agent'
- Python-urllib/3.4
+ Test-Agent


----------------------------------------------------------------------
Ran 1 test in 0.551s

FAILED (failures=1)


→ ./python.exe
Python 3.4.2rc1+ (3.4:8eb4eec8626c+, Sep 23 2014, 21:53:11) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.51)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib.request
>>> url = 'http://127.0.0.1/'
>>> opener = urllib.request.build_opener()
>>> request = urllib.request.Request(url)
>>> request.header_items()
[]
>>> request.headers
{}
>>> request.add_header('User-Agent', 'Test-Agent')
>>> request.headers
{'User-agent': 'Test-Agent'}
>>> request.header_items()
[('User-agent', 'Test-Agent')]
>>> opener.open(request)
<http.client.HTTPResponse object at 0x10c0aedc0>
>>> request.get_header('User-agent'), 'Test-Agent'
('Test-Agent', 'Test-Agent')
>>> request.header_items()
[('User-agent', 'Test-Agent'), ('Host', '127.0.0.1')]
>>> request.headers
{'User-agent': 'Test-Agent'}


OK so far so good.
And my server recorded 
127.0.0.1 - - [24/Sep/2014:17:07:41 +0900] "GET / HTTP/1.1" 200 9897 "-" "Test-Agent"


Let's do it the way, the test has been designed.

→ ./python.exe
Python 3.4.2rc1+ (3.4:8eb4eec8626c+, Sep 23 2014, 21:53:11) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.51)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib.request
>>> url = 'http://127.0.0.1/'
>>> opener = urllib.request.build_opener()
>>> request = urllib.request.Request(url)
>>> request.header_items()
[]
>>> opener.open(request)
<http.client.HTTPResponse object at 0x10e05aa80>
>>> request.header_items()
[('User-agent', 'Python-urllib/3.4'), ('Host', '127.0.0.1')]
>>> request.has_header('User-agent')
True
>>> request.add_header('User-Agent', 'Test-Agent')
>>> opener.open(request)
<http.client.HTTPResponse object at 0x10e05ab50>
>>> request.get_header('User-agent'), 'Test-Agent'
('Python-urllib/3.4', 'Test-Agent')
>>> request.add_header('Foo', 'bar')
>>> request.header_items()
[('User-agent', 'Test-Agent'), ('Host', '127.0.0.1'), ('Foo', 'bar')]
>>> opener.open(request)
<http.client.HTTPResponse object at 0x10e05ad58>
>>> request.header_items()
[('User-agent', 'Test-Agent'), ('Host', '127.0.0.1'), ('Foo', 'bar')]
>>> request.get_header('User-agent'), 'Test-Agent'
('Python-urllib/3.4', 'Test-Agent')
>>> request.headers
{'User-agent': 'Test-Agent', 'Foo': 'bar'}


And the server recorded.

127.0.0.1 - - [24/Sep/2014:17:12:52 +0900] "GET / HTTP/1.1" 200 9897 "-" "Python-urllib/3.4"
127.0.0.1 - - [24/Sep/2014:17:12:52 +0900] "GET / HTTP/1.1" 200 9897 "-" "Python-urllib/3.4"
127.0.0.1 - - [24/Sep/2014:17:14:15 +0900] "GET / HTTP/1.1" 200 9897 "-" "Python-urllib/3.4"

So it seems that User-Agent is immutable once it has been set the first time. Not in  the same dictionary.


>>> request.unredirected_hdrs
{'User-agent': 'Python-urllib/3.4', 'Host': '127.0.0.1'}
msg227421 - (view) Author: karl (karlcow) * Date: 2014-09-24 08:33
Ah! the User-Agent (or anything which is in unredirected_hdrs) will not be updated if it has already been set once. 
https://hg.python.org/cpython/file/064f6baeb6bd/Lib/urllib/request.py#l1154


>>> headers = dict(request.unredirected_hdrs)
>>> headers
{'User-agent': 'Python-urllib/3.4', 'Host': '127.0.0.1'}
>>> request.headers
{'User-agent': 'Test-Agent', 'Foo': 'cool'}
>>> headers.update(dict((k, v) for k, v in request.headers.items() if k not in headers))
>>> headers
{'User-agent': 'Python-urllib/3.4', 'Host': '127.0.0.1', 'Foo': 'cool'}
msg227429 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2014-09-24 09:39
I suspect that you have your interpreters confused. (For e.g, I see 3.4 run against the trunk and the error messages are leading me to believe that 2.7 version in run on 3 code).

Or your local copy is not in right shape.

I tested it on 3.4 and cpython default and everything was OK.

[localhost 3.4]$ ./python.exe -m unittest -v Lib/test/test_urllib2net.py
...
Ran 15 tests in 36.672s

OK
[localhost 3.4]$ hg log -r 3.4
changeset:   92551:bce1594023f9
branch:      3.4
parent:      92548:381d6362c7bc
parent:      92546:ff2cb4dc36e7
user:        Serhiy Storchaka <storchaka@gmail.com>
date:        Tue Sep 23 23:23:41 2014 +0300
description:
Merge heads

[localhost cpython]$ ./python.exe -m unittest -v Lib/test/test_urllib2net.py
...
Ran 15 tests in 32.779s

OK
[localhost cpython]$ hg log -r tip
changeset:   92555:064f6baeb6bd
tag:         tip
user:        Georg Brandl <georg@python.org>
date:        Wed Sep 24 09:08:12 2014 +0200
files:       Python/importlib.h
description:
Update importlib.h frozen bytecode (changed due to commit c0ca9d32aed4).
msg227430 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2014-09-24 09:41
The buildbots are not showing this error too. I suggest you reset your local copy with the pristine one from remote, do a make distclean; ./configure; make and then run the tests.
History
Date User Action Args
2022-04-11 14:58:08adminsetgithub: 66668
2014-09-24 09:41:11orsenthilsetstatus: open -> closed
type: behavior
messages: + msg227430

assignee: orsenthil
resolution: works for me
stage: resolved
2014-09-24 09:39:38orsenthilsetmessages: + msg227429
2014-09-24 08:44:05ned.deilysetnosy: + orsenthil
2014-09-24 08:33:53karlcowsetmessages: + msg227421
2014-09-24 08:22:07karlcowsetmessages: + msg227420
2014-09-24 07:49:29karlcowcreate