classification
Title: urllib2.urlopen throws HTTP Error in python 2.7.2, 2.7.3, but not in python 2.7
Type: behavior Stage:
Components: Versions: Python 2.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: fertig, orsenthil, pitrou, r.david.murray
Priority: normal Keywords:

Created on 2012-10-02 15:15 by fertig, last changed 2012-10-05 13:31 by pitrou. This issue is now closed.

Messages (10)
msg171811 - (view) Author: Christian Fertig (fertig) Date: 2012-10-02 15:15
wespe:/tmp/python2.7 # cat /etc/SuSE-release
openSUSE 12.1 (x86_64)
VERSION = 12.1
CODENAME = Asparagus
wespe:/tmp/python2.7 # rpm -q python
python-2.7.2-7.17.1.x86_64


wespe:/tmp/python2.7 # python
Python 2.7.2 (default, Aug 19 2011, 20:41:43) [GCC] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib2; print urllib2.urlopen('https://172.23.6.222/') # not working
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "urllib2.py", line 397, in open
    response = meth(req, response)
  File "urllib2.py", line 510, in http_response
    'http', request, response, code, msg, hdrs)
  File "urllib2.py", line 435, in error
    return self._call_chain(*args)
  File "urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "urllib2.py", line 518, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 405: Method Not Allowed

>>> import urllib; print urllib.urlopen('https://172.23.6.222/') # working
<addinfourl at 10613608 whose fp = <socket._fileobject object at 0x7f7c20a41d50>>


Other machine:

hornisse:/usr/lib64 # cat /etc/SuSE-release
openSUSE 11.4 (x86_64)
VERSION = 11.4
CODENAME = Celadon
hornisse:/usr/lib64 # rpm -q python
python-2.7-9.40.1.x86_64

hornisse:/usr/lib64 # python
Python 2.7 (r27:82500, Aug 07 2010, 16:54:59) [GCC] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib2; print urllib2.urlopen('https://172.23.6.222/') # working
<addinfourl at 7832424 whose fp = <socket._fileobject object at 0x756a50>>
>>> import urllib; print urllib.urlopen('https://172.23.6.222/') # working
<addinfourl at 10130856 whose fp = <socket._fileobject object at 0x756850>>



Sometimes the HTTP Error is a 400 Bad Request, but 99% of the time a 405. I've testet openSuSE 12.2 with python 2.7.3 too, not working.

I don't know, what to provide more on information, please contact me, if you need more.
msg171819 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-10-02 17:21
That's a non-public IP, so you'll have to provide a trace of the http traffic.  It is likely the error is correct, and is reported only post 2.7.0 because a bug was fixed.
msg172058 - (view) Author: Christian Fertig (fertig) Date: 2012-10-05 07:53
ok, here's a run with debuglevel for the headers:

python 2.7 on openSuSE 11.4:

fertig@hornisse:/home/fertig > python
Python 2.7 (r27:82500, Aug 07 2010, 16:54:59) [GCC] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib2
>>> request = urllib2.Request('https://172.23.6.222')
>>> opener = urllib2.build_opener(urllib2.HTTPSHandler(debuglevel=111))
>>> feeddata = opener.open(request).read()
send: 'GET / HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: 172.23.6.222\r\nConnection: close\r\nUser-Agent: Python-urllib/2.7\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Date: Fri, 05 Oct 2012 09:46:33 GMT
header: Server: Virata-EmWeb/R6_2_0
header: Connection: close
header: Transfer-Encoding: chunked
header: Content-Type: text/html
header: Expires: Fri, 05 Oct 2012 09:46:33 GMT
header: Last-Modified: Fri, 05 Oct 2012 09:46:33 GMT
header: Cache-Control: no-cache
>>>

python 2.7.2 on openSuSE 12.1:

fertig@wespe:/home/fertig > python
Python 2.7.2 (default, Aug 19 2011, 20:41:43) [GCC] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib2
>>> request = urllib2.Request('https://172.23.6.222')
>>> opener = urllib2.build_opener(urllib2.HTTPSHandler(debuglevel=111))
>>> feeddata = opener.open(request).read()
<urllib2.Request instance at 0x90d878>
send: 'GET / HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: 172.23.6.222\r\nConnection: close\r\nUser-Agent: Python-urllib/2.7\r\n\r\n'
reply: 'HTTP/1.1 405 Method Not Allowed\r\n'
header: Date: Fri, 05 Oct 2012 09:44:11 GMT
header: Server: Virata-EmWeb/R6_2_0
header: Connection: close
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.7/urllib2.py", line 401, in open
    response = meth(req, response)
  File "/usr/lib64/python2.7/urllib2.py", line 514, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib64/python2.7/urllib2.py", line 439, in error
    return self._call_chain(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 372, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 522, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 405: Method Not Allowed
>>>
msg172059 - (view) Author: Christian Fertig (fertig) Date: 2012-10-05 07:57
the IP is private and even for testing I can't put it in a public IP range, as the device is an IP Phone with a custom firmware (Siemens optiPoint 420).
msg172078 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-10-05 13:03
It looks like the data input is the same in both cases. (Although why does your second example show '<urllib2.Request instance at 0x90d878>'?  Does urllib2 really output that as part of the debug output?)  It is hard to see what could be causing the divergence in response here.  Is it possible for there to be something in the SSL setup that triggers a different response from your server?

Any chance you could try Python 3.2 or 3.3 just to see if there is a difference there?
msg172080 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-10-05 13:13
By using "openssl s_client -connect 172.23.6.222:443" you can open a telnet-like session (but using SSL) to your server, and try to type/paste an HTTP request there. That would help check whether it's a Python problem, or something else due to OpenSSL changes.
msg172081 - (view) Author: Christian Fertig (fertig) Date: 2012-10-05 13:18
python3 example, not working on both machines (3.2.1 on openSuSE 12.1 and 3.1.3 on openSuSE 11.4)

fertig@wespe:/home/fertig > python3
Error in sitecustomize; set PYTHONVERBOSE for traceback:
SyntaxError: invalid syntax (sitecustomize.py, line 89)
Python 3.2.1 (default, Jul 18 2011, 16:24:40) [GCC] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import http.client
>>> import urllib.request
>>> http.client.HTTPConnection.debuglevel = 1
>>> request = urllib.request.urlopen('https://172.23.6.222')
send: b'GET / HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: 172.23.6.222\r\nConnection: close\r\nUser-Agent: Python-urllib/3.2\r\n\r\n'
reply: 'HTTP/1.1 400 Bad Request\r\n'
header: Date header: Server Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.2/urllib/request.py", line 138, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib64/python3.2/urllib/request.py", line 375, in open
    response = meth(req, response)
  File "/usr/lib64/python3.2/urllib/request.py", line 487, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib64/python3.2/urllib/request.py", line 413, in error
    return self._call_chain(*args)
  File "/usr/lib64/python3.2/urllib/request.py", line 347, in _call_chain
    result = func(*args)
  File "/usr/lib64/python3.2/urllib/request.py", line 495, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request
>>> request = urllib.request.urlopen('https://172.23.6.222')
send: b'GET / HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: 172.23.6.222\r\nConnection: close\r\nUser-Agent: Python-urllib/3.2\r\n\r\n'
reply: 'HTTP/1.1 405 Method Not Allowed\r\n'
header: Date header: Server header: Connection Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.2/urllib/request.py", line 138, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib64/python3.2/urllib/request.py", line 375, in open
    response = meth(req, response)
  File "/usr/lib64/python3.2/urllib/request.py", line 487, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib64/python3.2/urllib/request.py", line 413, in error
    return self._call_chain(*args)
  File "/usr/lib64/python3.2/urllib/request.py", line 347, in _call_chain
    result = func(*args)
  File "/usr/lib64/python3.2/urllib/request.py", line 495, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 405: Method Not Allowed
msg172082 - (view) Author: Christian Fertig (fertig) Date: 2012-10-05 13:23
ok actually it isn't working either with curl or wget. So probably this is not a python problem, it seems more something in the underlying libs:

Output from curl on the machine, where python 2.7 is working:

fertig@hornisse:/home/fertig > curl --insecure 'https://172.23.6.222/' --verbose
* About to connect() to 172.23.6.222 port 443 (#0)
*   Trying 172.23.6.222... connected
* Connected to 172.23.6.222 (172.23.6.222) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs/
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using AES256-SHA
* Server certificate:
*        subject: C=DE; L=Munich; O=Siemens Enterprise Communications; OU=Enterprise Systems; CN=optiPoint - WBM
*        start date: 2007-04-11 12:26:43 GMT
*        expire date: 2022-04-10 12:26:43 GMT
*        common name: optiPoint - WBM (does not match '172.23.6.222')
*        issuer: C=DE; L=Munich; O=Siemens AG; OU=Com Enterprise Systems; CN=Siemens Com ESY HD Security Office; emailAddress=hipath_security_office@siemens.com
*        SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
> GET / HTTP/1.1
> User-Agent: curl/7.21.2 (x86_64-unknown-linux-gnu) libcurl/7.21.2 OpenSSL/1.0.0c zlib/1.2.5 libidn/1.15 libssh2/1.2.7
> Host: 172.23.6.222
> Accept: */*
>
< HTTP/1.1 405 Method Not Allowed
< Date: Fri, 05 Oct 2012 11:03:02 GMT
< Server: Virata-EmWeb/R6_2_0
< Connection: close
<
405 Method Not Allowed
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
fertig@hornisse:/home/fertig >


No idea anymore :/ But I fear you can close this ticket...
msg172083 - (view) Author: Christian Fertig (fertig) Date: 2012-10-05 13:28
for completeness: the openssl output from the machine, where python 2.7 is working:

fertig@hornisse:/home/fertig > openssl s_client -connect 172.23.6.222:443
CONNECTED(00000003)
depth=1 C = DE, L = Munich, O = Siemens AG, OU = Com Enterprise Systems, CN = Siemens Com ESY HD Security Office, emailAddress = hipath_security_office@siemens.com
verify error:num=19:self signed certificate in certificate chain
verify return:0
---
Certificate chain
 0 s:/C=DE/L=Munich/O=Siemens Enterprise Communications/OU=Enterprise Systems/CN=optiPoint - WBM
   i:/C=DE/L=Munich/O=Siemens AG/OU=Com Enterprise Systems/CN=Siemens Com ESY HD Security Office/emailAddress=hipath_security_office@siemens.com
 1 s:/C=DE/L=Munich/O=Siemens AG/OU=Com Enterprise Systems/CN=Siemens Com ESY HD Security Office/emailAddress=hipath_security_office@siemens.com
   i:/C=DE/L=Munich/O=Siemens AG/OU=Com Enterprise Systems/CN=Siemens Com ESY HD Security Office/emailAddress=hipath_security_office@siemens.com
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIEwDCCA6igAwIBAgIBFjANBgkqhkiG9w0BAQUFADCBtDELMAkGA1UEBhMCREUx
[..]
dEYuDjj21U+xronZ0FYVieIP1jk=
-----END CERTIFICATE-----
subject=/C=DE/L=Munich/O=Siemens Enterprise Communications/OU=Enterprise Systems/CN=optiPoint - WBM
issuer=/C=DE/L=Munich/O=Siemens AG/OU=Com Enterprise Systems/CN=Siemens Com ESY HD Security Office/emailAddress=hipath_security_office@siemens.com
---
No client certificate CA names sent
---
SSL handshake has read 2748 bytes and written 407 bytes
---
New, TLSv1/SSLv3, Cipher is AES256-SHA
Server public key is 1024 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : AES256-SHA
    Session-ID: 506EDFB9B197A00F67A43F34B3313B660DBC64FF7353112836F813723B30071C
    Session-ID-ctx:
    Master-Key: 6928F9EF19DB7DBEEEA6D09392EF2F848613E40C1B6CE6226C342AB3D211AA1A8E04EF27BDEE4CD03AB4F6AD7317EBB3
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1349443513
    Timeout   : 300 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
---
GET / HTTP/1.1

HTTP/1.1 400 Bad Request
Date: Fri, 05 Oct 2012 15:25:21 GMT
Server: Virata-EmWeb/R6_2_0
Connection: close

400 Bad Request
msg172084 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-10-05 13:31
Ok, closing then. Thanks for the debugging, and good luck :)
History
Date User Action Args
2012-10-05 13:31:22pitrousetstatus: open -> closed
resolution: not a bug
messages: + msg172084
2012-10-05 13:28:30fertigsetmessages: + msg172083
2012-10-05 13:23:33fertigsetmessages: + msg172082
2012-10-05 13:18:24fertigsetmessages: + msg172081
2012-10-05 13:13:54pitrousetmessages: + msg172080
2012-10-05 13:03:59r.david.murraysetnosy: + orsenthil, pitrou
messages: + msg172078
2012-10-05 07:57:52fertigsetmessages: + msg172059
2012-10-05 07:53:43fertigsetmessages: + msg172058
2012-10-02 17:21:16r.david.murraysetnosy: + r.david.murray
messages: + msg171819
2012-10-02 15:15:08fertigcreate