classification
Title: urllib2 ignores opener configuration under certain circumstances
Type: behavior Stage:
Components: Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder: urllib.parse.urlopen shouldn't ignore installed opener when called with any SSL argument
View: 18543
Assigned To: Nosy List: Arfrever, David Ford (FirefighterBlu3), benjamin.peterson, berker.peksag, crazyjurich, eht16, martin.panter, orsenthil
Priority: normal Keywords:

Created on 2015-01-04 21:13 by crazyjurich, last changed 2016-01-01 08:14 by martin.panter. This issue is now closed.

Files
File name Uploaded Description Edit
test.py crazyjurich, 2015-01-05 20:10
urllib_ssl_auth_test.py eht16, 2015-01-21 20:21 Testcase for HTTP basic auth with disabled SSL cert verification
Messages (7)
msg233427 - (view) Author: Jurijs Vorotilovs (crazyjurich) Date: 2015-01-04 21:13
Python 2.7.9 has a bug in urllib2.py:urlopen().
It creates HTTPSHandler instances by its own when it should not.
One may have assigned custom openers with subclassed HTTPSHandler or HTTPSHandler instance with debug enabled or etc.
msg233441 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2015-01-05 06:52
Thanks for the report. Could you provide an example to reproduce the issue you described?
msg233477 - (view) Author: Jurijs Vorotilovs (crazyjurich) Date: 2015-01-05 20:10
Attached a script demonstrating two failing cases
msg233484 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2015-01-05 23:36
I guess there needs to be some generic way to pass ssl information to handlers.
msg234444 - (view) Author: Enrico Tröger (eht16) * Date: 2015-01-21 20:21
I got the same error suddenly with Python 2.7.9.

I think this is quite unfortunate because it somewhat breaks existing behaviour, especially that SSL certificate verification is enabled by default.
Don't get me wrong, this is the right thing in general and it is important. Still, adding this feature in a 2.7 patch level release and enabling it by default feels quite hard.
I guess this will break many scripts and applications which rely on non-verification of SSL certs (which is bad but it was the exisiting behaviour).

Anyway, attached is my use case where I use a HTTPS request coupled with HTTP basic authentication and disabled SSL cert verification.
As described above, passing a context to urlopen() will override previously configured handlers, unfortunately.

In the attached script there is also a workaround which works for me by not using urlopen() but instead calling opener.open() manually after adding the necessary handlers myself.
Not nice but works for the moment.
msg246129 - (view) Author: David Ford (FirefighterBlu3) (David Ford (FirefighterBlu3)) * Date: 2015-07-03 04:37
I've made a patch for 3.4 that addresses this issue. See issue 18543, latest patch, and test file
msg257281 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-01-01 08:14
I think these are essentially the same problem. It defeats any custom installed opener, not just custom HTTPS handlers.
History
Date User Action Args
2016-01-01 08:14:34martin.pantersetstatus: open -> closed

nosy: + martin.panter
messages: + msg257281

superseder: urllib.parse.urlopen shouldn't ignore installed opener when called with any SSL argument
resolution: duplicate
2015-07-03 04:37:58David Ford (FirefighterBlu3)setnosy: + David Ford (FirefighterBlu3)
messages: + msg246129
2015-02-13 01:26:58demian.brechtsetnosy: - demian.brecht
2015-01-21 20:21:47eht16setfiles: + urllib_ssl_auth_test.py
nosy: + eht16
messages: + msg234444

2015-01-11 00:48:36Arfreversetnosy: + Arfrever
2015-01-09 17:16:34demian.brechtsetnosy: + demian.brecht
2015-01-05 23:36:20benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg233484
2015-01-05 20:10:44crazyjurichsetfiles: + test.py

messages: + msg233477
2015-01-05 06:52:09berker.peksagsetnosy: + berker.peksag, orsenthil
messages: + msg233441
2015-01-04 21:13:56crazyjurichcreate