classification
Title: urllib.request.URLopener raises different exceptions based on implementation detail
Type: enhancement Stage:
Components: Library (Lib) Versions: Python 3.9, Python 3.8, Python 3.7, Python 3.6, Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Ido Michael, Petter S, remi.lapeyre, vstinner
Priority: normal Keywords:

Created on 2020-05-18 21:30 by Petter S, last changed 2020-05-25 15:04 by Ido Michael.

Messages (6)
msg369291 - (view) Author: Petter S (Petter S) * Date: 2020-05-18 21:30
The following code crashes with TypeError:

URLopener().open("unknown_proxy://test")

the expected behavior is an OSError, which is normally raised for unknown protocols.

This implementation of directly calling a method called "unknown_proxy" is fragile and was recently the subject of a security issue: https://bugs.python.org/issue35907 (CVE-2019-9948)

Would be good to make this more robust.
msg369293 - (view) Author: Petter S (Petter S) * Date: 2020-05-18 21:32
I can fix this, but it does not have high priority.
msg369294 - (view) Author: RĂ©mi Lapeyre (remi.lapeyre) * Date: 2020-05-18 21:37
URLOpener has been deprecated since Python3.3 (https://github.com/python/cpython/blob/master/Lib/urllib/request.py#L1703-L1705) maybe it should just be removed from the codebase?
msg369295 - (view) Author: Petter S (Petter S) * Date: 2020-05-18 21:38
I see. Perhaps that is the resolution then.
msg369300 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-18 22:16
> URLOpener has been deprecated since Python3.3 (https://github.com/python/cpython/blob/master/Lib/urllib/request.py#L1703-L1705) maybe it should just be removed from the codebase?

It emits a DeprecationWarning since at least Python 3.6, so yes, it's now fine to remove the feature. It's also documented as deprecated in:
https://docs.python.org/dev/library/urllib.request.html#urllib.request.URLopener

Maybe we kept it longer than usual because it was common in Python 2.7 to use it, and we tried to keep support for code base compatible with Python 2 and Python 3? I don't know.

I'm fine with removing the class in Python 3.10, but IMO it's too late for Python 3.9 (feature freeze is today).
msg369886 - (view) Author: Ido Michael (Ido Michael) * Date: 2020-05-25 15:04
Does it includes the FancyURLopener? This class is derived from URLopener.

Also what needs to be done?
Remove the URLopener, URLopener_Tests and DummyURLopeners ?
History
Date User Action Args
2020-05-25 15:04:23Ido Michaelsetnosy: + Ido Michael
messages: + msg369886
2020-05-18 22:16:22vstinnersetnosy: + vstinner
messages: + msg369300
2020-05-18 21:38:40Petter Ssetmessages: + msg369295
2020-05-18 21:37:07remi.lapeyresetnosy: + remi.lapeyre
messages: + msg369294
2020-05-18 21:32:25Petter Ssetmessages: + msg369293
2020-05-18 21:31:00Petter Ssettitle: UrlOpener raises different exceptions based on implementation detail -> urllib.request.URLopener raises different exceptions based on implementation detail
2020-05-18 21:30:13Petter Screate