The Unnecessary scheme exists in urlopen() urllib

when people would protect to read file system in HTTP request of urlopen(), they often filter like this against SSRF.

# Vulnerability PoC
import urllib
print urllib.urlopen('local_file:///etc/passwd').read()[:30]
the result is
##
# User Database
# 
# Note t

but if we use a scheme like this, parsing URL cannot parse scheme with urlparse()
this is the parsed result.
ParseResult(scheme='', netloc='', path='local_file:/etc/passwd', params='', query='', fragment='')

def request(url):
    from urllib import urlopen
    from urlparse import urlparse

    result = urlparse(url)
    scheme = result.scheme
    if not scheme:
        return False #raise Exception("Required scheme")
    if scheme == 'file':
        return False #raise Exception("Don't open file")
    res = urlopen(url)
    content = res.read()
    print url, content[:30]
    return True

assert request('file:///etc/passwd') == False
assert request(' file:///etc/passwd') == False
assert request('File:///etc/passwd') == False
assert request('http://www.google.com') != False

if they filter only file://, this mitigation can be bypassed against SSRF. 
with this way.

assert request('local-file:/etc/passwd') == True
ParseResult(scheme='local-file', netloc='', path='/etc/passwd', params='', query='', fragment='')
parseing URL also can be passed.

# Attack scenario
this is the unnecessary URL scheme("local_file").
even if it has filtering, An Attacker can read arbitrary files as bypassing with it.

# Root Cause

URLopener::open in urllib.py
from 203 lin

name = 'open_' + urltype
self.type = urltype
name = name.replace('-', '_') #it can also allows local-file
if not hasattr(self, name): #passed here hasattr(URLopener, 'open_local_file')
    if proxy:
        return self.open_unknown_proxy(proxy, fullurl, data)
    else:
        return self.open_unknown(fullurl, data)
try:
    if data is None:
        return getattr(self, name)(url)
    else:
        return getattr(self, name)(url, data) #return URLopener::open_local_file

it may be just trick because people usually use whitelist (allow only http or https.
Even if but anyone may use blacklist like filtering file://, they will be affected with triggering SSRF

Thanks for your report. I'm having a hard time understanding your English. If I understand you correctly, your bug report is about the open_local_file() method and the surprising fact that urllib supports the local_file schema.

I agree, this looks like an implementation artefact. urllib should not expose the local_file schema. In Python 3 refuses local_file:// (tested with 3.4 to 3.7).

>>> import urllib.request
>>> urllib.request.urlopen('local_file:///etc/passwd').read()[:30]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib64/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/usr/lib64/python3.6/urllib/request.py", line 549, in _open
    'unknown_open', req)
  File "/usr/lib64/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/lib64/python3.6/urllib/request.py", line 1388, in unknown_open
    raise URLError('unknown url type: %s' % type)
urllib.error.URLError: <urlopen error unknown url type: local_file>

Only the Python 2 urllib module is affected. Python 2.7's urllib2 also correctly fails with local_file://

>>> import urllib2
>>> urllib2.urlopen('local_file:///etc/passwd').read()[:30]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/heimes/dev/python/2.7/Lib/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/home/heimes/dev/python/2.7/Lib/urllib2.py", line 429, in open
    response = self._open(req, data)
  File "/home/heimes/dev/python/2.7/Lib/urllib2.py", line 452, in _open
    'unknown_open', req)
  File "/home/heimes/dev/python/2.7/Lib/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/home/heimes/dev/python/2.7/Lib/urllib2.py", line 1266, in unknown_open
    raise URLError('unknown url type: %s' % type)
urllib2.URLError: <urlopen error unknown url type: local_file>

Sorry for my bad English.
Yes, exactly. Only python 2.7 has been affected. not python3.
So I chose only Python2.7 version.

and only urllib, not urllib2.

I'm not a native English speaker either. I wasn't sure if I understood you correctly. Thanks!

I am not also native English speaker. It's OK. Thank you for reading my report

This issue seems to have been assigned CVE-2019-9948 (https://nvd.nist.gov/vuln/detail/CVE-2019-9948) as noted in #11842 (comment)

Christian:

I agree, this looks like an implementation artefact. urllib should not expose the local_file schema. In Python 3 refuses local_file:// (tested with 3.4 to 3.7).

I'm not sure that I understand well the issue. urllib accepts various scheme by design: HTTP, HTTPS, FTP, FILE, etc.

For example, file:// scheme is legit and works as expected. Python 3 example:
---

import urllib.request
req = urllib.request.Request('file:///etc/passwd')
print(f"URL scheme: {req.type}")
fp = urllib.request.urlopen(req)
print(fp.read()[:30])
fp.close()

Output with Python 3:
---
URL scheme: file
b'root:x:0:0:root:/root:/bin/bas'
---

I get a similar output with this Python 2 example:
---

import urllib
req = urllib.urlopen('file:///etc/passwd')
print(req.read()[:30])
req.close()

Christian:

I agree, this looks like an implementation artefact. urllib should not expose the local_file schema.

I understand that Python 2 handles local_file://url as file://url. Ok. But is this a security issue? If you care of security, you ensure that the url scheme is HTTP or HTTPS, not only forbid FILE, no?

I'm asking because of:

Karthikeyan Singaravelan:

This issue seems to have been assigned CVE-2019-9948 (https://nvd.nist.gov/vuln/detail/CVE-2019-9948) ...

The issue is not about whether "file://" schema or not.

It's about the fact that urllib on Python 2 has two schemas that allow local file access. There is the well-known "file://" schema and there is the implementation artifact "local_file://". A careful, security-minded developer knows about the file:// schema and also knows how to block it. But the "local_file://" schema is a surprising side-effect of the implementation.

If you use directly the URLopener class, Python 3 has a similar issue:
---

import urllib.request
req = urllib.request.URLopener().open('local_file:///etc/passwd')
print(req.read()[:30])
req.close()

If developers allow only http:// or https:// as whitelist, it has no problem.
But, If someone blocks only one file://, attacker can bypass it.
This issue may provides attacker with bypassing method as new scheme.

New changeset b15bde8 by Victor Stinner (SH) in branch '2.7':
bpo-35907, CVE-2019-9948: urllib rejects local_file:// scheme (GH-11842)
b15bde8

New changeset 0c2b6a3 by Victor Stinner in branch 'master':
bpo-35907, CVE-2019-9948: urllib rejects local_file:// scheme (GH-13474)
0c2b6a3

New changeset 942c31d by Victor Stinner in branch '2.7':
bpo-35907: Complete test_urllib.test_local_file_open() (GH-13506)
942c31d

New changeset 34bab21 by Victor Stinner in branch '3.7':
bpo-35907, CVE-2019-9948: urllib rejects local_file:// scheme (GH-13474) (GH-13505)
34bab21

New changeset deffee5 by Victor Stinner in branch 'master':
bpo-35907: Clarify the NEWS entry (GH-13523)
deffee5

New changeset 1c9debd by Victor Stinner in branch 'master':
bpo-35907: Fix typo in the NEWS entry (GH-13559)
1c9debd

New changeset d9d1045 by Victor Stinner in branch '2.7':
bpo-35907: Clarify the NEWS entry (GH-13557)
d9d1045

New changeset cee4ac8 by Victor Stinner in branch '3.7':
bpo-35907: Clarify the NEWS entry (GH-13558)
cee4ac8

New changeset 4f06dae by Ned Deily (Victor Stinner) in branch '3.6':
bpo-35907, CVE-2019-9948: urllib rejects local_file:// scheme (GH-13513)
4f06dae

New changeset 4fe82a8 by larryhastings (Victor Stinner) in branch '3.5':
bpo-35907, CVE-2019-9948: urllib rejects local_file:// scheme (GH-13474) (GH-13505) (bpo-13510)
4fe82a8

We should whitelist the protocols. The current solution with getattr is really fragile.

For example, this crashes with a TypeError: URLopener().open("unknown_proxy://test")

We should whitelist the protocols. The current solution with getattr is really fragile. For example, this crashes with a TypeError: URLopener().open("unknown_proxy://test")

Would you mind to elaborate why do you consider that the solution is incomplete? Your issue doesn't show that Python is vulnerable. TypeError *is* the expected behavior.

Would you prefer another error message? If yes, please open a seperated issue.

The solution is incomplete because it fixes just this single security issue, not the inherent fragility of this file.

If, in the future someone happens to add another method starting with open to this class, we are at risk of having the same problem again.

As for the error message, it is of course a minor issue, but I don't think it is expected that "unknown_proxy://" and "something_else://" raise different exceptions, right?

The solution is incomplete because it fixes just this single security issue, not the inherent fragility of this file.

If you want to propose a change to make the file "less fragile", please open a *new* separated issue.

The issue is about an exact vulnerability, the "local_file://" scheme, which has been fixed. I close again the issue.

OK: https://bugs.python.org/issue40673