Title: Disallow relative files paths in urllib*.open()
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.2, Python 3.3, Python 2.7
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: orsenthil Nosy List: albert, amaury.forgeotdarc, martin.panter, orsenthil, python-dev, rbcollins
Priority: normal Keywords:

Created on 2009-08-03 13:33 by albert, last changed 2015-08-02 22:31 by rbcollins. This issue is now closed.

File name Uploaded Description Edit albert, 2009-08-03 13:33 patched function + some doc/test
Messages (9)
msg91222 - (view) Author: albert Mietus (albert) Date: 2009-08-03 13:33
The functions urlparse.url{,un}split() and urllib{,2}.open() do not work 
together for relative, local files, due a bug in urlunsplit.

Given a file f='./rel/path/to/file.html' it can be open directly by, but not in urllib2! as the later needs a scheme.
We can create a sound url with spilt/unspilt and a default scheme:
f2=urlparse.urlunsplit(urlparse.urlsplit(f,'file')); which works most 
cases, HOWEVER a bogus netloc is added for relative filepaths.

If have isolated this  "buggy" function, added some local testcode and 
made patch/workaround in my file '' Which is included. Hope 
this will contribute to a real patch.

--Groetjes, Albert

ALbert Mietus
                                                Don't send spam mail!
Mijn missie:      product, proces & imago.
Mijn leven in het kort:
msg91402 - (view) Author: albert Mietus (albert) Date: 2009-08-07 12:41
There was a bug in the workaround:

    if not ( scheme == 'file' and not netloc and url[0] != '/'):

The {{{and url[0] != '/'}}} was missing (above is corrected)

The effect: split/unspilt file:///path resulted in file:/path
msg100175 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2010-02-26 21:17
The bug here seems to me that urllib.urlopen() should not allow a relative file path like the one specified. f='./rel/path/to/file.html
urllib2's behavior seems proper that it is raising an Exception.

According to the RFCs the local files are to be acceessed by:

Both are absolute paths to the file where in the second one localhost is omitted.

Let me see if urllib's urlopen be made a little stricter.
msg151715 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-01-21 03:43
New changeset f6008e936fbc by Senthil Kumaran in branch '2.7':
Fix Issue6631 - Disallow relative files paths in urllib*.open()
msg151716 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-01-21 03:55
New changeset 4366c0df2c73 by Senthil Kumaran in branch '2.7':
NEWS entry for Issue6631

New changeset 514994d7a9f2 by Senthil Kumaran in branch '3.2':
Fix  Issue6631 - Disallow relative file paths in urllib urlopen
msg151726 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2012-01-21 08:35
Sorry, why was this change backported?
Does this fix a specific issue in 2.7 or 3.2?
On the contrary, it seems to me that code which (incorrectly) used urllib.urlopen() to allow both urls and local files will suddenly break.
msg151727 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2012-01-21 09:19
Actually, I saw this as a bug with urllib.urlopen and urllib2 had
exhibited proper behaviour previously. Now, both behaviour will be
consistent now.

But, you are right that an *incorrect* usage of urllib.urlopen would
break in 2.7.2. 

If we need to be lenient on that incorrect usage, then this change can
be there in 3.x series, because of urllib.request.urlopen would be
interface which users will be using and it can be reverted from 2.7.

Personally, I am +/- 0 on reverting this in 2.7. Initially, I saw this
as a bug, but later when I added tests for ValueError and checkedin,
I realized that it can break some incorrect usages, as you say.
msg241084 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-04-15 05:58
I’m confused what the intention of this bug is. The normal urllib.request.urlopen() function (or equivalent) still allows file URLs with relative paths, in various Python versions I tried, ranging from 2.6 to 3.5:

>>> import urllib.request
>>> urllib.request.urlopen("file:README")
<addinfourl at 140061019639088 whose fp = <_io.BufferedReader name='README'>>

Passing a relative URL without a scheme seems to have never been supported, for a different reason:

>>> urllib.request.urlopen("README")
Traceback (most recent call last):
[. . .]
  File "/home/proj/python/cpython/Lib/urllib/", line 321, in _parse
    raise ValueError("unknown url type: %r" % self.full_url)
ValueError: unknown url type: 'README'

Looking closer at the changes made here, they only seem to affect the urllib.request.URLopener class, which is listed as deprecated in 3.3 and never used by urlopen() as far as I know.

Personally, I would prefer to keep allowing relative paths in “file:” scheme URLs in urlopen(). This is useful e.g. if you don’t want to put the full working directory in a URL on the command line or whatever.

It is inconsistent that urljoin(), urlunsplit(), etc don’t support relative paths, but I would actually prefer that such support be added. (I think my patch for Issue 22852 would probably do the trick.) Currently, urljoin() reinterprets the path as absolute:

>>> urljoin("file:subdir/index.html", "link.html")
msg247907 - (view) Author: Robert Collins (rbcollins) * (Python committer) Date: 2015-08-02 22:31
test_relativelocalfile is still in place in the urllib tests, so its affecting urlopen to this point.

So I think the bug is fixed at least to the extent of the original report. I'm going to close this.
Date User Action Args
2015-08-02 22:31:35rbcollinssetstatus: open -> closed

nosy: + rbcollins
messages: + msg247907

stage: commit review -> resolved
2015-04-15 05:58:14martin.pantersetmessages: + msg241084
2014-07-13 01:21:58martin.pantersetnosy: + martin.panter
2012-01-21 09:19:51orsenthilsetstatus: pending -> open

messages: + msg151727
2012-01-21 08:35:31amaury.forgeotdarcsetstatus: closed -> pending

nosy: + amaury.forgeotdarc
messages: + msg151726

stage: resolved -> commit review
2012-01-21 03:57:05orsenthilsetstatus: open -> closed
type: performance -> behavior
stage: resolved
resolution: fixed
versions: + Python 2.7, Python 3.2, Python 3.3
2012-01-21 03:55:58python-devsetmessages: + msg151716
2012-01-21 03:43:26python-devsetnosy: + python-dev
messages: + msg151715
2012-01-21 03:42:49orsenthilsettitle: urlparse.urlunsplit() can't handle relative files (for urllib*.open() -> Disallow relative files paths in urllib*.open()
2010-02-26 21:17:27orsenthilsetassignee: orsenthil

messages: + msg100175
nosy: + orsenthil
2009-08-07 12:41:03albertsetmessages: + msg91402
2009-08-03 13:33:02albertcreate