classification
Title: os.path.abspath with unicode argument should call os.getcwdu
Type: behavior Stage: resolved
Components: Unicode Versions: Python 2.7, Python 2.6
process
Status: closed Resolution: fixed
Dependencies: Superseder: test_unicode_file fails with non-ascii path
View: 7669
Assigned To: ezio.melotti Nosy List: aimacintyre, amaury.forgeotdarc, brian.curtin, ezio.melotti, flox, ronaldoussoren, saturn_mimas
Priority: normal Keywords: buildbot, needs review, patch

Created on 2008-07-22 14:32 by saturn_mimas, last changed 2010-02-21 11:26 by ezio.melotti. This issue is now closed.

Files
File name Uploaded Description Edit
issue3426.diff ezio.melotti, 2010-01-13 00:52 Patch that uses os.getcwdu() instead of os.getcwd() when the arg of abspath is unicode + unittests + helper context manager
issue3426-2.diff ezio.melotti, 2010-01-14 04:57 Added fix for ntpath, refactored the tests
issue3426-3.diff ezio.melotti, 2010-01-16 03:52 Fix for posixpath, ntpath, macpath, and os2emxpath
issue3426_any_cwd.diff flox, 2010-02-20 17:07 Patch, apply to trunk
Messages (16)
msg70148 - (view) Author: Christian Häggström (saturn_mimas) Date: 2008-07-22 14:32
If current working directory contains non-ascii characters, calling
os.path.abspath(u".") will result in an error. I expect it to call the
underlying os.getcwdu() in this case.

>>> import os
>>> os.path.abspath(u".")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File
"/home/packages/python-2.5.1/x86-linux/lib/python2.5/posixpath.py", line
403, in abspath
    path = join(os.getcwd(), path)
  File
"/home/packages/python-2.5.1/x86-linux/lib/python2.5/posixpath.py", line
65, in join
    path += '/' + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 29:
ordinal not in range(128)

It works if I do it manually, using os.getcwdu():

>>> os.path.join(os.getcwdu(), u".")
u'/disk1/chn_local/work/test/sk\xe4rg\xe5rds\xf6-latin1/.'
msg70151 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-07-22 15:52
Well, os.path.supports_unicode_filenames is False on posix platforms.
So os.path.abspath is not supposed to work with unicode values.

I was about to close the issue, but python 3.0 also defines
supports_unicode_filenames to False! In addition, os.getcwd() fails when
the current dir has non-ascii characters. Should we drop its
implementation, and use os.getcwdu instead?
msg90101 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2009-07-04 03:06
This seems to work fine with Py 3.0 and 3.1 on Linux, it still fails
with Py 2.6 and 2.7.
msg97645 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2010-01-12 18:30
This also caused the failure in #7669.
I think that the functions in os.path are supposed to return unicode when they get unicode, so I agree that os.getcwdu should be used instead.
I'm not sure about os.path.supports_unicode_filenames though.
msg97676 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2010-01-13 00:52
Here is a patch that uses os.getcwdu() instead of os.getcwd() when the arg of abspath is unicode (with tests).

It also include an helper context manager in test_support used to create temp dirs and set them as cwd (this will be committed separately) and two helper methods (assertUnicode and assertStr) that will probably be useful when I (or someone else) will add more tests with unicode strings for the other functions.
msg97680 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2010-01-13 01:53
You could use assertIsInstance(s, unicode, '%r is not unicode' % s) in the tests instead of your assertTrue.

I think the rest of it looks good. Works for me.
msg97752 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2010-01-14 04:57
I added the fix on ntpath and more tests. I also refactored the tests I added for posixpath.
I don't know what is the situation of os2emxpath and macpath and if they should be fixed too. The code for abspath is currently the same on all 4 the modules but I don't see any easy way to avoid the duplication and move it to genericpath because they call the module-dependent join and normpath.
msg97799 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2010-01-15 04:09
assertStr and assertUnicode don't exist in test_ntpath so the tests fail on Windows. I copied/pasted the functions over from test_posixpath just to see and test_ntpath passes. Other than that, it looks good to me.
msg97801 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2010-01-15 07:02
I'll fix the patch. I added Ronald and Andrew to see if they have any opinion about macpath and os2emxpath. The main idea is that abspath should always return unicode if its arg is unicode or str otherwise.
msg97806 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2010-01-15 11:53
abspath is basically dead code in macpath, the module is used to manipulate classic MacOS9-style paths and is no longer used as os.path on any supported platform (it used to be os.path on MacOS9).

BTW. the module itself is not dead, OSX still uses OS9-style paths in a number of older APIs.
msg97861 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2010-01-16 03:52
For consistency I updated all 4 the modules. If the tests pass on both Windows and Mac the patch should be ready to go in.
msg98044 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2010-01-19 03:22
Passes on Windows, Mac, and Linux.
msg98650 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2010-02-01 07:46
Small note:
 - the unit tests could use assertIsInstance(..., str), assertIsInstance(..., unicode) instead of the custom methods.
 - some "assertTrue" may be replaced by "assertEqual"
msg98665 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2010-02-01 14:59
The patch intentionally doesn't use assertIsInstance because that method doesn't exist in 2.6.
msg99623 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2010-02-20 17:07
The attached patch proposes a decorator which can be used to strengthen any test which is affected by the CWD. (and fixes for test_(mac|nt|posix)path too)
msg99651 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2010-02-21 11:25
Fixed in r78247 (trunk) and r78248 (release26-maint) (plus a fix in r78272 and r78279 to avoid test failures when the filesystem encoding is ascii).
I didn't use the any_cwd decorator -- I might consider it in future if it turns out that there are more tests like these.
History
Date User Action Args
2010-02-21 11:26:27ezio.melottisetstatus: open -> closed
stage: patch review -> resolved
2010-02-21 11:25:34ezio.melottisetresolution: fixed
dependencies: - Add a context manager to change cwd in test.test_support and run the test suite in a temp dir.
messages: + msg99651
2010-02-20 17:07:31floxsetfiles: + issue3426_any_cwd.diff

messages: + msg99623
2010-02-01 14:59:50brian.curtinsetmessages: + msg98665
2010-02-01 07:46:57floxsetmessages: + msg98650
2010-01-19 03:22:15brian.curtinsetmessages: + msg98044
2010-01-16 05:43:38ezio.melottisetdependencies: + Add a context manager to change cwd in test.test_support and run the test suite in a temp dir.
2010-01-16 03:52:49ezio.melottisetfiles: + issue3426-3.diff

messages: + msg97861
2010-01-15 16:35:57floxsetkeywords: + buildbot
2010-01-15 11:53:00ronaldoussorensetmessages: + msg97806
2010-01-15 07:02:25ezio.melottisetnosy: + ronaldoussoren, aimacintyre
messages: + msg97801
2010-01-15 04:09:44brian.curtinsetmessages: + msg97799
2010-01-14 04:58:01ezio.melottisetfiles: + issue3426-2.diff

messages: + msg97752
2010-01-13 01:53:17brian.curtinsetnosy: + brian.curtin
messages: + msg97680
2010-01-13 00:53:32ezio.melottisetnosy: + flox
2010-01-13 00:52:40ezio.melottisetfiles: + issue3426.diff

nosy: - flox
messages: + msg97676

keywords: + patch, needs review
stage: test needed -> patch review
2010-01-12 18:43:14floxsetnosy: + flox
2010-01-12 18:30:57ezio.melottisetassignee: ezio.melotti
superseder: test_unicode_file fails with non-ascii path
messages: + msg97645
stage: test needed
2009-07-04 03:06:28ezio.melottisetpriority: normal
versions: + Python 2.6, Python 2.7, - Python 2.5, Python 2.4, Python 3.0
nosy: + ezio.melotti

messages: + msg90101
2008-07-22 15:52:32amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg70151
versions: + Python 3.0
2008-07-22 14:32:13saturn_mimascreate