This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: test_os failure on OS X, probably related to PEP 383
Type: behavior Stage:
Components: Versions: Python 3.1
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: mark.dickinson Nosy List: Matthias Braun, loewis, mark.dickinson, r.david.murray
Priority: critical Keywords: patch

Created on 2009-05-05 18:01 by mark.dickinson, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
issue5944.patch mark.dickinson, 2009-05-05 21:00
issue5944_v2.patch mark.dickinson, 2009-05-05 21:28
Messages (11)
msg87264 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2009-05-05 18:01
I'm getting what looks like a PEP 383-related failure in test_os, on OS 
X 10.5.6.  Here's the tail end of running Lib/test/test_os.py, on a
standard non-debug build of py3k.

test_setuid (__main__.PosixUidGidTests) ... ok
test_listdir (__main__.Pep383Tests) ... FAIL
test_open (__main__.Pep383Tests) ... ok
test_stat (__main__.Pep383Tests) ... ok

======================================================================
FAIL: test_listdir (__main__.Pep383Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Lib/test/test_os.py", line 725, in test_listdir
    self.assertEquals(found, expected)
AssertionError: Items in the first set but not the second:
'fooöbar'
'foo%F6bar'
Items in the second set but not the first:
'foo\udcf6bar'
'fooöbar'

----------------------------------------------------------------------
Ran 60 tests in 0.079s

FAILED (failures=1)
Traceback (most recent call last):
  File "Lib/test/test_os.py", line 758, in <module>
    test_main()
  File "Lib/test/test_os.py", line 754, in test_main
    Pep383Tests
  File "/Users/dickinsm/svn/python/py3k/Lib/test/support.py", line 878, 
in run_unittest
    _run_suite(suite)
  File "/Users/dickinsm/svn/python/py3k/Lib/test/support.py", line 861, 
in _run_suite
    raise TestFailed(err)
test.support.TestFailed: Traceback (most recent call last):
  File "Lib/test/test_os.py", line 725, in test_listdir
    self.assertEquals(found, expected)
AssertionError: Items in the first set but not the second:
'fooöbar'
'foo%F6bar'
Items in the second set but not the first:
'foo\udcf6bar'
'fooöbar'
msg87265 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2009-05-05 18:09
One other piece of information that may be relevant:

Macintosh-4:py3k dickinsm$ echo $LANG
en_US.UTF-8

If I unset the LANG environment variable then I get a different failure  
(see output below).

I'll investigate this further and see if I can figure out what's going 
wrong.



test_setreuid (__main__.PosixUidGidTests) ... ok
test_setuid (__main__.PosixUidGidTests) ... ok
test_listdir (__main__.Pep383Tests) ... FAIL
test_open (__main__.Pep383Tests) ... ok
test_stat (__main__.Pep383Tests) ... ok

======================================================================
FAIL: test_listdir (__main__.Pep383Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Lib/test/test_os.py", line 758, in <module>
    test_main()
  File "Lib/test/test_os.py", line 754, in test_main
    Pep383Tests
  File "/Users/dickinsm/svn/python/py3k/Lib/test/support.py", line 878, 
in run_unittest
    _run_suite(suite)
  File "/Users/dickinsm/svn/python/py3k/Lib/test/support.py", line 853, 
in _run_suite
    result = runner.run(suite)
  File "/Users/dickinsm/svn/python/py3k/Lib/unittest.py", line 1455, in 
run
    result.printErrors()
  File "/Users/dickinsm/svn/python/py3k/Lib/unittest.py", line 1424, in 
printErrors
    self.printErrorList('FAIL', self.failures)
  File "/Users/dickinsm/svn/python/py3k/Lib/unittest.py", line 1431, in 
printErrorList
    self.stream.writeln("%s" % err)
  File "/Users/dickinsm/svn/python/py3k/Lib/unittest.py", line 1340, in 
writeln
    self.write(arg)
UnicodeEncodeError: 'ascii' codec can't encode character '\u0308' in 
position 194: ordinal not in range(128)
msg87272 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2009-05-05 19:06
OS/X needs to be special cased in the tests, since it transforms the
invalid filenames on write.  And if I remember correctly older versions
do something different...throw an error, maybe?

The case where the letter looks the same (at least in my email) but is
apparently different may require some thought, too.
msg87273 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2009-05-05 19:11
> The case where the letter looks the same (at least in my email) but is
> apparently different may require some thought, too.

In case it's useful, when I do a list(map(ord, ...)) on those two 
lookalike strings I get (where 'expected' comes from the explicit .decode, 
while 'found' comes from listdir):

expected: [102, 111, 111, 246, 98, 97, 114]
found:    [102, 111, 111, 111, 776, 98, 97, 114]
msg87275 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2009-05-05 19:42
Even better:

>>> for s in map(unicodedata.name, found): print(s)
... 
LATIN SMALL LETTER F
LATIN SMALL LETTER O
LATIN SMALL LETTER O
LATIN SMALL LETTER O
COMBINING DIAERESIS
LATIN SMALL LETTER B
LATIN SMALL LETTER A
LATIN SMALL LETTER R
>>> for s in map(unicodedata.name, expected): print(s)
... 
LATIN SMALL LETTER F
LATIN SMALL LETTER O
LATIN SMALL LETTER O
LATIN SMALL LETTER O WITH DIAERESIS
LATIN SMALL LETTER B
LATIN SMALL LETTER A
LATIN SMALL LETTER R
>>> found == expected
False
>>> unicodedata.normalize('NFC', found) == unicodedata.normalize('NFC', 
expected)
True

So there are two separate issues, both of which just require fixing the 
tests:  one issue is that OS X transforms invalid filenames;  the other is 
that it normalizes valid filenames.
msg87280 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2009-05-05 21:00
Here's a patch that fixes the test failures for me.  It's not ideal, 
because it no longer tests that on OS X, it's possible to open a file 
created with an invalid filename, using that same name.
msg87283 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2009-05-05 21:15
I think skipping the test entirely would be even better on Darwin. The
test is really meant to pass only on systems that treat file names as
uninterpreted byte sequences (as such, it may also fail on Linux if the
file system driver enforces UTF-8).

To skip it, it probably would be best to move it out of the "not win32"
block, and into a "neither win32 nor darwin" block - or use the skipIf
decorator.
msg87286 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2009-05-05 21:28
> I think skipping the test entirely would be even better on Darwin.

Sounds good to me!  Here's a new (one-line) patch, using skipIf.
msg87288 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2009-05-05 21:32
Fine with me, please apply.
msg87290 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2009-05-05 21:37
Committed, r72359.
msg364386 - (view) Author: Matthias Braun (Matthias Braun) * Date: 2020-03-17 02:41
I believe my suggested pull request in https://bugs.python.org/issue39986 may solve this issue as a side effect because we no longer list the root directory but a temporary directory with controlled filenames.
History
Date User Action Args
2022-04-11 14:56:48adminsetgithub: 50194
2020-03-17 02:41:24Matthias Braunsetnosy: + Matthias Braun
messages: + msg364386
2009-05-05 21:37:13mark.dickinsonsetstatus: open -> closed
resolution: accepted -> fixed
messages: + msg87290
2009-05-05 21:32:22loewissetassignee: mark.dickinson
resolution: accepted
messages: + msg87288
2009-05-05 21:28:30mark.dickinsonsetfiles: + issue5944_v2.patch

messages: + msg87286
2009-05-05 21:15:38loewissetmessages: + msg87283
2009-05-05 21:00:20mark.dickinsonsetfiles: + issue5944.patch
keywords: + patch
messages: + msg87280
2009-05-05 19:42:05mark.dickinsonsetmessages: + msg87275
2009-05-05 19:11:28mark.dickinsonsetmessages: + msg87273
2009-05-05 19:06:48r.david.murraysetnosy: + r.david.murray
messages: + msg87272
2009-05-05 18:09:01mark.dickinsonsetmessages: + msg87265
2009-05-05 18:01:17mark.dickinsoncreate