Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tiger buildbot: test_pep277 failures #52670

Closed
vstinner opened this issue Apr 16, 2010 · 10 comments
Closed

tiger buildbot: test_pep277 failures #52670

vstinner opened this issue Apr 16, 2010 · 10 comments
Labels
OS-mac tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error

Comments

@vstinner
Copy link
Member

BPO 8423
Nosy @db3l, @ronaldoussoren, @mdickinson, @vstinner, @ned-deily, @voidspace, @florentx

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2010-10-29.00:08:14.643>
created_at = <Date 2010-04-16.16:35:48.814>
labels = ['OS-mac', 'type-bug', 'tests']
title = 'tiger buildbot: test_pep277 failures'
updated_at = <Date 2010-10-29.00:08:14.642>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2010-10-29.00:08:14.642>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2010-10-29.00:08:14.643>
closer = 'vstinner'
components = ['macOS', 'Tests']
creation = <Date 2010-04-16.16:35:48.814>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 8423
keywords = ['buildbot']
message_count = 10.0
messages = ['103340', '103469', '105648', '105651', '105662', '105669', '105674', '114280', '119663', '119829']
nosy_count = 7.0
nosy_names = ['db3l', 'ronaldoussoren', 'mark.dickinson', 'vstinner', 'ned.deily', 'michael.foord', 'flox']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'needs patch'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue8423'
versions = ['Python 2.7', 'Python 3.2']
@vstinner
Copy link
Member Author

http://www.python.org/dev/buildbot/3.x/builders/x86 Tiger 3.x/builds/6/steps/test/logs/stdio

test test_pep277 failed -- multiple errors occurred; run in verbose mode for details
Re-running test test_pep277 in verbose mode
test_directory (test.test_pep277.UnicodeFileTests) ... ok
test_failures (test.test_pep277.UnicodeFileTests) ... ok
test_listdir (test.test_pep277.UnicodeFileTests) ... FAIL
test_normalize (test.test_pep277.UnicodeFileTests) ... ERROR
test_open (test.test_pep277.UnicodeFileTests) ... ok
test_rename (test.test_pep277.UnicodeFileTests) ... ok
test_directory (test.test_pep277.UnicodeNFCFileTests) ... ok
test_failures (test.test_pep277.UnicodeNFCFileTests) ... ok
test_listdir (test.test_pep277.UnicodeNFCFileTests) ... FAIL
test_normalize (test.test_pep277.UnicodeNFCFileTests) ... ERROR
test_open (test.test_pep277.UnicodeNFCFileTests) ... ok
test_rename (test.test_pep277.UnicodeNFCFileTests) ... ok
test_directory (test.test_pep277.UnicodeNFDFileTests) ... ok
test_failures (test.test_pep277.UnicodeNFDFileTests) ... ok
test_listdir (test.test_pep277.UnicodeNFDFileTests) ... ok
test_normalize (test.test_pep277.UnicodeNFDFileTests) ... ERROR
test_open (test.test_pep277.UnicodeNFDFileTests) ... ok
test_rename (test.test_pep277.UnicodeNFDFileTests) ... ok
test_directory (test.test_pep277.UnicodeNFKCFileTests) ... ok
test_failures (test.test_pep277.UnicodeNFKCFileTests) ... ok
test_listdir (test.test_pep277.UnicodeNFKCFileTests) ... FAIL
test_normalize (test.test_pep277.UnicodeNFKCFileTests) ... ERROR
test_open (test.test_pep277.UnicodeNFKCFileTests) ... ok
test_rename (test.test_pep277.UnicodeNFKCFileTests) ... ok
test_directory (test.test_pep277.UnicodeNFKDFileTests) ... ok
test_failures (test.test_pep277.UnicodeNFKDFileTests) ... ok
test_listdir (test.test_pep277.UnicodeNFKDFileTests) ... ok
test_normalize (test.test_pep277.UnicodeNFKDFileTests) ... ERROR
test_open (test.test_pep277.UnicodeNFKDFileTests) ... ok
test_rename (test.test_pep277.UnicodeNFKDFileTests) ... ok

======================================================================
ERROR: test_normalize (test.test_pep277.UnicodeFileTests)
----------------------------------------------------------------------

test test_pep277 crashed -- <class 'UnicodeEncodeError'>: 'ascii' codec can't encode characters in position 222-239: ordinal not in range(128)
Traceback (most recent call last):
  File "./Lib/test/regrtest.py", line 905, in runtest_inner
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/test_pep277.py", line 195, in test_main
    UnicodeNFKDFileTests,
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/support.py", line 1000, in run_unittest
    _run_suite(suite)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/support.py", line 974, in _run_suite
    result = runner.run(suite)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/runner.py", line 158, in run
    result.printErrors()
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/runner.py", line 108, in printErrors
    self.printErrorList('ERROR', self.errors)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/runner.py", line 116, in printErrorList
    self.stream.writeln("%s" % err)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/runner.py", line 24, in writeln
    self.write(arg)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 222-239: ordinal not in range(128)

@vstinner
Copy link
Member Author

See also bpo-8422.

@florentx florentx mannequin added OS-mac tests Tests in the Lib/test dir labels May 13, 2010
@florentx florentx mannequin assigned ronaldoussoren May 13, 2010
@florentx florentx mannequin added the type-bug An unexpected behavior, bug, or error label May 13, 2010
@florentx
Copy link
Mannequin

florentx mannequin commented May 13, 2010

Some diagnosis information ...

  • happens for 2 tests inside test_pep277: test_listdir and test_normalize
  • happens only for 4 weird filenames (added with bpo-8180)
  • happens only on "x86 Tiger" buildbot ("PPC Tiger" does not report this error)

I've added some lines to regrtest.py to identify each platform:
"x86 Tiger": Darwin-8.10.1-i386-32bit little-endian (OS X 10.4.10)
"PPC Tiger": Darwin-8.11.0-Power_Macintosh-powerpc-32bit big-endian (OS X 10.4.11)

I don't know how the endianness may be linked with this issue.

Failing files (4):
u'\u0385\u03d3\u03d4',
u'\u00a8\u0301\u03d2\u0301\u03d2\u0308', # == NFD(u'\u0385\u03d3\u03d4')
u'\u0020\u0308\u0301\u038e\u03ab', # == NFKC(u'\u0385\u03d3\u03d4')
u'\u1e9b\u1fc1\u1fcd\u1fce\u1fcf\u1fdd\u1fde\u1fdf\u1fed',

One solution is to skip these files on all 'darwin' platforms, without losing more time on this issue.

@florentx florentx mannequin unassigned ronaldoussoren May 13, 2010
@ronaldoussoren
Copy link
Contributor

It prefer if it were clear why these tests fail (but don't have time to look into this at the moment). That is, I don't mind if the tests get disabled on darwin, but only if there is a documented reason for it because otherwise disabling the test might hide a real issue.

@mdickinson
Copy link
Member

It prefer if it were clear why these tests fail

I agree. There's little point having tests if we're just going to disable them when they fail without understanding why they're failing first.

At that point a green buildbot just means "Hey, we disabled all failing tests!", rather than being an indication that Python's working well on that platform.

@florentx
Copy link
Mannequin

florentx mannequin commented May 13, 2010

The gory details...

>>> import os, unicodedata
>>> tempdir = u'temp_pep277'
>>> name = u'\u0385\u03d3\u03d4'
>>> os.mkdir(tempdir)
>>> with open(os.path.join(tempdir, name), 'w') as f:
...     f.write('foo')
>>> unicodedata.normalize('NFD', name)
u'\xa8\u0301\u03d2\u0301\u03d2\u0308'
>>> os.listdir(tempdir)
[u'\xa8\u0301\u03d2\u03d2\u0301\u0308']

The filename should use the Normal Decomposition (NFD) of the original filename. But the decomposition looks wrong on this platform.

The problem is similar for the other special filenames. Examples:

  • u'\u0385\u03d3\u03d4'
    ==> NFD: u'\xa8\u0301\u03d2\u0301\u03d2\u0308'
    ==> OSX: u'\xa8\u0301\u03d2\u03d2\u0301\u0308'
  • u' \u0308\u0301\u038e\u03ab'
    ==> NFD: u' \u0308\u0301\u03a5\u0301\u03a5\u0308'
    ==> OSX: u' \u0308\u0301\u03a5\u03a5\u0301\u0308'
  • u'\u1e9b\u1fc1\u1fcd\u1fce\u1fcf\u1fdd\u1fde\u1fdf\u1fed'
    ==> NFD: u'\u017f\u0307\xa8\u0342\u1fbf\u0300\u1fbf\u0301\u1fbf\u0342\u1ffe\u0300\u1ffe\u0301\u1ffe\u0342\xa8\u0300'
    ==> OSX: u'\u017f\u0307\xa8\u0342\u1fbf\u0300\u1fbf\u0301\u1fbf\u1ffe\u1ffe\u1ffe\xa8\u0342\u0300\u0301\u0342\u0300'

I have no idea how to investigate further.
And the "PPC Tiger" buildbot does not fail here.

Reference about OSX decomposition:
http://developer.apple.com/mac/library/qa/qa2001/qa1173.html

@vstinner
Copy link
Member Author

Reference about OSX decomposition:
http://developer.apple.com/mac/library/qa/qa2001/qa1173.html

Mac OS X source code:
http://www.opensource.apple.com/source/xnu/xnu-1504.3.12/bsd/vfs/vfs_utfconv.c?txt

@vstinner
Copy link
Member Author

bpo-8207 was the same issue and r79426 (for trunk and r79426 in py3k) was supposed to fix it.

@vstinner
Copy link
Member Author

I opened the issue bpo-10209: "Mac OS X: Decompose filenames on encode, and precompose filenames on decode".

@vstinner
Copy link
Member Author

I opened the issue bpo-10209: "Mac OS X: Decompose filenames on encode,
and precompose filenames on decode".

It was a bad idea. I fixed test_pep277 instead with:

  • r85897 disables the filenames that are normalized differently by Python and by darwin
  • r85899 disables test_normalize and test_listdir tests

And test_pep277 now pass on "x86 Tiger 3.x" buildbot, and so I can close this issue (and issue bpo-10209).

@ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
OS-mac tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@vstinner @mdickinson @ronaldoussoren