classification
Title: test_warnings.CEnvironmentVariableTests.test_nonascii fails under an ascii terminal
Type: behavior Stage:
Components: Tests Versions: Python 3.2
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: vstinner Nosy List: brett.cannon, flox, michael.foord, r.david.murray, vstinner
Priority: normal Keywords: buildbot, patch

Created on 2010-05-01 11:05 by michael.foord, last changed 2010-09-17 23:41 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
sys_warnoptions_encoding.patch vstinner, 2010-05-18 23:26
Messages (19)
msg104709 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2010-05-01 11:05
If you run test_warnings.py under an ascii terminal (at least on Mac OS X) then test_warnings.CEnvironmentVariableTests.test_nonascii fails. Perhaps the test should be skipped? 

======================================================================
FAIL: test_nonascii (__main__.CEnvironmentVariableTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Lib/test/test_warnings.py", line 758, in <module>
    test_main()
  File "Lib/test/test_warnings.py", line 753, in test_main
    PyEnvironmentVariableTests
  File "/compile/python-trunk3/Lib/test/support.py", line 1053, in run_unittest
    _run_suite(suite)
  File "/compile/python-trunk3/Lib/test/support.py", line 1027, in _run_suite
    result = runner.run(suite)
  File "/compile/python-trunk3/Lib/unittest/runner.py", line 158, in run
    result.printErrors()
  File "/compile/python-trunk3/Lib/unittest/runner.py", line 109, in printErrors
    self.printErrorList('FAIL', self.failures)
  File "/compile/python-trunk3/Lib/unittest/runner.py", line 116, in printErrorList
    self.stream.writeln("%s" % err)
  File "/compile/python-trunk3/Lib/unittest/runner.py", line 24, in writeln
    self.write(arg)
UnicodeEncodeError: 'ascii' codec can't encode character '\xf3' in position 121: ordinal not in range(128)


Looks like the traceback here actually comes from unittest trying to report the failure.
msg104716 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-05-01 14:04
Victor has proposed a patch for the traceback problem for regrtest, I think.  I haven't looked at it, but I wonder if there is something that can instead be done to make unittest work in cases like this when run in an ascii terminal.  See issue 8522.
msg104718 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2010-05-01 14:22
What does issue 8522 have to do with it - did you mean a different issue? In unittest it could catch the UnicodeEncodeError and write the ascii repr instead.
msg104721 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-05-01 15:02
Heh, yes, I meant issue 8533.
msg104774 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-05-02 09:39
The bug only occurs on Mac OS X because file system encoding is hardcoded to UTF-8 on this OS and the test is skipped if the file system encoding is ASCII.

But the bug is not specific to test_warnings, as mentionned by R. David Murray: #8533 is the real bug. test_warnings is just an example to reproduce it.

I just commited a fix for #8533: it should fix also this issue.
msg105895 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-05-17 09:45
Does my fix this #8533 fixes this issue or not? I guess yes, but I would prefer a confirmation.
msg106010 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-05-18 22:22
The test was introduced by r80066 (issue #7301).
msg106012 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-05-18 22:59
os.environ is decoded with sys.getfilesystemencoding(), whereas PYTHONWARNINGS is decoded with the locale encoding (locale.getpreferredencoding()) :-/
msg106013 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-05-18 23:26
Attached patch creates PySys_AddWarnOptionUnicode() function and decodes PYTHONWARNINGS variable with the file system encoding (as it is done for os.environ, to be consistent) instead of the locale encoding.

The patch only concerns the POSIX version. Windows is not affected by this issue (see below).

--

This issue occurs if the file system encoding is different than the locale encoding (especially if the locale encoding is ASCII). It only occurs on Mac OS because on this OS, the file system encoding is hardcoded to UTF-8 whereas the locale encoding... depends on the locale.

I reproduced the bug on Linux by hardcoding the file system encoding to UTF-8.

--

About the test output (UnicodeEncodeError: 'ascii' codec can't encode character '\xf3' ...): I think that Michael Foord executed directly Lib/test/test_warnings.py instead of using "Lib/test/regrtest.py -v test_warnings". I only patched regrtest.py to use backslashreplace error handler on sys.stdout.
msg106054 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-05-19 12:32
tarek tested on Mac OS X: the patch fixes test_warnings issue.
msg106079 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-05-19 16:54
Ok, I commited my patch: r81358 (py3k). I'm waiting for the buildbots before backporting the fix to 3.1.
msg106093 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-05-19 20:31
Oh, Python 3.1 doesn't use the PYTHONWARNINGS variable: commit blocked in 3.1 (r81362).
msg114695 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2010-08-22 19:24
A similar issue occurs on "x86 debian parallel" buildbot:

======================================================================
FAIL: test_nonascii (test.test_warnings.CEnvironmentVariableTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home2/buildbot2/slave/3.x.loewis-parallel/build/Lib/test/test_warnings.py", line 731, in test_nonascii
    "['ignore:Deprecaci├│nWarning']".encode('utf-8'))
AssertionError: b'' != b"['ignore:Deprecaci\xc3\xb3nWarning']"

http://www.python.org/dev/buildbot/builders/x86%20debian%20parallel%203.x/builds/80
msg114697 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2010-08-22 19:47
There's a failure on the same buildbot, "x86 debian parallel", with test test_subprocess (test_undecodable_env).

This test was introduced with issue #8391.

The test_subprocess and test_warnings failures are specific to this buildbot.
msg114703 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-08-22 20:28
I know test_warnings failure and I will try to fix it soon.
msg116057 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-09-10 23:14
It should be fixed by r84694.
msg116106 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2010-09-11 16:18
Still happens with r84709 on PPC Tiger 3.x


======================================================================
FAIL: test_nonascii (test.test_warnings.CEnvironmentVariableTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.parc-tiger-1/build/Lib/test/test_warnings.py", line 731, in test_nonascii
    "['ignore:Deprecaci\xf3nWarning']".encode('utf-8'))
AssertionError: b"['ignore:Deprecaci\xc3\x83\xc2\xb3nWarning']" != b"['ignore:Deprecaci\xc3\xb3nWarning']"

======================================================================
FAIL: test_nonascii (test.test_warnings.PyEnvironmentVariableTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.parc-tiger-1/build/Lib/test/test_warnings.py", line 731, in test_nonascii
    "['ignore:Deprecaci\xf3nWarning']".encode('utf-8'))
AssertionError: b"['ignore:Deprecaci\xc3\x83\xc2\xb3nWarning']" != b"['ignore:Deprecaci\xc3\xb3nWarning']"

----------------------------------------------------------------------
Ran 71 tests in 9.113s

FAILED (failures=2)



http://www.python.org/dev/buildbot/builders/PPC%20Tiger%203.x/builds/566
msg116153 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-09-12 08:09
> Still happens with r84709 on PPC Tiger 3.x

It's not the same error, PYTHONWARNINGS is decoded from the wrong encoding: locale encodind instead of utf-8. r84731 should fix this bug (at least, it restores the encoding used because my last commit, r84694).

With r84731, Python decodes PYTHONWARNINGS with strict error handler on Mac OS X, instead of surrogateescape error handler. I don't think that it matters because PYTHONWARNINGS is not supposed to contain raw bytes, and anyway, Mac OS X rejects invalid utf-8 filenames.
msg116742 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-09-17 23:41
I don't see any test_warnings anymore on http://code.google.com/p/bbreport/wiki/PythonBuildbotReport. Close this issue.
History
Date User Action Args
2010-09-17 23:41:02vstinnersetstatus: open -> closed

messages: + msg116742
2010-09-12 08:09:08vstinnersetmessages: + msg116153
2010-09-11 16:18:51floxsetstatus: closed -> open

messages: + msg116106
2010-09-10 23:14:37vstinnersetstatus: open -> closed

messages: + msg116057
2010-08-22 20:28:47vstinnersetmessages: + msg114703
2010-08-22 19:49:16floxsetkeywords: + buildbot
2010-08-22 19:47:05floxsetmessages: + msg114697
2010-08-22 19:24:27floxsetstatus: closed -> open

nosy: + flox
messages: + msg114695

type: behavior
2010-05-19 20:31:32vstinnersetstatus: pending -> closed

dependencies: - regrtest: use backslashreplace error handler for stdout
messages: + msg106093
2010-05-19 16:54:35vstinnersetstatus: open -> pending
resolution: fixed
2010-05-19 16:54:14vstinnersetmessages: + msg106079
2010-05-19 12:32:19vstinnersetmessages: + msg106054
2010-05-18 23:27:01vstinnersetfiles: + sys_warnoptions_encoding.patch
keywords: + patch
messages: + msg106013
2010-05-18 22:59:31vstinnersetmessages: + msg106012
2010-05-18 22:22:37vstinnersetmessages: + msg106010
2010-05-17 09:45:05vstinnersetmessages: + msg105895
2010-05-02 09:39:27vstinnersetassignee: brett.cannon -> vstinner
dependencies: + regrtest: use backslashreplace error handler for stdout
messages: + msg104774
2010-05-01 15:02:33r.david.murraysetmessages: + msg104721
2010-05-01 14:22:22michael.foordsetmessages: + msg104718
2010-05-01 14:04:01r.david.murraysetnosy: + r.david.murray
messages: + msg104716
2010-05-01 11:06:41pitrousetassignee: brett.cannon

components: + Tests
nosy: + vstinner, brett.cannon
2010-05-01 11:05:18michael.foordcreate