classification
Title: test_gdb fails with Posix locale in 3.7
Type: behavior Stage:
Components: Versions: Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: dmalcolm, ncoghlan, serhiy.storchaka, vstinner, xtreak
Priority: normal Keywords:

Created on 2018-02-12 19:43 by serhiy.storchaka, last changed 2019-01-29 13:27 by ncoghlan.

Messages (6)
msg312069 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-02-12 19:43
$ LC_ALL=C ./python -We -m test -vuall -m test_strings test_gdb
...
======================================================================
FAIL: test_strings (test.test_gdb.PrettyPrintTests)
Verify the pretty-printing of unicode strings
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/serhiy/py/cpython3.7/Lib/test/test_gdb.py", line 331, in test_strings
    check_repr('\u2620')
  File "/home/serhiy/py/cpython3.7/Lib/test/test_gdb.py", line 323, in check_repr
    self.assertGdbRepr(text)
  File "/home/serhiy/py/cpython3.7/Lib/test/test_gdb.py", line 271, in assertGdbRepr
    % (gdb_repr, exp_repr, gdb_output)))
AssertionError: "'\\u2620'" != "'☠'"
- '\u2620'
+ '☠'
 : "'\\u2620'" did not equal expected "'☠'"; full output was:
Breakpoint 1 at 0x11a4df: file Python/bltinmodule.c, line 1215.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Breakpoint 1, builtin_id (self=self@entry=, v='\u2620') at Python/bltinmodule.c:1215
1215    {
#0  builtin_id (self=, v='\u2620') at Python/bltinmodule.c:1215


----------------------------------------------------------------------

This looks be related to PEP 538.
msg312095 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2018-02-13 02:33
I can reproduce this, but it looks to me like it's being triggered by UTF-8 mode rather than locale coercion (the "LC_ALL=C" setting will implicitly disable locale coercion entirely):

```
$ LANG=C PYTHONCOERCECLOCALE=warn ./python -We -m test -uall -m test_strings test_gdb
Python detected LC_CTYPE=C: LC_CTYPE coerced to C.UTF-8 (set another locale or PYTHONCOERCECLOCALE=0 to disable this locale coercion behavior).
Run tests sequentially
0:00:00 load avg: 0.34 [1/1] test_gdb
1 test OK.

Total duration: 2 sec
Tests result: SUCCESS

$ LC_ALL=C PYTHONUTF8=0 ./python -We -m test -uall -m test_strings test_gdb
Run tests sequentially
0:00:00 load avg: 0.17 [1/1] test_gdb
1 test OK.

Total duration: 2 sec
Tests result: SUCCESS
```

I expect the fix will be to adjust test_gdb's expectations in this case,  as there's going to be a genuine difference between gdb's locale dependent behaviour and Python's UTF-8 mode behaviour.
msg334420 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2019-01-27 12:02
Possibly fixed with issue34537 ?
msg334433 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-01-27 17:22
> Possibly fixed with issue34537 ?

"LC_ALL=C ./python -We -m test -vuall -m test_strings test_gdb" still fails for me on Fedora 29.

======================================================================
FAIL: test_strings (test.test_gdb.PrettyPrintTests)
Verify the pretty-printing of unicode strings
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/test/test_gdb.py", line 380, in test_strings
    check_repr('\u2620')
  File "/home/vstinner/prog/python/master/Lib/test/test_gdb.py", line 372, in check_repr
    self.assertGdbRepr(text)
  File "/home/vstinner/prog/python/master/Lib/test/test_gdb.py", line 302, in assertGdbRepr
    gdb_repr, gdb_output = self.get_gdb_repr('id(' + ascii(val) + ')')
  File "/home/vstinner/prog/python/master/Lib/test/test_gdb.py", line 269, in get_gdb_repr
    gdb_output = self.get_stack_trace(source, breakpoint=BREAKPOINT_FN,
  File "/home/vstinner/prog/python/master/Lib/test/test_gdb.py", line 249, in get_stack_trace
    self.assertEqual(unexpected_errlines, [])
AssertionError: Lists differ: ["Python Exception <class 'UnicodeEncodeEr[231 chars]): "] != []

First list contains 2 additional elements.
First extra element 0:
"Python Exception <class 'UnicodeEncodeError'> 'ascii' codec can't encode character '\\u2620' in position 1: ordinal not in range(128): "

+ []
- ["Python Exception <class 'UnicodeEncodeError'> 'ascii' codec can't encode "
-  "character '\\u2620' in position 1: ordinal not in range(128): ",
-  "Python Exception <class 'UnicodeEncodeError'> 'ascii' codec can't encode "
-  "character '\\u2620' in position 1: ordinal not in range(128): "]


Note: "LC_ALL=C ./python -m test -m test_strings test_gdb" (simpler command) also fails.
msg334478 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2019-01-28 16:17
This passes on my Ubuntu box and fails for the commit before issue34537 with the stack trace as per the original report.

karthi@ubuntu-s-1vcpu-1gb-blr1-01:~/cpython$ LC_ALL=C ./python -We -m test -vuall -m test_strings test_gdb
== CPython 3.8.0a0 (heads/master:ea446409cd, Jan 28 2019, 15:07:17) [GCC 5.4.0 20160609]
== Linux-4.4.0-127-generic-x86_64-with-glibc2.17 little-endian
== cwd: /home/karthi/cpython/build/test_python_19425
== CPU count: 1
== encodings: locale=UTF-8, FS=utf-8
Run tests sequentially
0:00:00 load avg: 0.89 [1/1] test_gdb
GDB version 8.2:
    GNU gdb (Ubuntu 8.2-0ubuntu1~16.04.1) 8.2
    Copyright (C) 2018 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.
test_strings (test.test_gdb.PrettyPrintTests)
Verify the pretty-printing of unicode strings ... ok

----------------------------------------------------------------------

Ran 1 test in 3.080s

OK

== Tests result: SUCCESS ==

1 test OK.

Total duration: 3 sec 571 ms
Tests result: SUCCESS
karthi@ubuntu-s-1vcpu-1gb-blr1-01:~/cpython$ git checkout 7279b5125e7c5d84a473d250b27d353cb7f6628e~1 
[...]
karthi@ubuntu-s-1vcpu-1gb-blr1-01:~/cpython$ LC_ALL=C ./python -We -m test -vuall -m test_strings test_gdb
== CPython 3.8.0a0 (heads/master:ea446409cd, Jan 28 2019, 15:07:17) [GCC 5.4.0 20160609]
== Linux-4.4.0-127-generic-x86_64-with-glibc2.17 little-endian
== cwd: /home/karthi/cpython/build/test_python_19484
== CPU count: 1
== encodings: locale=UTF-8, FS=utf-8
Run tests sequentially
0:00:00 load avg: 0.91 [1/1] test_gdb
GDB version 8.2:
    GNU gdb (Ubuntu 8.2-0ubuntu1~16.04.1) 8.2
    Copyright (C) 2018 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.
test_strings (test.test_gdb.PrettyPrintTests)
Verify the pretty-printing of unicode strings ... FAIL

======================================================================
FAIL: test_strings (test.test_gdb.PrettyPrintTests)
Verify the pretty-printing of unicode strings
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/karthi/cpython/Lib/test/test_gdb.py", line 340, in test_strings
    check_repr('\u2620')
  File "/home/karthi/cpython/Lib/test/test_gdb.py", line 332, in check_repr
    self.assertGdbRepr(text)
  File "/home/karthi/cpython/Lib/test/test_gdb.py", line 278, in assertGdbRepr
    self.assertEqual(gdb_repr, exp_repr,
AssertionError: "'\\u2620'" != "'☠'"
- '\u2620'
+ '☠'
 : "'\\u2620'" did not equal expected "'☠'"; full output was:
Breakpoint 1 at 0x5f9036: file Python/bltinmodule.c, line 1178.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Breakpoint 1, builtin_id (self=self@entry=, v='\u2620') at Python/bltinmodule.c:1178
1178
#0  builtin_id (self=, v='\u2620') at Python/bltinmodule.c:1178


----------------------------------------------------------------------

Ran 1 test in 1.979s

FAILED (failures=1)
test test_gdb failed
test_gdb failed

== Tests result: FAILURE ==

1 test failed:
    test_gdb

Total duration: 2 sec 576 ms
Tests result: FAILURE
msg334515 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2019-01-29 13:27
Added Dave Malcolm to the nosy list, as the more recent tracebacks are actually throwing an exception in the gdb hooks, rather than just failing the expected output comparison in the test suite.
History
Date User Action Args
2019-01-29 13:27:40ncoghlansetnosy: + dmalcolm
messages: + msg334515
2019-01-28 16:17:22xtreaksetmessages: + msg334478
2019-01-27 17:22:30vstinnersetmessages: + msg334433
2019-01-27 12:02:14xtreaksetnosy: + xtreak
messages: + msg334420
2018-02-13 02:33:33ncoghlansetnosy: + vstinner
messages: + msg312095
2018-02-12 19:43:13serhiy.storchakacreate