This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: test_gdb fails with Posix locale in 3.7
Type: behavior Stage: resolved
Components: Versions: Python 3.8, Python 3.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: dmalcolm, ncoghlan, serhiy.storchaka, vstinner, xtreak
Priority: normal Keywords:

Created on 2018-02-12 19:43 by serhiy.storchaka, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Messages (7)
msg312069 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-02-12 19:43
$ LC_ALL=C ./python -We -m test -vuall -m test_strings test_gdb
...
======================================================================
FAIL: test_strings (test.test_gdb.PrettyPrintTests)
Verify the pretty-printing of unicode strings
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/serhiy/py/cpython3.7/Lib/test/test_gdb.py", line 331, in test_strings
    check_repr('\u2620')
  File "/home/serhiy/py/cpython3.7/Lib/test/test_gdb.py", line 323, in check_repr
    self.assertGdbRepr(text)
  File "/home/serhiy/py/cpython3.7/Lib/test/test_gdb.py", line 271, in assertGdbRepr
    % (gdb_repr, exp_repr, gdb_output)))
AssertionError: "'\\u2620'" != "'☠'"
- '\u2620'
+ '☠'
 : "'\\u2620'" did not equal expected "'☠'"; full output was:
Breakpoint 1 at 0x11a4df: file Python/bltinmodule.c, line 1215.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Breakpoint 1, builtin_id (self=self@entry=, v='\u2620') at Python/bltinmodule.c:1215
1215    {
#0  builtin_id (self=, v='\u2620') at Python/bltinmodule.c:1215


----------------------------------------------------------------------

This looks be related to PEP 538.
msg312095 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2018-02-13 02:33
I can reproduce this, but it looks to me like it's being triggered by UTF-8 mode rather than locale coercion (the "LC_ALL=C" setting will implicitly disable locale coercion entirely):

```
$ LANG=C PYTHONCOERCECLOCALE=warn ./python -We -m test -uall -m test_strings test_gdb
Python detected LC_CTYPE=C: LC_CTYPE coerced to C.UTF-8 (set another locale or PYTHONCOERCECLOCALE=0 to disable this locale coercion behavior).
Run tests sequentially
0:00:00 load avg: 0.34 [1/1] test_gdb
1 test OK.

Total duration: 2 sec
Tests result: SUCCESS

$ LC_ALL=C PYTHONUTF8=0 ./python -We -m test -uall -m test_strings test_gdb
Run tests sequentially
0:00:00 load avg: 0.17 [1/1] test_gdb
1 test OK.

Total duration: 2 sec
Tests result: SUCCESS
```

I expect the fix will be to adjust test_gdb's expectations in this case,  as there's going to be a genuine difference between gdb's locale dependent behaviour and Python's UTF-8 mode behaviour.
msg334420 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2019-01-27 12:02
Possibly fixed with issue34537 ?
msg334433 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-01-27 17:22
> Possibly fixed with issue34537 ?

"LC_ALL=C ./python -We -m test -vuall -m test_strings test_gdb" still fails for me on Fedora 29.

======================================================================
FAIL: test_strings (test.test_gdb.PrettyPrintTests)
Verify the pretty-printing of unicode strings
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/test/test_gdb.py", line 380, in test_strings
    check_repr('\u2620')
  File "/home/vstinner/prog/python/master/Lib/test/test_gdb.py", line 372, in check_repr
    self.assertGdbRepr(text)
  File "/home/vstinner/prog/python/master/Lib/test/test_gdb.py", line 302, in assertGdbRepr
    gdb_repr, gdb_output = self.get_gdb_repr('id(' + ascii(val) + ')')
  File "/home/vstinner/prog/python/master/Lib/test/test_gdb.py", line 269, in get_gdb_repr
    gdb_output = self.get_stack_trace(source, breakpoint=BREAKPOINT_FN,
  File "/home/vstinner/prog/python/master/Lib/test/test_gdb.py", line 249, in get_stack_trace
    self.assertEqual(unexpected_errlines, [])
AssertionError: Lists differ: ["Python Exception <class 'UnicodeEncodeEr[231 chars]): "] != []

First list contains 2 additional elements.
First extra element 0:
"Python Exception <class 'UnicodeEncodeError'> 'ascii' codec can't encode character '\\u2620' in position 1: ordinal not in range(128): "

+ []
- ["Python Exception <class 'UnicodeEncodeError'> 'ascii' codec can't encode "
-  "character '\\u2620' in position 1: ordinal not in range(128): ",
-  "Python Exception <class 'UnicodeEncodeError'> 'ascii' codec can't encode "
-  "character '\\u2620' in position 1: ordinal not in range(128): "]


Note: "LC_ALL=C ./python -m test -m test_strings test_gdb" (simpler command) also fails.
msg334478 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2019-01-28 16:17
This passes on my Ubuntu box and fails for the commit before issue34537 with the stack trace as per the original report.

karthi@ubuntu-s-1vcpu-1gb-blr1-01:~/cpython$ LC_ALL=C ./python -We -m test -vuall -m test_strings test_gdb
== CPython 3.8.0a0 (heads/master:ea446409cd, Jan 28 2019, 15:07:17) [GCC 5.4.0 20160609]
== Linux-4.4.0-127-generic-x86_64-with-glibc2.17 little-endian
== cwd: /home/karthi/cpython/build/test_python_19425
== CPU count: 1
== encodings: locale=UTF-8, FS=utf-8
Run tests sequentially
0:00:00 load avg: 0.89 [1/1] test_gdb
GDB version 8.2:
    GNU gdb (Ubuntu 8.2-0ubuntu1~16.04.1) 8.2
    Copyright (C) 2018 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.
test_strings (test.test_gdb.PrettyPrintTests)
Verify the pretty-printing of unicode strings ... ok

----------------------------------------------------------------------

Ran 1 test in 3.080s

OK

== Tests result: SUCCESS ==

1 test OK.

Total duration: 3 sec 571 ms
Tests result: SUCCESS
karthi@ubuntu-s-1vcpu-1gb-blr1-01:~/cpython$ git checkout 7279b5125e7c5d84a473d250b27d353cb7f6628e~1 
[...]
karthi@ubuntu-s-1vcpu-1gb-blr1-01:~/cpython$ LC_ALL=C ./python -We -m test -vuall -m test_strings test_gdb
== CPython 3.8.0a0 (heads/master:ea446409cd, Jan 28 2019, 15:07:17) [GCC 5.4.0 20160609]
== Linux-4.4.0-127-generic-x86_64-with-glibc2.17 little-endian
== cwd: /home/karthi/cpython/build/test_python_19484
== CPU count: 1
== encodings: locale=UTF-8, FS=utf-8
Run tests sequentially
0:00:00 load avg: 0.91 [1/1] test_gdb
GDB version 8.2:
    GNU gdb (Ubuntu 8.2-0ubuntu1~16.04.1) 8.2
    Copyright (C) 2018 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.
test_strings (test.test_gdb.PrettyPrintTests)
Verify the pretty-printing of unicode strings ... FAIL

======================================================================
FAIL: test_strings (test.test_gdb.PrettyPrintTests)
Verify the pretty-printing of unicode strings
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/karthi/cpython/Lib/test/test_gdb.py", line 340, in test_strings
    check_repr('\u2620')
  File "/home/karthi/cpython/Lib/test/test_gdb.py", line 332, in check_repr
    self.assertGdbRepr(text)
  File "/home/karthi/cpython/Lib/test/test_gdb.py", line 278, in assertGdbRepr
    self.assertEqual(gdb_repr, exp_repr,
AssertionError: "'\\u2620'" != "'☠'"
- '\u2620'
+ '☠'
 : "'\\u2620'" did not equal expected "'☠'"; full output was:
Breakpoint 1 at 0x5f9036: file Python/bltinmodule.c, line 1178.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Breakpoint 1, builtin_id (self=self@entry=, v='\u2620') at Python/bltinmodule.c:1178
1178
#0  builtin_id (self=, v='\u2620') at Python/bltinmodule.c:1178


----------------------------------------------------------------------

Ran 1 test in 1.979s

FAILED (failures=1)
test test_gdb failed
test_gdb failed

== Tests result: FAILURE ==

1 test failed:
    test_gdb

Total duration: 2 sec 576 ms
Tests result: FAILURE
msg334515 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2019-01-29 13:27
Added Dave Malcolm to the nosy list, as the more recent tracebacks are actually throwing an exception in the gdb hooks, rather than just failing the expected output comparison in the test suite.
msg371280 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-11 13:47
I close the issue. I cannot reproduce the issue on Fedora 32. Please open it if you can still reproduce it on the master branch. In this case, please mention your GDB version, Python version used by GDB and tested Python version.

* Python master at commit 9727694f08cad4b019d2939224e3416312b1c0e1
* gcc (GCC) 10.1.1 20200507 (Red Hat 10.1.1-1)
* GNU gdb (GDB) Fedora 9.1-5.fc32

On Fedora 32, GDB is linked to Python 3.8:

$ gdb 
GNU gdb (GDB) Fedora 9.1-5.fc32
(gdb) python import sys; print(sys.version)
3.8.3 (default, May 15 2020, 00:00:00) 
[GCC 10.1.1 20200507 (Red Hat 10.1.1-1)]


test_gdb:

vstinner@apu$ LC_ALL=C ./python -We -m test -vuall -m test_strings test_gdb
== CPython 3.10.0a0 (heads/master:9727694f08, Jun 11 2020, 15:44:14) [GCC 10.1.1 20200507 (Red Hat 10.1.1-1)]
== Linux-5.6.16-300.fc32.x86_64-x86_64-with-glibc2.31 little-endian
== cwd: /home/vstinner/python/master/build/test_python_151860
== CPU count: 8
== encodings: locale=UTF-8, FS=utf-8
0:00:00 load avg: 1.68 Run tests sequentially
0:00:00 load avg: 1.68 [1/1] test_gdb
GDB version 9.1:
    GNU gdb (GDB) Fedora 9.1-5.fc32
    Copyright (C) 2020 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.
test_strings (test.test_gdb.PrettyPrintTests)
Verify the pretty-printing of unicode strings ... ok

----------------------------------------------------------------------

Ran 1 test in 1.834s

OK

== Tests result: SUCCESS ==

1 test OK.

Total duration: 2.4 sec
Tests result: SUCCESS
History
Date User Action Args
2022-04-11 14:58:57adminsetgithub: 77015
2020-06-11 13:47:29vstinnersetstatus: open -> closed
resolution: out of date
messages: + msg371280

stage: resolved
2019-01-29 13:27:40ncoghlansetnosy: + dmalcolm
messages: + msg334515
2019-01-28 16:17:22xtreaksetmessages: + msg334478
2019-01-27 17:22:30vstinnersetmessages: + msg334433
2019-01-27 12:02:14xtreaksetnosy: + xtreak
messages: + msg334420
2018-02-13 02:33:33ncoghlansetnosy: + vstinner
messages: + msg312095
2018-02-12 19:43:13serhiy.storchakacreate