classification
Title: test_gdb fails for UCS2 builds with UCS2 gdb
Type: behavior Stage: resolved
Components: Library (Lib), Tests Versions: Python 3.2, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: dmalcolm, pitrou
Priority: normal Keywords: patch

Created on 2010-07-07 13:17 by pitrou, last changed 2010-09-08 22:28 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
only-join-surrogate-pairs-when-widths-vary.patch dmalcolm, 2010-07-07 18:23
test_gdb2.patch pitrou, 2010-09-08 20:31
Messages (9)
msg109473 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-07-07 13:17
======================================================================
FAIL: test_unicode (test.test_gdb.PrettyPrintTests)
Verify the pretty-printing of unicode values
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/antoine/cpython/27/Lib/test/test_gdb.py", line 247, in test_unicode
    self.assertGdbRepr(u"\U0001D121")
  File "/home/antoine/cpython/27/Lib/test/test_gdb.py", line 178, in assertGdbRepr
    self.assertEquals(gdb_repr, repr(val), gdb_output)
AssertionError: Breakpoint 1 at 0x45cde8: file Objects/object.c, line 330.
[Thread debugging using libthread_db enabled]

Breakpoint 1, PyObject_Print (op=Traceback (most recent call last):
  File "/home/antoine/cpython/27/python-gdb.py", line 1084, in to_string
    return pyop.get_truncated_repr(MAX_OUTPUT_LEN)
  File "/home/antoine/cpython/27/python-gdb.py", line 183, in get_truncated_repr
    self.write_repr(out, set())
  File "/home/antoine/cpython/27/python-gdb.py", line 1054, in write_repr
    proxy2.append(unichr(code))
ValueError: unichr() arg not in range(0x10000) (narrow Python build)
, fp=0x7ffff7535780, flags=1) at Objects/object.c:330
330	    return internal_print(op, fp, flags, 0);
#0  PyObject_Print (op=Traceback (most recent call last):
  File "/home/antoine/cpython/27/python-gdb.py", line 1084, in to_string
    return pyop.get_truncated_repr(MAX_OUTPUT_LEN)
  File "/home/antoine/cpython/27/python-gdb.py", line 183, in get_truncated_repr
    self.write_repr(out, set())
  File "/home/antoine/cpython/27/python-gdb.py", line 1054, in write_repr
    proxy2.append(unichr(code))
ValueError: unichr() arg not in range(0x10000) (narrow Python build)
, fp=0x7ffff7535780, flags=1) at Objects/object.c:330
#1  0x0000000000431744 in file_PyObject_Print (op=Traceback (most recent call last):
  File "/home/antoine/cpython/27/python-gdb.py", line 1084, in to_string
    return pyop.get_truncated_repr(MAX_OUTPUT_LEN)
  File "/home/antoine/cpython/27/python-gdb.py", line 183, in get_truncated_repr
    self.write_repr(out, set())
  File "/home/antoine/cpython/27/python-gdb.py", line 1054, in write_repr
    proxy2.append(unichr(code))
ValueError: unichr() arg not in range(0x10000) (narrow Python build)
, f=0x7ffff7fc1280, flags=1) at Objects/fileobject.c:110
#2  0x00000000004369e2 in PyFile_WriteObject (v=Traceback (most recent call last):
  File "/home/antoine/cpython/27/python-gdb.py", line 1084, in to_string
    return pyop.get_truncated_repr(MAX_OUTPUT_LEN)
  File "/home/antoine/cpython/27/python-gdb.py", line 183, in get_truncated_repr
    self.write_repr(out, set())
  File "/home/antoine/cpython/27/python-gdb.py", line 1054, in write_repr
    proxy2.append(unichr(code))
ValueError: unichr() arg not in range(0x10000) (narrow Python build)
, f=<file at remote 0x7ffff7fc1280>, flags=1) at Objects/fileobject.c:2472
#3  0x00000000004c9709 in PyEval_EvalFrameEx (f=Frame 0x9190f0, for file <string>, line 1, in <module> (), 
    throwflag=0) at Python/ceval.c:1769
#4  0x00000000004cfd36 in PyEval_EvalCodeEx (co=0x7ffff7fb8930, globals=
    {'__builtins__': <module at remote 0x7ffff7fd2470>, '__name__': '__main__', '__doc__': None, '__package__': None}, locals=
    {'__builtins__': <module at remote 0x7ffff7fd2470>, '__name__': '__main__', '__doc__': None, '__package__': None}, args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3252
#5  0x00000000004c5dc4 in PyEval_EvalCode (co=0x7ffff7fb8930, globals=
    {'__builtins__': <module at remote 0x7ffff7fd2470>, '__name__': '__main__', '__doc__': None, '__package__': None}, locals=
    {'__builtins__': <module at remote 0x7ffff7fd2470>, '__name__': '__main__', '__doc__': None, '__package__': None}) at Python/ceval.c:666
#6  0x0000000000500a4c in run_mod (mod=0x903828, filename=0x5a1bbd "<string>", globals=
    {'__builtins__': <module at remote 0x7ffff7fd2470>, '__name__': '__main__', '__doc__': None, '__package__': None}, locals=
    {'__builtins__': <module at remote 0x7ffff7fd2470>, '__name__': '__main__', '__doc__': None, '__package__': None}, flags=0x7fffffffdbc0, arena=0x8bc5f0) at Python/pythonrun.c:1346
#7  0x00000000005008eb in PyRun_StringFlags (str=0x82a010 "print u'\\U0001d121'\n", start=257, globals=
    {'__builtins__': <module at remote 0x7ffff7fd2470>, '__name__': '__main__', '__doc__': None, '__package__': None}, locals=
    {'__builtins__': <module at remote 0x7ffff7fd2470>, '__name__': '__main__', '__doc__': None, '__package__': None}, flags=0x7fffffffdbc0) at Python/pythonrun.c:1309
#8  0x00000000004ff3f5 in PyRun_SimpleStringFlags (command=0x82a010 "print u'\\U0001d121'\n", flags=
    0x7fffffffdbc0) at Python/pythonrun.c:962
#9  0x00000000004155a4 in Py_Main (argc=4, argv=0x7fffffffdd48) at Modules/main.c:543
#10 0x0000000000414484 in main (argc=4, argv=0x7fffffffdd48) at ./Modules/python.c:23
msg109487 - (view) Author: Dave Malcolm (dmalcolm) (Python committer) Date: 2010-07-07 18:23
The traceback is
Traceback (most recent call last):
  File "/home/antoine/cpython/27/python-gdb.py", line 1084, in to_string
    return pyop.get_truncated_repr(MAX_OUTPUT_LEN)
  File "/home/antoine/cpython/27/python-gdb.py", line 183, in get_truncated_repr
    self.write_repr(out, set())
  File "/home/antoine/cpython/27/python-gdb.py", line 1054, in write_repr
    proxy2.append(unichr(code))
ValueError: unichr() arg not in range(0x10000) (narrow Python build)

and occurs within the gdb process whilst trying to pretty-print a PyUnicodeObject whilst running "print u'\\U0001d121'\n"

It looks like the gdb you're using has been built against a python built with narrow unicode, and the python being debugged is also built with narrow unicode.

The code in question was introduced in r81377 and replaces surrogate pairs with their UCS4 equivalent, converting UCS2 characters from the inferior process into UCS4 characters for use in the gdb process.  It appears to assume that the gdb executable was linked against a UCS4-build of python.

The attached patch (against release27-maint) tries to turn off this joining of surrogates, except in the case where inferior is UCS2 and gdb is UCS4.

Tested successfully with:
 - inferior:UCS2  gdb:UCS4
 - inferior:UCS4  gdb:UCS4

Not yet tested with gdb UCS2

Antoine, does the above sound sane?  Can you test this on your UCS2 gdb please?
msg109569 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-07-08 19:01
The patch works ok on 2.7 but fails to apply on 3.2. Can you provide a 3.2 patch as well?
Also, instead of `sys.maxunicode == 0x10ffff`, it would be better to use something more future-proof such as `sys.maxunicode >= 0x10000`.
msg115895 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-09-08 20:31
This is a patch for 3.x, tested in the following situations:
- gdb: UCS2, inferior: UCS2
- gdb: UCS2, inferior: UCS4
msg115896 - (view) Author: Dave Malcolm (dmalcolm) (Python committer) Date: 2010-09-08 20:48
Tested with UCS4 gdb:
(gdb) python import sys; print hex(sys.maxunicode)
0x10ffff

using latest py3k.

Works with both UCS2 and UCS4 builds of py3k
msg115897 - (view) Author: Dave Malcolm (dmalcolm) (Python committer) Date: 2010-09-08 20:49
where "works" means: all tests in test_gdb.py were run, and passed.

This was with a --with-pydebug-build in each case
msg115899 - (view) Author: Dave Malcolm (dmalcolm) (Python committer) Date: 2010-09-08 20:57
One minor quibble with the patch:

In the line:
   while i < field_length:
you're trusting that field_length has a sane value.  If field_str points somewhere readable, and field_length is huge (e.g. 0xfffffff), then gdb could sit there for a while reading all that data until it either completes, or hits memory that it can't read (leading to a python exception within gdb).

In other places in that code I used safe_range() to limit iterations to a safety threshold; I suggest the use of
  safety_limit(field_length)

But this is a minor nit.


I don't think this
msg115900 - (view) Author: Dave Malcolm (dmalcolm) (Python committer) Date: 2010-09-08 21:00
"I don't think this" was of course a browser misfire; I meant to type "I don't think this should block committing this fix"
msg115909 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-09-08 22:28
Fixed in r84635, r84636, r84638 (3.x) and r84637 (2.7). Also prompted the creation of issue9804.
History
Date User Action Args
2010-09-08 22:28:00pitrousetstatus: open -> closed
resolution: fixed
messages: + msg115909

stage: patch review -> resolved
2010-09-08 21:00:24dmalcolmsetmessages: + msg115900
2010-09-08 20:57:02dmalcolmsetmessages: + msg115899
2010-09-08 20:49:21dmalcolmsetmessages: + msg115897
2010-09-08 20:48:23dmalcolmsetmessages: + msg115896
2010-09-08 20:31:19pitrousetfiles: + test_gdb2.patch

messages: + msg115895
2010-07-08 19:01:59pitrousetmessages: + msg109569
2010-07-07 18:24:01dmalcolmsetfiles: + only-join-surrogate-pairs-when-widths-vary.patch
title: test_gdb fails on narrow unicode builds -> test_gdb fails for UCS2 builds with UCS2 gdb
messages: + msg109487

keywords: + patch
stage: patch review
2010-07-07 13:17:47pitroucreate