classification
Title: curses.unget_wch and test_curses fail when linked with ncurses 5.7 and earlier
Type: behavior Stage:
Components: Tests Versions: Python 3.5, Python 3.4, Python 3.3
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: geoffreyspear, georg.brandl, haypo, jcbollinger, loewis, ned.deily, python-dev
Priority: normal Keywords:

Created on 2012-06-08 16:08 by jcbollinger, last changed 2014-03-23 14:08 by geoffreyspear.

Messages (16)
msg162532 - (view) Author: John Bollinger (jcbollinger) * Date: 2012-06-08 16:08
I encountered this test failure while attempting to verify a patch for a separate issue, and I found that it occurs with the unmodified source on the default branch:

LD_LIBRARY_PATH="$PWD" ./python -bb -Wd -m test -r -w -uall -v test_curses
== CPython 3.3.0a4+ (default:4aeb5b9b62d7, Jun 8 2012, 10:23:35) [GCC 4.4.6
20110731 (Red Hat 4.4.6-3)]
==   Linux-2.6.32-220.4.1.el6.x86_64-x86_64-with-centos-6.2-Final
little-endian
==   /home/jbolling/cpython/build/test_python_26873
Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0,
dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0,
verbose=0, bytes_warning=2, quiet=0, hash_randomization=1)
Using random seed 3072318
[1/1] test_curses
test test_curses crashed -- Traceback (most recent call last):
  File "/home/jbolling/cpython/Lib/test/regrtest.py", line 1237, in
runtest_inner
    test_runner()
  File "/home/jbolling/cpython/Lib/test/test_curses.py", line 338, in
test_main
    main(stdscr)
  File "/home/jbolling/cpython/Lib/test/test_curses.py", line 324, in main
    test_unget_wch(stdscr)
  File "/home/jbolling/cpython/Lib/test/test_curses.py", line 283, in
test_unget_wch
    read = chr(read)
OverflowError: signed integer is greater than maximum

1 test failed:
    test_curses
Re-running failed tests in verbose mode
Re-running test 'test_curses' in verbose mode
test test_curses crashed -- Traceback (most recent call last):
  File "/home/jbolling/cpython/Lib/test/regrtest.py", line 1237, in
runtest_inner
    test_runner()
  File "/home/jbolling/cpython/Lib/test/test_curses.py", line 338, in
test_main
    main(stdscr)
  File "/home/jbolling/cpython/Lib/test/test_curses.py", line 324, in main
    test_unget_wch(stdscr)
  File "/home/jbolling/cpython/Lib/test/test_curses.py", line 283, in
test_unget_wch
    read = chr(read)
OverflowError: signed integer is greater than maximum

[123272 refs]


Python was built and the tests run on CentOS 6.2 / x86_64, using the platform's standard tool chain, configured with "--enable-shared --with-threads --with-pydebug".

All other tests pass for me, although test_builtin failed when run as part of the whole suite but passed when run by itself.


For what it's worth, it looks like that particular message is emitted in exactly one place: Python/getargs.c:661 (function convertsimple()), which in this case I guess is being called indirectly from Python/bltinmodule.c:526 (function builtin_chr()).  It's not obvious to me why that would be failing.
msg163320 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-06-21 10:34
I just ran into this problem on another platform.  I believe the problem is due to a sign-extension bug in the ncurses library unget_wch function (see link below).  It was apparently fixed in nurses 5.8; I've tested with the current ncurses 5.9 and test_ncurses now runs without error.

Are you using a version of libncurses/libncursew older than 5.8?  If so, can you try building with 5.8 or 5.9?

Another question is how to handle this for the 3.3.0 release.  It looks like not all platforms have the latest ncurses and, if not, the unget_wch function and test_curses are likely to fail.  Unfortunately, there doesn't seem to be an attribute available in the curses module that gives the version of the underlying curses/ncurses library.  At the least, the problem should probably be documented somewhere.

http://invisible-island.net/ncurses/NEWS.html#t20091010
msg163330 - (view) Author: John Bollinger (jcbollinger) * Date: 2012-06-21 13:02
The system on which I encountered the test failure uses ncurses 5.7, so that's consistent with the theory that the test is tickling an ncurses bug.

I'll have a look at testing with ncurses 5.8, but it is not available from RedHat or CentOS (and it never will be for the current and past versions of those systems), so that's not a good solution for most users.

On the other hand, it's not clear to me how serious is the bug revealed by the test failures, nor whether there is any viable workaround on the Python side.
msg163334 - (view) Author: John Bollinger (jcbollinger) * Date: 2012-06-21 14:11
Clarification: "so that's not a good solution for most users" ... of RedHat-family distros, version 6.2 and earlier.

In fact, it looks like RedHat is sticking with its current version of ncurses for RHEL 6.3, too, so no help is coming from that direction any time soon.
msg163336 - (view) Author: John Bollinger (jcbollinger) * Date: 2012-06-21 14:30
Ok, I confirm that the test passes after the system's ncurses library is upgraded to ncurses 5.8, and fails again when ncurses is downgraded back to version 5.7.
msg163361 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-06-21 20:09
Thanks for the testing.

Georg, haypo: I think a call should be made on what, if anything, to do about this prior to 3.3.0-final.  It seems that there are still OS distributions out there with older versions of ncurses.  Is documenting this bug sufficient?  If so, where?
msg164097 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2012-06-26 20:50
I agree it should be documented, probably wherever unget_wch is documented.

If there is a way to detect the ncurses version, the test should be skipped on < 5.8 as well.
msg166027 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2012-07-21 14:22
Documentation issue, can be resolved after b2.
msg167105 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2012-08-01 09:28
To debug this issue, it would help to have the following information:
 - locale encoding: encoding variable
 - tested character: ch
 - character read by ncurses: read

Can someone reproducing the issue try to add: print("encoding=%s, ch=%r, read=%r" % (encoding, ch, read)) before the error?

It may be an issue in the Python implementation of unget_wch() or get_wch().

--

> I believe the problem is due to a sign-extension bug in the ncurses library unget_wch function (see link below).

Can we workaround this issue in Python? For example, use value & 0xffffff?

--

The test should be modified to use the encoding of stdscr, not the locale encoding: encoding = stdscr.encoding. (In this test, both encodings should be the same.)
msg167563 - (view) Author: Roundup Robot (python-dev) Date: 2012-08-06 13:42
New changeset 8282e4846e43 by Ned Deily in branch 'default':
Issue #15037: Build OS X installers with local copy of ncurses 5.9 libraries
http://hg.python.org/cpython/rev/8282e4846e43
msg167564 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-08-06 13:50
I've updated the OS X installers to build and link with a local copy of ncurses 5.9 rather than older Apple-supplied ones, thus avoiding the library bug.  test_curses now passes for them.
msg168555 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2012-08-19 10:56
Anything left to do here?
msg168586 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-08-19 18:42
To be clear, I've eliminated the problem for the OS X installer builds by supplying newer versions of libncursesw so I wasn't planning on doing anything more on this issue myself.  It should be easy enough to reproduce on most platforms by installing ncursesw 5.7.
msg168977 - (view) Author: Roundup Robot (python-dev) Date: 2012-08-24 07:44
New changeset e587426d719f by Ned Deily in branch 'default':
Issue #15037: Use correct path to system terminfo database.
http://hg.python.org/cpython/rev/e587426d719f
msg169025 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-08-24 14:43
Why is this blocking the release? It looks like a problem that can be solved in a bug fix release, or else by requiring that Python 3.3 users use a recent ncurses release.
msg214583 - (view) Author: Geoffrey Spear (geoffreyspear) * Date: 2014-03-23 14:08
This test still fails in Python 3.5 on Snow Leopard with the system ncurses; it would be nice to at least skip the test on systems with older ncurses.
History
Date User Action Args
2016-09-17 20:39:33berker.peksaglinkissue16293 superseder
2014-03-23 14:08:47geoffreyspearsetnosy: + geoffreyspear

messages: + msg214583
versions: + Python 3.4, Python 3.5
2012-09-03 04:39:53ned.deilylinkissue15854 superseder
2012-08-24 14:43:10loewissetpriority: release blocker -> normal
nosy: + loewis
messages: + msg169025

2012-08-24 07:44:54python-devsetmessages: + msg168977
2012-08-19 18:42:45ned.deilysetmessages: + msg168586
2012-08-19 11:00:37georg.brandlsetpriority: deferred blocker -> release blocker
2012-08-19 10:56:00georg.brandlsetmessages: + msg168555
2012-08-06 13:50:57ned.deilysetmessages: + msg167564
2012-08-06 13:42:43python-devsetnosy: + python-dev
messages: + msg167563
2012-08-01 09:28:50hayposetmessages: + msg167105
2012-07-21 14:22:43georg.brandlsetpriority: release blocker -> deferred blocker

messages: + msg166027
2012-06-26 20:50:16georg.brandlsetpriority: deferred blocker -> release blocker

messages: + msg164097
2012-06-21 20:09:42ned.deilysetpriority: high -> deferred blocker

messages: + msg163361
title: test_curses fails with OverflowError -> curses.unget_wch and test_curses fail when linked with ncurses 5.7 and earlier
2012-06-21 14:30:56jcbollingersetmessages: + msg163336
2012-06-21 14:11:38jcbollingersetmessages: + msg163334
2012-06-21 13:02:33jcbollingersetmessages: + msg163330
2012-06-21 10:34:29ned.deilysetpriority: normal -> high
nosy: + georg.brandl, haypo, ned.deily
messages: + msg163320

2012-06-08 16:08:32jcbollingercreate