classification
Title: curses.unget_wch and test_curses fail when linked with ncurses 5.7 and earlier
Type: behavior Stage: resolved
Components: Tests Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: davin, geoffreyspear, georg.brandl, jcbollinger, loewis, ned.deily, python-dev, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2012-06-08 16:08 by jcbollinger, last changed 2017-11-01 17:23 by serhiy.storchaka. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 3826 merged serhiy.storchaka, 2017-09-29 16:50
PR 4218 merged python-dev, 2017-11-01 14:04
Messages (20)
msg162532 - (view) Author: John Bollinger (jcbollinger) * Date: 2012-06-08 16:08
I encountered this test failure while attempting to verify a patch for a separate issue, and I found that it occurs with the unmodified source on the default branch:

LD_LIBRARY_PATH="$PWD" ./python -bb -Wd -m test -r -w -uall -v test_curses
== CPython 3.3.0a4+ (default:4aeb5b9b62d7, Jun 8 2012, 10:23:35) [GCC 4.4.6
20110731 (Red Hat 4.4.6-3)]
==   Linux-2.6.32-220.4.1.el6.x86_64-x86_64-with-centos-6.2-Final
little-endian
==   /home/jbolling/cpython/build/test_python_26873
Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0,
dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0,
verbose=0, bytes_warning=2, quiet=0, hash_randomization=1)
Using random seed 3072318
[1/1] test_curses
test test_curses crashed -- Traceback (most recent call last):
  File "/home/jbolling/cpython/Lib/test/regrtest.py", line 1237, in
runtest_inner
    test_runner()
  File "/home/jbolling/cpython/Lib/test/test_curses.py", line 338, in
test_main
    main(stdscr)
  File "/home/jbolling/cpython/Lib/test/test_curses.py", line 324, in main
    test_unget_wch(stdscr)
  File "/home/jbolling/cpython/Lib/test/test_curses.py", line 283, in
test_unget_wch
    read = chr(read)
OverflowError: signed integer is greater than maximum

1 test failed:
    test_curses
Re-running failed tests in verbose mode
Re-running test 'test_curses' in verbose mode
test test_curses crashed -- Traceback (most recent call last):
  File "/home/jbolling/cpython/Lib/test/regrtest.py", line 1237, in
runtest_inner
    test_runner()
  File "/home/jbolling/cpython/Lib/test/test_curses.py", line 338, in
test_main
    main(stdscr)
  File "/home/jbolling/cpython/Lib/test/test_curses.py", line 324, in main
    test_unget_wch(stdscr)
  File "/home/jbolling/cpython/Lib/test/test_curses.py", line 283, in
test_unget_wch
    read = chr(read)
OverflowError: signed integer is greater than maximum

[123272 refs]


Python was built and the tests run on CentOS 6.2 / x86_64, using the platform's standard tool chain, configured with "--enable-shared --with-threads --with-pydebug".

All other tests pass for me, although test_builtin failed when run as part of the whole suite but passed when run by itself.


For what it's worth, it looks like that particular message is emitted in exactly one place: Python/getargs.c:661 (function convertsimple()), which in this case I guess is being called indirectly from Python/bltinmodule.c:526 (function builtin_chr()).  It's not obvious to me why that would be failing.
msg163320 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-06-21 10:34
I just ran into this problem on another platform.  I believe the problem is due to a sign-extension bug in the ncurses library unget_wch function (see link below).  It was apparently fixed in nurses 5.8; I've tested with the current ncurses 5.9 and test_ncurses now runs without error.

Are you using a version of libncurses/libncursew older than 5.8?  If so, can you try building with 5.8 or 5.9?

Another question is how to handle this for the 3.3.0 release.  It looks like not all platforms have the latest ncurses and, if not, the unget_wch function and test_curses are likely to fail.  Unfortunately, there doesn't seem to be an attribute available in the curses module that gives the version of the underlying curses/ncurses library.  At the least, the problem should probably be documented somewhere.

http://invisible-island.net/ncurses/NEWS.html#t20091010
msg163330 - (view) Author: John Bollinger (jcbollinger) * Date: 2012-06-21 13:02
The system on which I encountered the test failure uses ncurses 5.7, so that's consistent with the theory that the test is tickling an ncurses bug.

I'll have a look at testing with ncurses 5.8, but it is not available from RedHat or CentOS (and it never will be for the current and past versions of those systems), so that's not a good solution for most users.

On the other hand, it's not clear to me how serious is the bug revealed by the test failures, nor whether there is any viable workaround on the Python side.
msg163334 - (view) Author: John Bollinger (jcbollinger) * Date: 2012-06-21 14:11
Clarification: "so that's not a good solution for most users" ... of RedHat-family distros, version 6.2 and earlier.

In fact, it looks like RedHat is sticking with its current version of ncurses for RHEL 6.3, too, so no help is coming from that direction any time soon.
msg163336 - (view) Author: John Bollinger (jcbollinger) * Date: 2012-06-21 14:30
Ok, I confirm that the test passes after the system's ncurses library is upgraded to ncurses 5.8, and fails again when ncurses is downgraded back to version 5.7.
msg163361 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-06-21 20:09
Thanks for the testing.

Georg, haypo: I think a call should be made on what, if anything, to do about this prior to 3.3.0-final.  It seems that there are still OS distributions out there with older versions of ncurses.  Is documenting this bug sufficient?  If so, where?
msg164097 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2012-06-26 20:50
I agree it should be documented, probably wherever unget_wch is documented.

If there is a way to detect the ncurses version, the test should be skipped on < 5.8 as well.
msg166027 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2012-07-21 14:22
Documentation issue, can be resolved after b2.
msg167105 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2012-08-01 09:28
To debug this issue, it would help to have the following information:
 - locale encoding: encoding variable
 - tested character: ch
 - character read by ncurses: read

Can someone reproducing the issue try to add: print("encoding=%s, ch=%r, read=%r" % (encoding, ch, read)) before the error?

It may be an issue in the Python implementation of unget_wch() or get_wch().

--

> I believe the problem is due to a sign-extension bug in the ncurses library unget_wch function (see link below).

Can we workaround this issue in Python? For example, use value & 0xffffff?

--

The test should be modified to use the encoding of stdscr, not the locale encoding: encoding = stdscr.encoding. (In this test, both encodings should be the same.)
msg167563 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-08-06 13:42
New changeset 8282e4846e43 by Ned Deily in branch 'default':
Issue #15037: Build OS X installers with local copy of ncurses 5.9 libraries
http://hg.python.org/cpython/rev/8282e4846e43
msg167564 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-08-06 13:50
I've updated the OS X installers to build and link with a local copy of ncurses 5.9 rather than older Apple-supplied ones, thus avoiding the library bug.  test_curses now passes for them.
msg168555 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2012-08-19 10:56
Anything left to do here?
msg168586 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-08-19 18:42
To be clear, I've eliminated the problem for the OS X installer builds by supplying newer versions of libncursesw so I wasn't planning on doing anything more on this issue myself.  It should be easy enough to reproduce on most platforms by installing ncursesw 5.7.
msg168977 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-08-24 07:44
New changeset e587426d719f by Ned Deily in branch 'default':
Issue #15037: Use correct path to system terminfo database.
http://hg.python.org/cpython/rev/e587426d719f
msg169025 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-08-24 14:43
Why is this blocking the release? It looks like a problem that can be solved in a bug fix release, or else by requiring that Python 3.3 users use a recent ncurses release.
msg214583 - (view) Author: Geoffrey Spear (geoffreyspear) * Date: 2014-03-23 14:08
This test still fails in Python 3.5 on Snow Leopard with the system ncurses; it would be nice to at least skip the test on systems with older ncurses.
msg303318 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-09-29 09:23
Seems there is the same cause of the failure of test_curses on OpenBSD.

======================================================================
ERROR: test_window_funcs (test.test_curses.TestCurses) (meth='window.getkey')
Test the methods of windows
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/serhiy/py/cpython3.7/Lib/test/test_curses.py", line 99, in test_window_funcs
    meth()
ValueError: chr() arg not in range(0x110000)

======================================================================
FAIL: test_unget_wch (test.test_curses.TestCurses)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/serhiy/py/cpython3.7/Lib/test/test_curses.py", line 355, in test_unget_wch
    self.assertEqual(read, ch)
AssertionError: -61 != 'é'

----------------------------------------------------------------------

System ncurses version is 5.7.
msg303342 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-09-29 16:54
It is easy to write a workaround for the first case (but it is not ). The workaround for the second case would be too complex. I prefer to skip the test. Unfortunately the version of ncurses is not exposed on Python level, thus the skipping is OS-specific.
msg305383 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-11-01 14:03
New changeset 7e68790f3db75a893d5dd336e6201a63bc70212b by Serhiy Storchaka in branch 'master':
bpo-15037: Add a workaround for getkey() in curses for ncurses 5.7 and earlier. (#3826)
https://github.com/python/cpython/commit/7e68790f3db75a893d5dd336e6201a63bc70212b
msg305384 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-11-01 14:38
New changeset 1f81ea85e8e20347ec396001e5b869d36fe38398 by Serhiy Storchaka (Miss Islington (bot)) in branch '3.6':
bpo-15037: Add a workaround for getkey() in curses for ncurses 5.7 and earlier. (GH-3826) (#4218)
https://github.com/python/cpython/commit/1f81ea85e8e20347ec396001e5b869d36fe38398
History
Date User Action Args
2017-11-01 17:23:58serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2017-11-01 14:38:37serhiy.storchakasetmessages: + msg305384
2017-11-01 14:04:48python-devsetpull_requests: + pull_request4186
2017-11-01 14:03:42serhiy.storchakasetmessages: + msg305383
2017-09-29 16:54:29serhiy.storchakasetmessages: + msg303342
2017-09-29 16:50:22serhiy.storchakasetkeywords: + patch
stage: patch review
pull_requests: + pull_request3810
2017-09-29 09:44:37serhiy.storchakasetnosy: + davin
2017-09-29 09:23:46serhiy.storchakasetnosy: + serhiy.storchaka

messages: + msg303318
versions: + Python 3.6, Python 3.7, - Python 3.3, Python 3.4, Python 3.5
2016-09-17 20:39:33berker.peksaglinkissue16293 superseder
2014-03-23 14:08:47geoffreyspearsetnosy: + geoffreyspear

messages: + msg214583
versions: + Python 3.4, Python 3.5
2012-09-03 04:39:53ned.deilylinkissue15854 superseder
2012-08-24 14:43:10loewissetpriority: release blocker -> normal
nosy: + loewis
messages: + msg169025

2012-08-24 07:44:54python-devsetmessages: + msg168977
2012-08-19 18:42:45ned.deilysetmessages: + msg168586
2012-08-19 11:00:37georg.brandlsetpriority: deferred blocker -> release blocker
2012-08-19 10:56:00georg.brandlsetmessages: + msg168555
2012-08-06 13:50:57ned.deilysetmessages: + msg167564
2012-08-06 13:42:43python-devsetnosy: + python-dev
messages: + msg167563
2012-08-01 09:28:50vstinnersetmessages: + msg167105
2012-07-21 14:22:43georg.brandlsetpriority: release blocker -> deferred blocker

messages: + msg166027
2012-06-26 20:50:16georg.brandlsetpriority: deferred blocker -> release blocker

messages: + msg164097
2012-06-21 20:09:42ned.deilysetpriority: high -> deferred blocker

messages: + msg163361
title: test_curses fails with OverflowError -> curses.unget_wch and test_curses fail when linked with ncurses 5.7 and earlier
2012-06-21 14:30:56jcbollingersetmessages: + msg163336
2012-06-21 14:11:38jcbollingersetmessages: + msg163334
2012-06-21 13:02:33jcbollingersetmessages: + msg163330
2012-06-21 10:34:29ned.deilysetpriority: normal -> high
nosy: + georg.brandl, vstinner, ned.deily
messages: + msg163320

2012-06-08 16:08:32jcbollingercreate