classification
Title: Curses Unicode Support
Type: behavior Stage: resolved
Components: Unicode Versions: Python 3.0
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: asmodai, atagar1, vstinner
Priority: normal Keywords:

Created on 2008-12-30 22:57 by atagar1, last changed 2009-05-08 11:28 by asmodai. This issue is now closed.

Messages (14)
msg78563 - (view) Author: Damian (atagar1) Date: 2008-12-30 22:57
Hi, in switching to Python 3.0 I've run into an issue with displaying
Unicode characters via curses. In Python 2.x a simple hello-world looks
like:

#!/usr/bin/python
# coding=UTF-8

import curses
import locale

locale.setlocale(locale.LC_ALL,"")

def doStuff(stdscr):
  message = u"hello わたし!"
  stdscr.addstr(0, 0, message.encode("utf-8"), curses.A_BLINK)
  stdscr.getch() # pauses until a key's hit

curses.wrapper(doStuff)

This works. However, when I try to come up with an equivalent for Python
3.0:

#!/usr/bin/python

import curses
import locale

locale.setlocale(locale.LC_ALL,"")

def doStuff(stdscr):
  message = "hello わたし!"
  stdscr.addstr(0, 0, message, curses.A_BLINK)
  stdscr.getch() # pauses until a key's hit

curses.wrapper(doStuff)

It fails (printing gibberish to the console). It seems that the problem
is that the curses module isn't respecting the system's preferred
encoding (utf-8) which was set via the setlocale function (as per
instructions at
http://docs.python.org/dev/3.0/library/curses.html#module-curses).

My apologies in advance if this is my mistake. Cheers! -Damian
msg78564 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2008-12-30 23:07
I tested your example on Linux (Ubuntu Gutsy) and it works correctly. 
What is your:
- OS (name, version)
- locale (charset?)
msg78566 - (view) Author: Damian (atagar1) Date: 2008-12-30 23:11
My OS is Ubuntu 8.04 (Hardy) and the locale is utf-8:

>>> locale.setlocale(locale.LC_ALL,"")
'en_US.UTF-8'

You do mean the Python 3.0 example didn't work, right? The Python3.0
header is:
Python 3.0 (r30:67503, Dec 21 2008, 02:16:52) 
[GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu3)] on linux2

in case that makes a difference.
msg78567 - (view) Author: Damian (atagar1) Date: 2008-12-30 23:13
Ack - sorry, typo. I meant "You do mean the Python 3.0 example did work,
right?"
msg78568 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2008-12-30 23:14
> You do mean the Python 3.0 example didn't work, right?

I only tested yje Python3 example and it works correctly on my 
computer. I'm using Python3 trunk but I don't think that the curses 
module changed after the 3.0 release.

$ ./python
Python 3.1a0 (py3k:67973M, Dec 28 2008, 17:53:52)
[GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2
>>> import locale; locale.setlocale(locale.LC_ALL, '')
'fr_FR.UTF-8'
msg78569 - (view) Author: Damian (atagar1) Date: 2008-12-30 23:20
Doing a checkout of the trunk - I'll let you know if it works. Thanks!
msg78571 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2008-12-30 23:53
I think that I catched the problem:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=379447

_curses.so should be linked to libncursesw.so.5 and not 
libncurses.so.5. I tested on Hardy and I doesn't work because 
_curses.so was linked to libncurses.so.5.

Few commands...
 sudo apt-get install libncursesw5-dev
 make clean
 make

and it works ;-) Possible workaround: reject non-unicode ncurses 
library in setup.py. The ugly patch:

Index: setup.py
===================================================================
--- setup.py    (révision 67973)
+++ setup.py    (copie de travail)
@@ -848,23 +848,6 @@
             panel_library = 'panelw'
             exts.append( Extension('_curses', ['_cursesmodule.c'],
                                    libraries = curses_libs) )
-        elif (self.compiler.find_library_file(lib_dirs, 'ncurses')):
-            curses_libs = ['ncurses']
-            exts.append( Extension('_curses', ['_cursesmodule.c'],
-                                   libraries = curses_libs) )
-        elif (self.compiler.find_library_file(lib_dirs, 'curses')
-              and platform != 'darwin'):
-                # OSX has an old Berkeley curses, not good enough for
-                # the _curses module.
-            if 
(self.compiler.find_library_file(lib_dirs, 'terminfo')):
-                curses_libs = ['curses', 'terminfo']
-            elif 
(self.compiler.find_library_file(lib_dirs, 'termcap')):
-                curses_libs = ['curses', 'termcap']
-            else:
-                curses_libs = ['curses']
-
-            exts.append( Extension('_curses', ['_cursesmodule.c'],
-                                   libraries = curses_libs) )
         else:
             missing.append('_curses')

Ugly because it may break the curses module compilation on many OS 
(BSD and Solaris?).
msg78572 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2008-12-30 23:56
See also the old (and closed) issue #1428494: Prefer linking against 
ncursesw over ncurses library.
msg78573 - (view) Author: Damian (atagar1) Date: 2008-12-31 00:01
Just finished recompiling and works perfectly. My hat's off to you -
many thanks! -Damian
msg78574 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2008-12-31 00:09
ncursesw looks to be available for:
 - Linux: eg. packaged in Ubunbut
 - NetBSD: 
ftp://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc/devel/ncursesw/README.html
 - FreeBSD: ncurses package with USE="unicode"
 - Mac OS X: http://ncursesw.darwinports.com/
 - Windows: by Cygwin (libncurses5 package)

I think that libncursesw is working well on Linux and BSD, but Windows 
port may be buggy.
msg78576 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2008-12-31 00:13
For Solaris, see this bug report of nano editor:
https://savannah.gnu.org/bugs/?24028

libcurses has been added to OpenSolaris in Septembre 2008. But it 
don't see the unicode version (libncusesw) :-/
msg78588 - (view) Author: Damian (atagar1) Date: 2008-12-31 01:50
Looks like this was my mistake, not a bug. According to:
http://mail.python.org/pipermail/python-list/2007-July/450133.html

Python 2.5 also requires the addition of libcursesw but it was working
for the Ubuntu release because they specifically added it. There's no
missing functionality - just my mistake. Cheers! -Damian
msg78600 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2008-12-31 12:44
> Python 2.5 also requires the addition of libcursesw 
> but it was working for the Ubuntu release because 
> they specifically added it.

What do you mean by "the addition of libcursesw"? _curses.so of Python 
2.5 is linked to libncursesw.so on Ubuntu Gutsy and Hardy.

If _curses.so is linked to libncurses.so, you can not used multibyte 
charset like UTF-8.
msg87436 - (view) Author: Jeroen Ruigrok van der Werven (asmodai) * (Python committer) Date: 2009-05-08 11:28
On FreeBSD there's no need for the ncurses package in most case. Since
somewhere along 6.x and 7.x line Rong-En Fan switched FreeBSD to do what
I did for DragonFly BSD a long time: to have both normal and wide curses
available.

As far as Python is concerned, it supports linking to ncursesw, so I
think this issue can be safely closed since it is not a problem within
Python (as also the original author said).

If this is not ok, feel free to reopen.
History
Date User Action Args
2009-05-08 11:28:27asmodaisetstatus: open -> closed

nosy: + asmodai
messages: + msg87436

resolution: not a bug
stage: resolved
2008-12-31 12:44:49vstinnersetmessages: + msg78600
2008-12-31 01:50:05atagar1setmessages: + msg78588
2008-12-31 00:13:01vstinnersetmessages: + msg78576
2008-12-31 00:09:09vstinnersetmessages: + msg78574
2008-12-31 00:01:01atagar1setmessages: + msg78573
2008-12-30 23:56:42vstinnersetmessages: + msg78572
2008-12-30 23:53:14vstinnersetmessages: + msg78571
2008-12-30 23:20:40atagar1setmessages: + msg78569
2008-12-30 23:14:39vstinnersetmessages: + msg78568
2008-12-30 23:13:46atagar1setmessages: + msg78567
2008-12-30 23:11:58atagar1setmessages: + msg78566
2008-12-30 23:07:53vstinnersetnosy: + vstinner
messages: + msg78564
2008-12-30 22:57:42atagar1create