The control-spacebar binding is used in IDLE to 
force open the completions window.  It's causing 
IDLE to exit with a utf8 decode error.  Attached 
is 
a Tkinter cut-down 
exhibiting the problem and a patch.

The cutdown runs ok on 2.6 but not on py3k because 
the latter uses PyUnicode_FromString on all the 
arguments and errs out when it encounters a 
character outside the utf-8 range.

Strangely, on my system, control-spacebar is 
sending a two byte 
string, C0E8 via the %A parameter.  Control-2 
does 
the same.  Other keys with combinations of 
modifier 
keys send one byte.

Linux trader 2.6.18-ARCH #1 SMP PREEMPT Sun Nov 
19 
09:14:35 CET 2006 i686 Intel(R) Pentium(R) 4 CPU 
2.40GHz GenuineIntel GNU/Linux

Can the problem be confirmed?

Using PyUnicode_FromUnicode on %A works because 
the 
unicode string is copied instead of decoded, and 
that parameter is supposed to be unicode, in any 
case.

The patch fixes the problem on my system but 
should 
be reviewed, especially whether the cast in the 
call 
to PyUnicode_FromUnicode is suitably cross-
platform.

Assigning to Neal since he's working a lot of 
Unicode issues right now.  I can check it in if I 
get approval.

Heh, I see we have the same damn problem SF had: when a comment is 
edited, 
it doesn't re-wrap properly when submitted.  You have to remove the 
returns 
manually after editing.

Nope, you have to make sure not to type too wide.

Well, maybe someday Tk will send a multibyte unicode
character.  Update the patch.

I can confirm the problem and that your patch fixes the problem.  Go
ahead and check it in.  Thanks!

OK, thanks for the review! I suppose Tk is sending a bad string.
r57540

Sorry, I reverted r57540 because it caused segfault at IDLE exit. (See
issue4313) I reopened this issue.

I can reproduce it here with tk8.4, using tk8.5 doesn't cause this.

Here is a patch that doesn't use magic numbers :P
I didn't hit the problem described in issue4313 with this one, and
PythonCmd should be doing this anyway, but ideally we should move to
Tcl_CreateObjCommand.

Removed some repeated code in the patch

I confirmed PythonCmd_check_for_utf.diff worked on my machine. IDLE
didn't crash.

>I can reproduce it here with tk8.4, using tk8.5 doesn't cause this.

That is, this is a bug of tk8.4, and solved in tk8.5 which is already
stable release? If so, I feel python don't have to workaround this bug.

# I'm a little worry about performance because Tcl_NumUtfChars() will be
called for every command string.

By the way, I cannot reproduce this bug with tk8.4.12(on windows). What
is your tk version? Maybe older than that?

tk 8.4.19 here, but windows and linux almost surely uses different
window managers (you could run gnome and others under windows, but I'm
betting it is not the case).

Now, it is very hard to say that we shouldn't care about this bug here.
Tcl has it documented that its string arguments to Tcl_CmdProc are
encoded in normalized utf-8 since tcl 8.1 which was released almost 10
years ago. I guess we are just luck that this was the first time the bug
was noticed.

It also says that Tcl_CreateCommand shouldn't be used anymore, instead
Tcl_CreateObjCommand should be used like I said in the previous comment.

I suceeded to reproduce this issue with coLinux + UltraVNC on Win2000.

Yes, py3k claimed utf-8 error, so I tried trunk. Here is result.

*** event.keycode:  8
*** event.state:  0
*** event.char:  ''

*** event.keycode:  16
*** event.state:  4
*** event.char:  '\xc0\x80'

This '\xc0\x80' seems to be used in tcl as null byte '\0'. You can see
this magic value in tcl source and google.

I think we should convert this to '\x00' at python side. (shouldn't
treat this as utf-16)

I can see py3k + adhok.patch can output this result.

*** event.keycode:  8
*** event.state:  0
*** event.char:  ''

*** event.keycode:  16
*** event.state:  4
*** event.char:  '\x00'

Probably Tcl_GetUnicode does this conversion inside. (I'm not sure,
because I didn't look into source code so deeply) And I'm not sure why
this error doesn't happen with tk8.5.

I did little modification to tkintertest.py. Please use this line.

    my_print("*** event.char: ", repr(event.char))

You are missing the point on using Tcl_CreateObjCommand, I didn't mean
to just go and and do s/Tcl_CreateCommand/Tcl_CreateObjCommand/ because
if you are going to convert everything to unicode then there is no point
in using Tcl_CreateObjCommand.
Also, Tcl_ObjCmdProc should use Tcl_Obj *CONST objv[] instead of Tcl_Obj
*const objv[] because Tcl may define CONST as nothing, and it uses CONST
when defining Tcl_ObjCmdProc.

I'm sorry if it sounded like I were bashing you, I was just pointing out
my view of the patch -- you didn't need to remove it. The patch I
submitted here can also be improved (although it "works"), but I'm
leaving it as a possible idea for someone else that might look into
this, since I can't invest much time into this right now.

>You are missing the point on using Tcl_CreateObjCommand, I didn't mean
>to just go and and do s/Tcl_CreateCommand/Tcl_CreateObjCommand/ because
>if you are going to convert everything to unicode then there is no
>point in using Tcl_CreateObjCommand.

I'm not tcl/tk expert, so probably missng many things. :-(
Can you explain how to solve this issue by moving to Tcl_CreateObjCommand?

>Also, Tcl_ObjCmdProc should use Tcl_Obj *CONST objv[] instead of
>Tcl_Obj *const objv[] because Tcl may define CONST as nothing, and it
>uses CONST when defining Tcl_ObjCmdProc.

I created adhok.patch just for explanation. This is not solution. I used
Tcl_CreateObjCommand + Tcl_GetUnicode to demonstrate Tcl converts
'\xc0\x80' to null byte. (adhok.patch contained Japanese characters, so
I'll repost that as just_for_explanation.patch)

> Hirokazu Yamamoto added the comment:
>
>>You are missing the point on using Tcl_CreateObjCommand, I didn't mean
>>to just go and and do s/Tcl_CreateCommand/Tcl_CreateObjCommand/ because
>>if you are going to convert everything to unicode then there is no
>>point in using Tcl_CreateObjCommand.
>
> I'm not tcl/tk expert, so probably missng many things. :-(
> Can you explain how to solve this issue by moving to Tcl_CreateObjCommand?
>

By moving to Tcl_CreateObjCommand we would start using the FromObj
function present in _tkinter.c that is responsible for converting tcl
objects to python objects. Then what remains to be verified is how
compatible this would be with current tkinter code, and checking how
correct FromObj is nowadays.

Some more clarifications about this bug:

Tcl shouldn't be giving us a UTF-8 string with a 0xC0 byte, since that
is not valid UTF-8. I'm aware that Tcl uses the sequence 0xC0 0x80 for
special purposes but it is also said that such sequences shouldn't be
passed as is when exported.

This bug doesn't affect python 2.x because it uses PyString_FromString
to convert such value to a Python string, where python 3.x uses
PyUnicode_FromString which assumes that it is receiving a valid utf-8
string but it turns out that is not always the case here.

It is indeed related to tk 8.4, but not sure which ones exactly (I hit
it with tk 8.4.19).

I've been working on a new _tkinter (named it as "plumage") these days
and I hit this same problem for trusting too much that nothing from tcl,
including tk and extensions, would give me this embedded null.

Checking another bridge to Tcl (one done for Perl) it is possible to
notice that it also chose to verify for these bytes and convert them to
something else, a 0. The code for this for Python can be found at
http://code.google.com/p/plumage/source/browse/trunk/src/utils.c#42 up
to line 76, it could/should be adapted to the _tkinter in py3k and also
for python 2.x.

This problem exists for me on Ubuntu8.04 with both tk/tcl8.4.16 and 8.5.

Can you tell what:

print(tkinter.Tcl().tk.call('info', 'patchlevel'))

prints ? Specifically to know which tk 8.5.x has the problem.

8.5.0
This is still an issue with both tk versions in the 3.0.1 python release.

More users reported this problem in #6144 and #6512.

I have the problem described in issue6512 and here is some information 

Python version - hand compiled on Ubuntu 9.04:
Python 3.1 (r31:73572, Jul 18 2009, 11:13:40) 
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import tkinter
>>> print(tkinter.Tcl().tk.call('info', 'patchlevel'))
8.5.6

The previous Tcl/Tk version I had initially installed was a 8.4.x
version - which fails on Control-2 error described in detail in issue6512. 
- I decided to upgrade Tcl/Tk to version 8.5. 
- So I made a "make distclean" 
- copied the contents of /usr/include/tcl85 one level higher, so Python
could access the necessary tk.h and tcl.h files
- cd to my Python 3.1 source
- ./configure
- make 
- sudo make install

and tested again and my test program is now happily responding to a
Control-2 keystroke.

Attaching a patch against trunk, I believe this solves the problems
described here.

Uhm, in the long run I believe it will be better to move to
Tcl_CreateObjCommand since it is said that commands created by it are
significantly faster than the ones created by Tcl_CreateCommand (more
information about this can be found at tcl documentation).

I'm only writing this because, as other places that deal with tcl
objects, more care must be taken. For instance, I have applied the
issue1028.diff on the tk_and_idle_maintenance branch and found two
problems that are now patched by adjusts1.diff. It is very likely that
there are other bugs around, I'll be trying some tkinter applications to
try to find some of them but help is very much needed. Note that there
are some tkinter tests on this tk_and_idle_maintenance and they all
pass, but they do not fully cover tkinter at this moment so improving
them would be good too.

Today I noticed the StringObj manpage (from tcl) says that the bytes
that represent an tcl object should be treated as read-only (although it
uses char *) so this issue1028.diff may very well cause a segfault at
some point.

I'm attaching a new patch that fixes this and also uses
Tcl_GetStringFromObj, instead of directly accessing the bytes member of
a tcl object, so we know its string representation is not invalid.

Isn't this better implemented via a codec error handler?

I can confirm that it works on Windows (using 3.1.1).

exists with 3.1.2 and tcl 8.5.8

This has been reported in #9231 (and #6144, #6512, #7884, #6920, #6424, #5156) too.

Could someone with commit privileges please review the patch with a view to committing, thanks.

Python 3.1.2, Ubuntu (Lucid)

Caused by Control + Shift + Spacebar 

Debugger output from terminal: 

Traceback (most recent call last):
  File "/usr/bin/idle-python3.1", line 5, in <module>
    main()
  File "/usr/lib/python3.1/idlelib/PyShell.py", line 1420, in main
    root.mainloop()
  File "/usr/lib/python3.1/tkinter/__init__.py", line 1012, in mainloop
    self.tk.mainloop(n)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0-1: illegal encoding

Will try the patch.

Do we know the status of this issue? Have not seen update in four months. Currently is a major headache on my linux box.

Nudge: report on the Ubuntu bug tracker that this is still an issue with 3.2:

https://bugs.launchpad.net/bugs/517552

Tcl/Tk uses modified utf-8 internally.  This includes using 0xC080, a multibyte Unicode null character, for embedded nulls that work with C's null terminated strings.  Java does the same.

Note that typing Ctrl-space and Ctrl-2 are conventional ways to enter a null from the keyboard.  That's the reason a null char is associated with those key combinations.

When Tcl exports Unicode, it is supposed to be strict utf-8.  Until Tcl8.5, the %A (Unicode character corresponding to an event) was incorrectly leaking the modified Unicode null.

_tkinter.c.2.patch is narrowly focused: if PythonCmd raises a UnicodeDecodeError and if the string passed in an arg is 0xC080, it is replaced with the Unicode null 0x00.

New changeset 82cfbe2ddfbb by Kurt B. Kaiser in branch '3.1':
Issue #1028: Tk returns invalid Unicode null in %A: UnicodeDecodeError.
http://hg.python.org/cpython/rev/82cfbe2ddfbb

I'm working on #2857 which adds the "Modified UTF-8" ("utf-8-java"?) codec to Python. We can maybe use it instead of raising an error in 3.3?

r70039 3.1 forward ported > 3.2 > default.  Will be in 3.2.1.

Having a modified utf-8 codec will be useful.  That said, it is an error
for Tcl/Tk to expose modified utf-8 externally, and that was fixed at
some point in Tk8.5.  Since Tk is no longer sending 0xC080 for the %A
char, switching codecs in _tkinter.c won't accomplish anything.

This fix was to correct a long-standing problem in IDLE using Tk8.4,
which is most easily solved by catching the leaked invalid null in
_tkinter.c.

It seems to me that, once you switch to modified utf-8 and allow the
embedded nulls, you have to make sure everything you are doing uses
the modified utf-8 encoding/decoding.

Hello,

I am still able to reproduce this issue with Python 3.2.2. It seems as if this bug was closed with a the note:

r70039 3.1 forward ported > 3.2 > default.  Will be in 3.2.1.

This leads me to believe that either 3.2.2 has a regression or the patch never made it into 3.2.1. 

Can anyone chime in with some more details?

Thanks,

Mike

Figured I should capture the exception. See below.

3.2.2+ (default, Jan  8 2012, 07:22:26) 
[GCC 4.6.2]

Traceback (most recent call last):
  File "/usr/bin/idle3", line 5, in <module>
    main()
  File "/usr/lib/python3.2/idlelib/PyShell.py", line 1429, in main
    root.mainloop()
  File "/usr/lib/python3.2/tkinter/__init__.py", line 1012, in mainloop
    self.tk.mainloop(n)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc0 in position 0: invalid start byte

On Win 7, a<cntl-space> brings up the box on all the latest releases: 2.7.3, 3.2.3, and 3.3.0a3. (These all come with recent tk 8.5.x.)

Mike, please retest with 3.2.3 and specify os and tk version and exactly what you entered if there is still a problem.

Looking good in 3.2.3! Tested on Debian Wheezy using packages
python3-tk 3.2.3-1 and idle3 3.2.3~rc1-2.

/*
 * Mike Perry
 * mike@cogs....com
 */

On Sun, May 27, 2012 at 4:10 PM, Terry J. Reedy <report@bugs.python.org> wrote:
>
> Terry J. Reedy <tjreedy@udel.edu> added the comment:
>
> On Win 7, a<cntl-space> brings up the box on all the latest releases: 2.7.3, 3.2.3, and 3.3.0a3. (These all come with recent tk 8.5.x.)
>
> Mike, please retest with 3.2.3 and specify os and tk version and exactly what you entered if there is still a problem.
>
> ----------
> nosy: +terry.reedy
> versions:  -Python 3.1
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue1028>
> _______________________________________