The control-spacebar binding is used in IDLE to 
force open the completions window.  It's causing 
IDLE to exit with a utf8 decode error.  Attached 
is 
a Tkinter cut-down 
exhibiting the problem and a patch.

The cutdown runs ok on 2.6 but not on py3k because 
the latter uses PyUnicode_FromString on all the 
arguments and errs out when it encounters a 
character outside the utf-8 range.

Strangely, on my system, control-spacebar is 
sending a two byte 
string, C0E8 via the %A parameter.  Control-2 
does 
the same.  Other keys with combinations of 
modifier 
keys send one byte.

Linux trader 2.6.18-ARCH #1 SMP PREEMPT Sun Nov 
19 
09:14:35 CET 2006 i686 Intel(R) Pentium(R) 4 CPU 
2.40GHz GenuineIntel GNU/Linux

Can the problem be confirmed?

Using PyUnicode_FromUnicode on %A works because 
the 
unicode string is copied instead of decoded, and 
that parameter is supposed to be unicode, in any 
case.

The patch fixes the problem on my system but 
should 
be reviewed, especially whether the cast in the 
call 
to PyUnicode_FromUnicode is suitably cross-
platform.

Assigning to Neal since he's working a lot of 
Unicode issues right now.  I can check it in if I 
get approval.

Heh, I see we have the same damn problem SF had: when a comment is 
edited, 
it doesn't re-wrap properly when submitted.  You have to remove the 
returns 
manually after editing.

Nope, you have to make sure not to type too wide.

Well, maybe someday Tk will send a multibyte unicode
character.  Update the patch.

I can confirm the problem and that your patch fixes the problem.  Go
ahead and check it in.  Thanks!

OK, thanks for the review! I suppose Tk is sending a bad string.
r57540

Sorry, I reverted r57540 because it caused segfault at IDLE exit. (See
issue4313) I reopened this issue.

I can reproduce it here with tk8.4, using tk8.5 doesn't cause this.

Here is a patch that doesn't use magic numbers :P
I didn't hit the problem described in issue4313 with this one, and
PythonCmd should be doing this anyway, but ideally we should move to
Tcl_CreateObjCommand.

Removed some repeated code in the patch

I confirmed PythonCmd_check_for_utf.diff worked on my machine. IDLE
didn't crash.

>I can reproduce it here with tk8.4, using tk8.5 doesn't cause this.

That is, this is a bug of tk8.4, and solved in tk8.5 which is already
stable release? If so, I feel python don't have to workaround this bug.

# I'm a little worry about performance because Tcl_NumUtfChars() will be
called for every command string.

By the way, I cannot reproduce this bug with tk8.4.12(on windows). What
is your tk version? Maybe older than that?

tk 8.4.19 here, but windows and linux almost surely uses different
window managers (you could run gnome and others under windows, but I'm
betting it is not the case).

Now, it is very hard to say that we shouldn't care about this bug here.
Tcl has it documented that its string arguments to Tcl_CmdProc are
encoded in normalized utf-8 since tcl 8.1 which was released almost 10
years ago. I guess we are just luck that this was the first time the bug
was noticed.

It also says that Tcl_CreateCommand shouldn't be used anymore, instead
Tcl_CreateObjCommand should be used like I said in the previous comment.

I suceeded to reproduce this issue with coLinux + UltraVNC on Win2000.

Yes, py3k claimed utf-8 error, so I tried trunk. Here is result.

*** event.keycode:  8
*** event.state:  0
*** event.char:  ''

*** event.keycode:  16
*** event.state:  4
*** event.char:  '\xc0\x80'

This '\xc0\x80' seems to be used in tcl as null byte '\0'. You can see
this magic value in tcl source and google.

I think we should convert this to '\x00' at python side. (shouldn't
treat this as utf-16)

I can see py3k + adhok.patch can output this result.

*** event.keycode:  8
*** event.state:  0
*** event.char:  ''

*** event.keycode:  16
*** event.state:  4
*** event.char:  '\x00'

Probably Tcl_GetUnicode does this conversion inside. (I'm not sure,
because I didn't look into source code so deeply) And I'm not sure why
this error doesn't happen with tk8.5.

I did little modification to tkintertest.py. Please use this line.

    my_print("*** event.char: ", repr(event.char))

You are missing the point on using Tcl_CreateObjCommand, I didn't mean
to just go and and do s/Tcl_CreateCommand/Tcl_CreateObjCommand/ because
if you are going to convert everything to unicode then there is no point
in using Tcl_CreateObjCommand.
Also, Tcl_ObjCmdProc should use Tcl_Obj *CONST objv[] instead of Tcl_Obj
*const objv[] because Tcl may define CONST as nothing, and it uses CONST
when defining Tcl_ObjCmdProc.

I'm sorry if it sounded like I were bashing you, I was just pointing out
my view of the patch -- you didn't need to remove it. The patch I
submitted here can also be improved (although it "works"), but I'm
leaving it as a possible idea for someone else that might look into
this, since I can't invest much time into this right now.

>You are missing the point on using Tcl_CreateObjCommand, I didn't mean
>to just go and and do s/Tcl_CreateCommand/Tcl_CreateObjCommand/ because
>if you are going to convert everything to unicode then there is no
>point in using Tcl_CreateObjCommand.

I'm not tcl/tk expert, so probably missng many things. :-(
Can you explain how to solve this issue by moving to Tcl_CreateObjCommand?

>Also, Tcl_ObjCmdProc should use Tcl_Obj *CONST objv[] instead of
>Tcl_Obj *const objv[] because Tcl may define CONST as nothing, and it
>uses CONST when defining Tcl_ObjCmdProc.

I created adhok.patch just for explanation. This is not solution. I used
Tcl_CreateObjCommand + Tcl_GetUnicode to demonstrate Tcl converts
'\xc0\x80' to null byte. (adhok.patch contained Japanese characters, so
I'll repost that as just_for_explanation.patch)

> Hirokazu Yamamoto added the comment:
>
>>You are missing the point on using Tcl_CreateObjCommand, I didn't mean
>>to just go and and do s/Tcl_CreateCommand/Tcl_CreateObjCommand/ because
>>if you are going to convert everything to unicode then there is no
>>point in using Tcl_CreateObjCommand.
>
> I'm not tcl/tk expert, so probably missng many things. :-(
> Can you explain how to solve this issue by moving to Tcl_CreateObjCommand?
>

By moving to Tcl_CreateObjCommand we would start using the FromObj
function present in _tkinter.c that is responsible for converting tcl
objects to python objects. Then what remains to be verified is how
compatible this would be with current tkinter code, and checking how
correct FromObj is nowadays.

Some more clarifications about this bug:

Tcl shouldn't be giving us a UTF-8 string with a 0xC0 byte, since that
is not valid UTF-8. I'm aware that Tcl uses the sequence 0xC0 0x80 for
special purposes but it is also said that such sequences shouldn't be
passed as is when exported.

This bug doesn't affect python 2.x because it uses PyString_FromString
to convert such value to a Python string, where python 3.x uses
PyUnicode_FromString which assumes that it is receiving a valid utf-8
string but it turns out that is not always the case here.

It is indeed related to tk 8.4, but not sure which ones exactly (I hit
it with tk 8.4.19).

I've been working on a new _tkinter (named it as "plumage") these days
and I hit this same problem for trusting too much that nothing from tcl,
including tk and extensions, would give me this embedded null.

Checking another bridge to Tcl (one done for Perl) it is possible to
notice that it also chose to verify for these bytes and convert them to
something else, a 0. The code for this for Python can be found at
http://code.google.com/p/plumage/source/browse/trunk/src/utils.c#42 up
to line 76, it could/should be adapted to the _tkinter in py3k and also
for python 2.x.

This problem exists for me on Ubuntu8.04 with both tk/tcl8.4.16 and 8.5.

Can you tell what:

print(tkinter.Tcl().tk.call('info', 'patchlevel'))

prints ? Specifically to know which tk 8.5.x has the problem.

8.5.0
This is still an issue with both tk versions in the 3.0.1 python release.

More users reported this problem in #6144 and #6512.

I have the problem described in issue6512 and here is some information 

Python version - hand compiled on Ubuntu 9.04:
Python 3.1 (r31:73572, Jul 18 2009, 11:13:40) 
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import tkinter
>>> print(tkinter.Tcl().tk.call('info', 'patchlevel'))
8.5.6

The previous Tcl/Tk version I had initially installed was a 8.4.x
version - which fails on Control-2 error described in detail in issue6512. 
- I decided to upgrade Tcl/Tk to version 8.5. 
- So I made a "make distclean" 
- copied the contents of /usr/include/tcl85 one level higher, so Python
could access the necessary tk.h and tcl.h files
- cd to my Python 3.1 source
- ./configure
- make 
- sudo make install

and tested again and my test program is now happily responding to a
Control-2 keystroke.

Attaching a patch against trunk, I believe this solves the problems
described here.

Uhm, in the long run I believe it will be better to move to
Tcl_CreateObjCommand since it is said that commands created by it are
significantly faster than the ones created by Tcl_CreateCommand (more
information about this can be found at tcl documentation).

I'm only writing this because, as other places that deal with tcl
objects, more care must be taken. For instance, I have applied the
issue1028.diff on the tk_and_idle_maintenance branch and found two
problems that are now patched by adjusts1.diff. It is very likely that
there are other bugs around, I'll be trying some tkinter applications to
try to find some of them but help is very much needed. Note that there
are some tkinter tests on this tk_and_idle_maintenance and they all
pass, but they do not fully cover tkinter at this moment so improving
them would be good too.

Today I noticed the StringObj manpage (from tcl) says that the bytes
that represent an tcl object should be treated as read-only (although it
uses char *) so this issue1028.diff may very well cause a segfault at
some point.

I'm attaching a new patch that fixes this and also uses
Tcl_GetStringFromObj, instead of directly accessing the bytes member of
a tcl object, so we know its string representation is not invalid.

Isn't this better implemented via a codec error handler?