Это из IDLE:
Python 2.7.2+ (default, Oct  4 2011, 20:03:08) 
[GCC 4.6.1] on linux2
Type "copyright", "credits" or "license()" for more information.
>>> import sys
>>> import locale
>>> sys.getdefaultencoding()
'ascii'
>>> locale.getpreferredencoding()
'UTF-8'
>>> s = u'Русский текст'
>>> print s.encode('utf_8')
Русский текст
>>> print s.encode('latin1')
Русский текст

Это из консоли:
>>> import sys
>>> import locale
>>> sys.getdefaultencoding()
'ascii'
>>> locale.getpreferredencoding()
'UTF-8'
>>> s = u'Русский текст'
>>> print s.encode('utf_8')
Русский текст

Looks like your IDLE is set to use latin-1, but s.encode('latin1') should have failed with an UnicodeEncodeError.  Are you sure you copied the snippet correctly?
The default source encoding refers to the .py files you open with IDLE, and doesn't affect the encoding used by IDLE itself while printing.

(Please use English to report issues.)

Text is correct. See attachment.

The problem is that IDLE passes an UTF-8 encoded source string to compile, and compile, in the absence of a source encoding, uses the PEP 263 default source encoding, i.e. Latin-1.

As the consequence, the variable s has the value

u'\\xd0\\xa0\\xd1\\x83\\xd1\\x81\\xd1\\x81\\xd0\\xba\\xd0\\xb8\\xd0\\xb9 \\xd1\\x82\\xd0\\xb5\\xd0\\xba\\xd1\\x81\\xd1\\x82'

IDLE's "Default Source Encoding" is irrelevant - it only applies to editor windows.

One solution for that is the attached patch. However, this patch isn't right, since it will cause all source to be interpreted as UTF-8. This would be wrong when the sys.stdin.encoding is not UTF-8, and byte string objects are created in interactive mode.

Interactive mode manages to get it right by looking up sys.stdin.encoding during compilation, but it does so only when in interactive mode (i.e. when tok->prompt != NULL.

I don't see any way to fix this problem in Python 2. It is fixed in Python 3, basically by always assuming that the source encoding is UTF-8, by making all string objects Unicode objects, and disallowing non-ASCII characters in bytes literals

> However, this patch isn't right, since it will cause all source to be interpreted as UTF-8. This would be wrong when the sys.stdin.encoding is not UTF-8, and byte string objects are created in interactive mode.

Can you show how to reproduce the error that you're talking about?  I have found no issues running the bare Python and IDLE (with your patch, of course) with files in different encodings under different locales.

The problem (as I understand it) is that if Martin's patch fixes an unicode literals, it breaks a string literals.

LC_ALL=ru_RU.cp1251 LANG=ru_RU.cp1251 ./python Lib/idlelib/idle.py

>>> print u'йцук'
йцук
>>> print 'йцук'
йцук

Here is a different patch, which fixes unicode strings and preserve byte strings.

>>> print u'йцук'
йцук
>>> print 'йцук'
йцук

I've combined the nosy lists and will close the other issue.  Serhiy, what if the source already has a coding cookie line?

A coding cookie line will be ignored. Note that this affects only very obscure case when you are pasting a multiline code with a coding cookie line into IDLE shell (this is only way to get a coding cookie line in the shell). Running a file does not pass through runsource(). runsource() used only for a code entered in the shell (may be Martin or Roger correct me if I wrong)

An interactive Python console ignores a coding cookie line too.

$ cat test.py
# -*- coding: koi8-r -*-
print repr('йцук'), 'йцук', repr(u'йцук'), u'йцук'
$ LC_ALL=ru_RU.cp1251 LANG=ru_RU.cp1251 ./python test.py
'\xe9\xf6\xf3\xea' йцук u'\u0418\u0416\u0421\u0419' ИЖСЙ
$ LC_ALL=ru_RU.cp1251 LANG=ru_RU.cp1251 ./python
Python 2.7.4+ (2.7:0f31f38e8a17+, Apr 13 2013, 21:06:36) 
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> # -*- coding: koi8-r -*-
... print repr('йцук'), 'йцук', repr(u'йцук'), u'йцук'
'\xe9\xf6\xf3\xea' йцук u'\u0439\u0446\u0443\u043a' йцук

('\xe9\xf6\xf3\xea' is 'йцук' in cp1251 and 'ИЖСЙ' in koi8-r)

Here's a tangentially related issue: #14326

IDLE doesn't handle pasting multi-line code properly (issue3559), IDLE2 will silently ignore code after the first executable statement. IDLE3 may give an error.

Can't we just make IDLE's shell default to UTF-8?

> Here's a tangentially related issue: #14326

Yes, this issue looks as a prerequisite for it.

> IDLE doesn't handle pasting multi-line code properly (issue3559), IDLE2 will silently ignore code after the first executable statement. IDLE3 may give an error.

Well, then the patch doesn't introduce a significant regression.

> Can't we just make IDLE's shell default to UTF-8?

This is easy. Just set IOBinding.encoding to "utf-8". But I'm not sure we don't lost something in this case.

If there are no objections I will commit the patch soon.

There is a problem. Adding the encoding comment to the top of the source causes off-by-one line errors in the traceback.

Take as an example:

>>> 1/0

Traceback (most recent call last):
  File "<pyshell#0>", line 2, in <module>
ZeroDivisionError: integer division or modulo by zero
>>>

Good catch. Thank you, Roger.

How about passing on UnicodeError? Will it cause other problems?

Sorry, it's probably has the same problem as what Martin suggested. Although I think it's better than having program interrupted.

> How about passing on UnicodeError?

I don't see how it will resolve any problem.

Oops! My problem is that I get "Unsupported characters in input" when trying to type any Unicode string in IDLE's interactive mode.

e.g. a simple command like: s = u'Русский текст' gives: Unsupported characters in input

Probably unrelated to the issue at hand. I don't know how I ended up in this thread. Again, sorry.

I really think this information might help, if not, I promise not to post anything else. :)

This is a sample program I run:

'''
# -*- coding: utf-8 -*-
import sys
import locale

de = sys.getdefaultencoding()
pd = locale.getpreferredencoding()
print de, pd

s1 = 'سلام'
print s1
s2 = u'سلام'
print s2
'''

I tried to run it before and after applying suggested patches.
Before applying any patch:

'''
Python 2.7.5 (default, May 15 2013, 22:44:16) [MSC v.1500 64 bit (AMD64)] on win32
Type "copyright", "credits" or "license()" for more information.
>>> ================================ RESTART ================================
>>> 
ascii cp1256
ط³ظ„ط§ظ…
سلام
>>> s3 = 'سلام'
>>> s4 = u'سلام'
>>> s3
'\xd3\xe1\xc7\xe3'
>>> s4
u'\xd3\xe1\xc7\xe3'
>>> print s3
سلام
>>> print s4
ÓáÇã
>>> s = u'Русский текст'
Unsupported characters in input
'''

After applying loewis's patch:

'''
Python 2.7.5 (default, May 15 2013, 22:44:16) [MSC v.1500 64 bit (AMD64)] on win32
Type "copyright", "credits" or "license()" for more information.
>>> ================================ RESTART ================================
>>> 
ascii cp1256
ط³ظ„ط§ظ…
سلام
>>> s3 = 'سلام'
>>> s4 = u'سلام'
>>> s3
'\xd8\xb3\xd9\x84\xd8\xa7\xd9\x85'
>>> s4
u'\u0633\u0644\u0627\u0645'
>>> print s3
ط³ظ„ط§ظ…
>>> print s4
سلام
>>> s = u'Русский текст'
>>> print s
Русский текст
>>> 
'''

After applying serhiy.storchaka's patch:

'''
Python 2.7.5 (default, May 15 2013, 22:44:16) [MSC v.1500 64 bit (AMD64)] on win32
Type "copyright", "credits" or "license()" for more information.
>>> ================================ RESTART ================================
>>> 
ascii cp1256
ط³ظ„ط§ظ…
سلام
>>> s3 = 'سلام'
>>> s4 = u'سلام'
>>> s3
'\xd3\xe1\xc7\xe3'
>>> s4
u'\u0633\u0644\u0627\u0645'
>>> print s3
سلام
>>> print s4
سلام
>>> s = u'Русский текст'
Unsupported characters in input
'''

My point is that printing s3 and s4 in interactive mode, should produce the same results as printing s1 and s2 from source file. Loewis's patch handled this as I expected. Also this patch solves my problem of not being able to print u'Русский текст' (that is due to my Windows locale being set to Persian, not Russian.)

Use cp1256 encoding in your source file. It is expected that usually your source files encoding is same as your locale encoding. In such case printing string literals and Unicode string literals produces same result (as they look in the sources).

s1 is '\xd8\xb3\xd9\x84\xd8\xa7\xd9\x85' (u'سلام'.encode('utf-8')) and when it printed in utf-8 locale it produces "سلام", but when it printed in cp1256 locale it produces a mojibake.

When you convert your source file to cp1256 and change a header, s1 will be '\xd3\xe1\xc7\xe3' (u'سلام'.encode('cp1256')) and will produce "سلام" when printed in your cp1256 locale.

The same program will behave different in Windows and Mac.

utf-8 works on Mac (10.6.8), cp1256 does not print some lines

cp1256 works on Windows 7, utf-8 prints some characters in a wrong way

For the record, I use accentuated letters from Portuguese alphabet ( ã ã, for example).

Well, if there is no other way around this, I think it's better to apply Martin's patch. At least then we will be able to enter any valid utf-8 character in IDLE (although print statement won't print correctly unless the string is decoded first).

(As a Windows user, currently I can't print u'йцук' in interactive mode and get an "Unsupported characters in input" error because my default system encoding (cp1256) can't encode Russian.)

Also it will be more Unicode friendly which is not a bad thing.

Here is a patch with a hack which corrects a line number in displayed traceback. It doesn't solve a problem totally, but perhaps it is a good enough approximation.

Thank you Serhiy for working on this. This patch solves the problem when all your input characters are encodable using system preferred encoding. But issue19625 persists. I still can't print something like 'Русский текст' in interactive mode.

Another problem is that output of interactive mode and running a module is different if source encoding is something other that system's default. For example:

(With current patch) in interactive mode:

>>> print 'آ'
آ
>>> print u'آ'
آ

But in when running same commands from a file with utf-8 encoding:

ط¢
آ

I know, you siad to use cp1256 encoding in my source file. But I really prefer to work with utf-8 than working with a codepage that is not fully compatible with my language and I believe many other programmers are the same as me (even if there is a codepage they can work with).

(cp1256 is only for Arabic, but (when a program does not support unicode) Microsoft uses it as a second option for some other similar languages like Urdu, Persian. But it does not include all the characters they need.)

IMO these two problems -- being a able to type any Unicode characters in interactive mode and getting the same output as running modules -- are more important than a mere representation problem in interactive mode. (and considering that I use utf-8 for my source files, both of them were solved by martin's patch. [Of course I'm not saying it's the solution, just that it worked better for me])

Terry, what do you think? This bug is critical for non-ASCII-only users of IDLE.

I'm going to commit my patch in few days. This is not perfect solution, but I believe it is better than current state.

It took me awhile to re-recognize Это из консоли: as 'made in konsole'.
I agree that the discrepancy between Idle shell and console is a bug relative to a general goal of having them work as nearly the same as possible and sensible.

> This bug is critical for non-ASCII-only users of IDLE.

This bug, like many others involving non-Ascii chars, has already been fixed in 3.x, though it is not only a 2 versus 3 issue. Given that there are technical limits to imitating the console, we could have closed this issue as already fixed, "wont fix" in 2.7, upgrade to a version that has the fix.  However, since this is also a within 2.7, console versus Idle issue, improvement would be good.

The +coding cookie, -1 line hack induces a non-obvious dependency between files that I think should be documented by comments such as
# run.cleanup_traceback() compensates for added line
# compensation for coding cookie added in pyshel..runsource()

I think it ok to push less than perfect patches that make definite improvements and do not preclude possibly better patches in the future.  I have not seen any comment from alex whether this fixes his particular example on his machine, but if you are convinced that this fixes some problems and only causes a micro slowdown of working uses, go ahead.

Alternative solution is to force UTF-8 in IDLE console. But I think this will be more surprising, especially for Windows users.

Idle 3.x has Martin's patch, except that the 'if' clause is merely commented out rather than removed. (Perhaps someone wanted to test the removal first.  It should now be deleted.)

My 2.7 Idle has Default Source Encoding ... [x] None. Since there is nothing for this in either config-main.def, I presume it is the default -- at least for Windows. But Martin says it is not relevant.

'Surprising' suggests that you think or know that doing the same in 2.7 might introduce other discrepancies from console behavior. I am reluctant to do that, though it is possible that it might make Idle *better* than the console.

I would definitely be reluctant to do something more wide-ranging than your patch without extensive tests, and even that patch should have some tests, some of which pass now and continue to pass, and other that fail now and pass with the patch.

Irdb said "As a Windows user, currently I can't print u'йцук' in interactive mode and get an "Unsupported characters in input" error because my default system encoding (cp1256) can't encode Russian.)"

Idle 2.7
>>> u'йцук'
Unsupported characters in input

Console 2.7
>>> u'????'
u'????'
>>> u'????'[0] == '?'
True

When someone reads file in locale encoding (either as str or unicode in 2.7), he gets printable result. When someone lists directory, he gets printable result (str or unicode). When someone enter string literal (either str or unicode) in interactive mode, he gets printable result. This is expected behavior. And this means that default encoding for text files, filesystem encoding and encoding used in interactive Python (or IDLE console) should be the same, locale encoding.

> Irdb said "As a Windows user, currently I can't print u'йцук' in interactive mode and get an "Unsupported characters in input" error because my default system encoding (cp1256) can't encode Russian.)"

This is different issue. It would be better if IDLE will filter pasted text and replace unencodable characters or underscore them with wavy red line.

I wonder if we should consider extracting the body of
        if isinstance(source, types.UnicodeType):
as a function, call it ufix_latin1; add one or more alterntives, ufix_utf8, ufix_locale, ufix_irdb(?); keep ufix_latin1 as default; but switch to an alternative according to an Idle command line switch (or environmental variable?)?

Only ufix_utf8 (with the utf-8 encoding of stdin/stdout/stderr) and ufix_locale (with hacked tracebacks) make sense.

#19625, with a bit of discussion, was closed as a duplicate of this issue.

New changeset 247f003b42ea by Serhiy Storchaka in branch '2.7':
Issue #15809: IDLE shell now uses locale encoding instead of Latin1 for
https://hg.python.org/cpython/rev/247f003b42ea

Any update on this?

Serhiy, unless you are planning to do something more, please close this.  The extended extended maintenance period for 2.7 is mainly for build and security issues, and I don't expect to do any more 2.7-only patches.

It would be nice to add configuration option for IDLE Shell encoding and highlight non-encodable characters in Shell and Editor. But this feature is needed mainly for 2.7.