in bug 1528802 ( see
https://sourceforge.net/tracker/index.php?func=detail&aid=1528802&group_id=5470&atid=105470
) , I noticed that idle shell behaviour WRT
non-ascii chars was different than python console, and
possibly broken.

For example, IDLE produces:

>>> print u"á"
á
>>> print len(u"á")
2
>>> print "á"
á
>>> print len("á")
2

-------
a python shell (gnome-terminal):

>>> print u"á"
á
>>> print len(u"á")
1
>>> print "á"
á
>>> print len("á")
2

Both are using es_ES.utf-8 system encoding.

IDLE can manage unicode, it is just input that gives
problems:

>>> import unicodedata
>>> print unicodedata.lookup("LATIN SMALL LETTER A
WITH ACUTE")
á
>>> print len(unicodedata.lookup("LATIN SMALL
LETTER A WITH
ACUTE"))
1

Not that I like that much the violation of the least
surprising behaviour that python console offers with
non-ascii letters, but at least some internal
consistency would be great, until python 3000 gives us
true strings.

I'm using python 2.5 (svn trunk) --with-unicode=ucs4

works in python 3ka2 (svn as of today):

>>> print("á")
á
>>> print(b"á")
SyntaxError: bytes can only contain ASCII literal characters.
(<pyshell#5>, line 1)

as it should, so the problem appears in 2.* only.

This is about a disparity between IDLE and the python shell. I'm
guessing different encodings are to blame here and that this is invalid.

The disparity is present in an UCS4 build (IDLE shows UCS2-like
behavior[1], maybe because it's using UTF8?).

[1] http://mail.python.org/pipermail/python-dev/2008-July/080886.html

This issue is caused by compile() behavior.

Following sample is in codepage 932.

>>> 'あ'
'\x82\xa0'  # OK - 'あ' is '\x82\xa0' in cp932
>>> u'あ'
u'\u3042'   # OK - u'あ' is '\u3042' in UCS-2

compile as byte string.
>>> c = compile("'あ'", 'test', 'single')
>>> exec c
'\x82\xa0'  # OK
>>> c = compile("u'あ'", 'test', 'single')
>>> exec c
u'\x82\xa0' # NG!!!

compile as unicode string.
>>> c = compile(u"'あ'", 'test', 'single')
>>> exec c
'\xe3\x81\x82' # NG!!!
>>> c = compile(u"u'あ'", 'test', 'single')
>>> exec c
u'\u3042'  # OK

compile as byte string with pep 0263
>>> c = compile("# coding: mbcs\n'あ'", 'test', 'single')
>>> exec c
'\x82\xa0' # OK
>>> c = compile("# coding: mbcs\nu'あ'", 'test', 'single')
>>> exec c
u'\u3042'  # OK

This patch is for iplib/PyShell.py#ModifiedInterpreter.runsource.

         if isinstance(source, types.UnicodeType):
             import IOBinding
             try:
                 source = source.encode(IOBinding.encoding)
+                source = "# coding: %s\n%s" % (IOBinding.encoding, source)
             except UnicodeError:

Updating the components as the error surfaces in the compile builtin.
the compile builtin works when given unicode, but fails when using a
utf8 (local input encoding) string.

Rather than adding a "coding" string to compile, my guess is that
compile should be fixed or fed a unicode string. See the effects on the
shell:

>>> print len('à')
2
>>> print len(u'à')
1
>>> exec compile("print len('à')",'test', 'single')
2
>>> exec compile("print len(u'à')",'test', 'single')
2
>>> exec compile("print len('à')".decode("utf8"),'test', 'single')
2
>>> exec compile("print len(u'à')".decode("utf8"),'test', 'single')
1
>>> 

So the error disappears when the string fed to exec compile is properly
decoded to unicode.

In idlelib there is an attempt to encode the input to
IOBindings.encoding, but IOBindings.encoding is broken here, as
locale.nl_langinfo(locale.CODESET) gives 'ANSI_X3.4-1968', which looks
up as 'ascii', while locale.getpreferredencoding() gives 'UTF-8' (as it
should).


If I comment the whole attempt, idle works (for this test, not fully
tested):

sgala@marlow ~ $ diff -u /tmp/PyShell.py 
/usr/lib64/python2.6/idlelib/PyShell.py
--- /tmp/PyShell.py	2009-04-12 11:01:01.000000000 +0200
+++ /usr/lib64/python2.6/idlelib/PyShell.py	2009-04-12
10:59:16.000000000 +0200
@@ -592,14 +592,14 @@
         self.more = 0
         self.save_warnings_filters = warnings.filters[:]
         warnings.filterwarnings(action="error", category=SyntaxWarning)
-        if isinstance(source, types.UnicodeType):
-            import IOBinding
-            try:
-                source = source.encode(IOBinding.encoding)
-            except UnicodeError:
-                self.tkconsole.resetoutput()
-                self.write("Unsupported characters in input\n")
-                return
+        #if isinstance(source, types.UnicodeType):
+        #    import IOBinding
+        #    try:
+        #        source = source.encode(IOBinding.encoding)
+        #    except UnicodeError:
+        #        self.tkconsole.resetoutput()
+        #        self.write("Unsupported characters in input\n")
+        #        return
         try:
             # InteractiveInterpreter.runsource() calls its runcode()
method,
             # which is overridden (see below)


>>> print len('á')
2
>>> print len(u'á')
1
>>> print 'á'
á
>>> print u'á'
á
>>> 


Now using Python 2.6.1 (r261:67515, Apr 10 2009, 14:34:00) on x86_64

utf-8 is not locale encoding.

>>> f = open('á.txt')

If this line compiled into utf-8 and locale encoding is not utf-8, can't 
open 'á.txt'.

IMHO, in case of Python 2.x, correct approach is fix IOBindings.encoding  
and compile() with pep0263.

How to use locale.getpreferredencoding() instead of 
locale.nl_langinfo(locale.CODESET).

--- IOBinding.py.back	Sun Apr 12 19:54:52 2009
+++ IOBinding.py	Sun Apr 12 20:02:58 2009
@@ -35,40 +35,16 @@
 # Encoding for file names
 filesystemencoding = sys.getfilesystemencoding()
 
-encoding = "ascii"
-if sys.platform == 'win32':
-    # On Windows, we could use "mbcs". However, to give the user
-    # a portable encoding name, we need to find the code page
-    try:
-        encoding = locale.getdefaultlocale()[1]
-        codecs.lookup(encoding)
-    except LookupError:
-        pass
-else:
-    try:
-        # Different things can fail here: the locale module may not be
-        # loaded, it may not offer nl_langinfo, or CODESET, or the
-        # resulting codeset may be unknown to Python. We ignore all
-        # these problems, falling back to ASCII
-        encoding = locale.nl_langinfo(locale.CODESET)
-        if encoding is None or encoding is '':
-            # situation occurs on Mac OS X
-            encoding = 'ascii'
-        codecs.lookup(encoding)
-    except (NameError, AttributeError, LookupError):
-        # Try getdefaultlocale well: it parses environment variables,
-        # which may give a clue. Unfortunately, getdefaultlocale has
-        # bugs that can cause ValueError.
-        try:
-            encoding = locale.getdefaultlocale()[1]
-            if encoding is None or encoding is '':
-                # situation occurs on Mac OS X
-                encoding = 'ascii'
-            codecs.lookup(encoding)
-        except (ValueError, LookupError):
-            pass
+encoding = "utf-8"
 
-encoding = encoding.lower()
+preferredencoding = None
+try:
+    preferredencoding = locale.getpreferredencoding()
+    codecs.lookup(preferredencoding)
+    encoding = preferredencoding.lower()
+except LookupError:
+    pass
+del preferredencoding
 
 coding_re = re.compile("coding[:=]\s*([-\w_.]+)")

As indicated in msg59954, it works fine on 3.x, so removing these versions.

For 2.7, I don't think it's possible to really fix this. I see the following options:

A. current status. Byte strings are compiled correctly, Unicode strings are not.
B. compile source as a Unicode string, as proposed in msg85886. Unicode strings are compiled propertly, byte strings are not (they get compiled as UTF-8, when they should get compiled in the locale encoding)
C. prefix source with encoding declaration, as proposed in msg85882. Both Unicode strings and byte strings get compiled correctly, but line numbers in tracebacks are wrong.

Given that it's not possible to fix this without breaking something else, and given that it's fixed in Python 3, I propose to declare this as "won't fix" for Python 2.7.

In any case, the bug is certainly not in compile(), which is behaving exactly as specified, so I revert the title change.

Closing as won't fix.