classification
Title: Documents input/output effects of how IDLE runs user code
Type: enhancement Stage: needs patch
Components: Documentation, IDLE Versions: Python 3.5, Python 3.4, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: terry.reedy Nosy List: Al.Sweigart, akira, docs@python, irdb, martin.panter, ned.deily, serhiy.storchaka, terry.reedy, willingc
Priority: normal Keywords:

Created on 2015-01-11 00:54 by Al.Sweigart, last changed 2016-03-14 10:28 by akira.

Messages (19)
msg233828 - (view) Author: Al Sweigart (Al.Sweigart) * Date: 2015-01-11 00:54
IDLE cannot display the \b backspace character correctly. From IDLE's interactive shell:

>>> print('hello\b\b\b\b\bHELLO')
hello◙◙◙◙◙HELLO

Whereas from cmd:

>>> print('hello\b\b\b\b\bHELLO')
HELLO
msg233830 - (view) Author: Carol Willing (willingc) * (Python committer) Date: 2015-01-11 02:31
FWIW: On Mac OS X 10.9.5 using Python 3.4.2, IDLE's interactive shell (started from the IDLE.app icon):

>>>print('hello\b\b\b\b\bHELLO')
helloHELLO

From the command line using python3 interactive shell:
>>>print('hello\b\b\b\b\bHELLO')
HELLO

Both return a <class 'str'> for type('hello\b\b\b\b\bHELLO') 
Interestingly, both behave the same when executing:
>>>'hello\b\b\b\b\bHELLO'
'hello\x08\x08\x08\x08\x08HELLO'


I'm not sure that IDLE is used much on OS X since the Terminal is easily available. Since K12 education may use it, it would be nice to have consistency across the OSes.
msg233831 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-01-11 02:50
As far as I understand, Idle doesn’t interpret any terminal control codes apart from a plain \n for a new line. I know it doesn’t do a carriage return for \r either.
msg233832 - (view) Author: Al Sweigart (Al.Sweigart) * Date: 2015-01-11 04:09
For clarification, this happens on Windows 7.
msg233838 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2015-01-11 07:22
It's helpful to keep in mind that IDLE is a Tk application written in Python and, as such, depends on Tk to do nearly everything associated with writing to displays and reading from keyboards and pointing devices.  If you try outputting the same string using the Tk shell, wish, you'll see exactly the same behavior as you do in IDLE:

$ wish
% .text insert 1.0 "hello\b\b\b\b\bHELLO"
% pack .text

And you'll see the same Tk platform differences.  With the native OS X Tk's, the backspace characters aren't displayed; with a Linux or OS X X11 Tk, empty box characters are displayed.  That may also depend on which font is in use.  In any case, the Tk text widget does not behave the same way in this regard as most terminal emulator windows do would do with backspace characters.  So it's up to any Tk app to figure out how it wants to deal with them.

This issue has come up before for Tkinter in general: back in 2001, effbot suggested some code to search for and edit backspace runs in Tk text.  Presumably something similar could be added to IDLE if there was general agreement that this was desirable (after verifying that it works on all of the Tk platforms that IDLE supports).  See https://mail.python.org/pipermail/python-list/2001-December/085908.html
msg233862 - (view) Author: Carol Willing (willingc) * (Python committer) Date: 2015-01-11 20:19
Ned, Thanks for the detailed example and confirming my gut instinct that Tk was the root cause of the differences seen between the IDLE's Python interactive shell (https://docs.python.org/3.4/library/idle.html) and the interactive interpreter invoked from the command line (https://docs.python.org/3.4/tutorial/interpreter.html#tut-invoking).

As an end user learning Python (such as the elementary education market), the current Standard Library documentation on IDLE guides me to the incorrect conclusion that the "Python shell window (aka interactive interpreter)" in IDLE would behave the same as invoking the interactive interpreter from the command line.

It seems reasonable to explicitly state in the Standard Library doc that:
"In rare cases, such as text handling with certain special characters (i.e. '\b' in a string), the IDLE's interactive Python shell may return a different response than the Python interactive interpreter invoked from the command line. This is due to IDLE's low level dependence on Tk (Tk itself is not part of Python; it is maintained at ActiveState. reference: first paragraph of https://docs.python.org/3.4/library/tkinter.html#module-tkinter)."
msg234162 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2015-01-17 04:58
Going back to msdos, there are graphic chars for all 256 bytes, including may single and double line box-drawing chars. Many In Idle, I see 5 solid white circles.  In FireFox, there are 5 empty circles (on dark background, which are chr(9689).  When I copy from FF back to Idle (3.4.2, Win7), there are 5 of each. I have no idea if the 9689s are on the site or added by FF.

Here is another difference.

>>> print('\x03')  # console
♥  # heart

>>> print('\x03')  # idle
  # lower left single line corner in Idle, box on FF

Trying to match console-Idle(tk) print output for control chars even on Windows would be tough.  
---

I have been planning to add a subsection of the doc that mentions known differences between console interpreter and Idle shell.  The result of print() is one of them.  Another print difference is that Idle displays many unicode chars that Windows replaces with boxes or ?s, depending on the codepage.
msg246596 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2015-07-11 08:00
I closed #24572 as a duplicate of this. It is the same issue except for printing \r instead of \b.  These issues are not about responses to keyboard actions when entering text into an Idle editor window.  During entry, just about any cntl/alt/function/shift/key combination can be intercepted to and translated to arbitrary action. They are also not about REPL echo.  For 3.4+ Win7, both console Python and Shell echo '\b\t\x08\x09' as '\x08\t\x08\t'. Carol's report suggest that the same is true on Mac also.

Both issues *are* about print(s, file=outfile), where s is the result of processing all args except 'file' and outfile defaults to sys.stdout. The print call is the same as outfile.write(s), so outfile (and sys.stdout) could be any object with a write method.

>>> print('hello\b\b\b\b\bHELLO')
helloHELLO
>>> import sys; sys.stdout.write('hello\b\b\b\b\bHELLO'+'\n')
helloHELLO
(I actually see the same circles as Al, but copy-paste does not seem to work as well for me.)

So both issues are about the effect of writing 'control chars', in particular \b and \r, to a file. Well, that depends on the file.  Some possibilities are copy as is (StringIO), encode and copy (disk file, socket), ignore, display as one glyph, display as multiple chars, non-destructively backspace (like backspace on typewriters and printing terminals and left-arrow elsewhere), or destructively backspace (like backspace on as most (all?) screen keyboards).  After non-destructive movement of the 'cursor' position, the possibilities for following graphical chars are overwrite (like typewriters), replace, and insert (the modes sometimes selected by the Insert key). Non-destructive backspace followed by overwrite (meaning 'HELLO' printed on top of 'hello') is the original meaning of 'backspace'.

Having said all this, I am sympathetic to the idea that there should be an option to have 'print(ascii_string)' in user code give the same result in the console and Idle.  I think this would best be accomplished by a least-common-denominator SimpleTerm subclass of tkinter.Text put somewhere in the tkinter package. (This would be a new issue that should start on python-ideas list.)  However, I would consider something Idle specific.

Does the following work the same on *nix and Mac as Windows?

>>> print('hello\rHELLO')
HELLO  # helloHELLO with Win7 Idle

Are there any control-chars (other than \n) that work might work in the consoles on all three systems and that should be included?

Carol, another difference between the Windows console and Idle is that tk and hence Idle support the entire BMP subset of unicode.  This should also be mentioned in the doc.
msg246602 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-07-11 12:29
Control characters are named control characters because they are control the output device. Different devices have different capabilities and reacts different on the same codes. Windows console, different sorts of Linux terminals and Tk text widget are different devices. Some prints funny characters, others not, some beeps, others not, some interprets particular flavor of ESC sequences, others not, some allows color and positioning, others not. Python can't unify the behavior of these devices without lost most of functionality as it can't unify the behavior of black-white matrix printer, graphical plotter and 24-bit color LCD monitor.

I would close this issue as not related to Python.
msg246614 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2015-07-11 21:35
Serhiy, thanks for the paraphrase of what I tried to say.  This issue has already been changed to a doc issue to explain the input/output effects of Idle's way of running user code.  I should have retitled before.  Control codes are just one of the effects that have come up on the tracker, python-list, or stackoverflow. Misunderstanding had lead to user puzzlement and even undeserved insults directed as Python.

I propose to something like the following to the introductory section of the Idle doc, https://docs.python.org/3/library/idle.html#index-0, after the feature list.

---
When Idle runs user code from either the shell or an editor window, the result is nearly always be the same as running the same code directly with python in either interactive or batch mode.  However, environmental differences can have visible effects. For instance, Idle import statements add numerous entries to sys.modules and a few module attributes to modules such as tkinter.  The line `1import tkinter; tkinter.colorchooser`` normally raises AttributeError but works when run with Idle because Idle has already imported tkinter.colorchooser. To detect whether code is running under Idle, run ``import idlelib; idlelib.run`` within a try-except statement.

More important are the input/output differences.  Python normally runs in a text console process with direct access to keyboard and screen. For code run with Idle, the keyboard and screen are controlled by the tk graphics subsystem and sys.stdin, sys.stdout, and sys.stderr are bound to objects that connect to the gui. (Note that ``print`` calls ``sys.stdout.write``.)  Here are some of the effects of keyboard and screen access being indirect.
  * Operating system (OS) specific functions that directly access the keyboard may not work.
  * The Idle shell works with complete statements rather than individual lines of code.  One can edit and retrieve complete multiline statements instead of single lines.
  * User code gets colorized. Normal output and error output to the shell get their own colors.
  * Tk supports the Basic Multilingual Plane (BMP) subset of Unicode characters.  This is worse than consoles that support supplementary planes (with appropriate fonts in use), but better than the Windows console, which only supports restricted subsets of the BMP, depending on the code page in use.
  * Tk handling of ascii control chars depends on the OS and is usually different from the text console.
---
msg246626 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-07-12 02:41
“run ``import idlelib; idlelib.run`` within a try-except statement”: It might be nice to say what exceptions are expected. My guess is ImportError if Idle or TK is not available, or AttributeError if it is but Idle is not running.

“Tk handling of ascii control chars”: I presume you mean stdout and stderr handling, which this bug was originally about (or also input, source code, etc as well?). It might be good to say that output of Unix newlines (\n) is guaranteed to be supported. Also might be worth explicitly pointing out that output of CRLFs is not supported, even if os.linesep is "\r\n". In my experiments between Linux and Wine, this does not appear to depend on the OS.
msg246629 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2015-07-12 03:55
I was thinking AttributeError, as mentioned in the previous sentence.  But you are correct that ImportError is possible too.  Maybe I should just give the code.

try:
    import idlelib
    idlelib.run
    running_idle = True
except (ImportError, AttributeError):
    running_idle = False

tk does not 'handle' stdout. Idle does, by inserting strings into a tk text widget. tk does not care where inserted chars come from. tk \b behavior is OS dependent. tk may always ignore \r, but this is different from (at least some) consoles. Attempt 2, with and added paragraph.

...
  * Except for newline ('\n'), tk handling of ascii control chars may depend on the OS and may by different from text consoles.  Both are true for backspace ('\b') and the latter for return ('\r'), which tk ignores.

These differences noted above are not bugs. If an application is going to be run repeatedly after being developed with Idle, it should usually be run directly, without Idle.  (An exception would be non-gui Windows apps that need tk's better unicode support.) That means testing at least once without Idle.
---

Thanks for the comments.
msg246633 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-07-12 04:59
I wouldn’t say TK ignores carriage returns, though I agree it would be better if Idle stripped them out. Currently I get a glyph displayed for them, similarly to \b. They wouldn’t copy to my clipboard, so I fudged them after pasting here:

>>> _ = stdout.write("CRLF\r\n")  # Linux: box-drawing corner piece
CRLF┌
>>> _ = stdout.write("CRLF\r\n")  # Wine: euro sign
CRLF€
msg246673 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2015-07-12 21:27
OK. "Both are true for backspace ('\b') and return ('\r')."
msg251825 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2015-09-29 06:51
Issue #21995 discussed same issue from cause side, as opposed to result. The new section is partly based on what I wrote above.  I am not satisfied with it yet.

New changeset ac6ade0c5927 by Terry Jan Reedy in branch '2.7':
Issue 21995: Explain some differences between IDLE and console Python.
https://hg.python.org/cpython/rev/ac6ade0c5927

New changeset ca6c9cc77c20 by Terry Jan Reedy in branch '3.4':
Issue 21995: Explain some differences between IDLE and console Python.
https://hg.python.org/cpython/rev/ca6c9cc77c20
msg261528 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-03-10 22:42
Different \r behavior is the gist of https://stackoverflow.com/questions/35895864/what-is-the-difference-between-cmd-and-idle-when-using-tqdm.  The new section should have more on the different effect of control characters on different display devices and windows. It should also better differentiate between Python result, which is the string written to stdout, and which is nearly always the same in IDLE and console, and visual display result, which is  outside of Python's control.  This separation is the point of Serhiy's msg246602.
msg261529 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-03-10 23:24
Part of this issue is the difference between typing a character at the keyboard, which produces a key event whose effect is determined by key bindings (some default, some added) and inserting a character into a Text widget, which does not produce an event (except, in some sense, \t and \n).  Whether or not a key event results in character insertion depends on key bindings.  The visual effect of inserted chars other than \t and \n is determined by the font.  (I believe I got the above right.) I believe some terminals treat chars coming over the wire the same as those typed.  I am not sure what the Win10 console is doing with control-x keystrokes.

>>> import sys; out=sys.stdout.write
# use direct write to avoid str(), repr() conversion of control chars
>>> out('\x03')
1  # The first char is an empty thin box in my IDLE.
>>> # type ^C
KeyboardInterrupt
msg261531 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-03-10 23:32
As for StackOverflow question, there is tkinter.ttk.Progressbar.
msg261738 - (view) Author: Akira Li (akira) * Date: 2016-03-14 10:28
IDLE can implement functionality similar to what colorama [1] module does on Windows: translate ANSI escape character sequences into corresponding GUI method calls.

For example, \b might be implemented using a .delete() call, \r using .mark_set(), etc.

[1] https://pypi.python.org/pypi/colorama
History
Date User Action Args
2016-03-14 10:28:59akirasetnosy: + akira
messages: + msg261738
2016-03-10 23:32:38serhiy.storchakasetmessages: + msg261531
2016-03-10 23:24:04terry.reedysetmessages: + msg261529
2016-03-10 22:42:17terry.reedysetmessages: + msg261528
2015-12-04 02:27:56irdbsetnosy: + irdb
2015-09-29 06:51:54terry.reedysetassignee: docs@python -> terry.reedy
messages: + msg251825
stage: patch review -> needs patch
2015-07-12 21:27:52terry.reedysetmessages: + msg246673
2015-07-12 04:59:42martin.pantersetmessages: + msg246633
2015-07-12 03:55:25terry.reedysetmessages: + msg246629
stage: needs patch -> patch review
2015-07-12 02:41:16martin.pantersetmessages: + msg246626
2015-07-11 21:35:39terry.reedysetmessages: + msg246614
title: IDLE does not display \b backspace correctly. -> Documents input/output effects of how IDLE runs user code
2015-07-11 12:29:21serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg246602
2015-07-11 08:00:56terry.reedysettype: enhancement
messages: + msg246596
stage: needs patch
2015-07-11 01:59:03terry.reedylinkissue24572 superseder
2015-01-17 04:58:27terry.reedysetnosy: + terry.reedy, docs@python
messages: + msg234162

assignee: docs@python
components: + Documentation
2015-01-11 20:19:22willingcsetmessages: + msg233862
2015-01-11 07:22:51ned.deilysetnosy: + ned.deily
messages: + msg233838
2015-01-11 04:09:37Al.Sweigartsetmessages: + msg233832
2015-01-11 04:07:15Al.Sweigartsetversions: + Python 2.7
2015-01-11 02:50:55martin.pantersetnosy: + martin.panter
messages: + msg233831
components: + IDLE
2015-01-11 02:31:10willingcsetnosy: + willingc
messages: + msg233830
2015-01-11 00:54:47Al.Sweigartcreate